Tag Archives: metamodel

A good baseline presentation on creating semantic data repositories

8 Jul

I have been meaning to give a shout out to Dan MCCreary http://www.danmccreary.com/ for some time now, and am only just getting around to it. What caught my attention was a presentation Dan gave at the 2006 Semantic Technology Conference. The presentation walked the audience through the process of creating a well-structured data repository enabling players within the Minnesota education system to ask complex across all of the data available. While the presentation is rather dated, it is one of the better presentations I have seen that takes things back to a level most people can understand. Additionally, I don’t see that the state of the industry has changed that much in terms of tools to support the creation well-structured data repositories that support broad “knowledge management” goals.

Dan focuses on metadata as the means to achieve the goals of the participants: Minnesota Department of Education; the Wisconsin Department of Public Instruction and the Michigan Department of Education. The goal is to create a semantic understanding of the data available. Semantic is one of those words that means many things to different people. Once you combine “semantic” with “knowledge management” or ”knowledge discovery”, everyone seems to have a different opinion as to what you are talking about – IT and business folks alike. It becomes very important to level set your audience. For the purposes of this discussion, creating a semantic repository involves:

  • Enhancing data with metadata that focuses on describing the data asset as a whole, and with respect to its component content parts;
  • Creating descriptive and content metadata that describes the data in specific terms, and in more general “conceptual” terms.
  • Creating a description of the data that reveals connections to other concepts; related terms; multiple standards.
  • Creating highly linked data that is organized through taxonomies and ontologies to reveal links that are perhaps not obvious to users.

The desired end state is the ability for a broad range of end users to query disparate data sources and understand how available date can be used in their particular context.

Within the referenced deck (http://www.danmccreary.com/presentations/semweb2006/), there are a number of sections worth drawing your attention to:

ISO 11179 (Starting on page 15). Most people I speak to that know about ISO 11179 (http://www.iso.org/iso/home/search.htm?qt=11179&published=on&active_tab=standards&sort_by=rel ) are somewhat skeptical as the spec is a heavy on as written. Once you get your head around all the good ideas that have gone onto the specification, the question becomes how much of it do you adopt? The idea behind the spec is that each piece of metadata has its own metadata that allows a user to know exactly how that metadata is to be used. Dan presents the 11179 concept within his discussion of the National Information Exchange Model (NIEMS https://www.niem.gov/Pages/default.aspx) which brings us to another interesting point…

Mapping to Standards (Page 16). I like the way that the process of mapping to standards is presented. In this instance, the Educational System is only trying to map to one standard. However, you can imagine an environment where there are multiple standards of interest. In this instance, the mapping approach is organized around the OWL “SameAs” terminology (Web Ontology Language http://www.w3.org/TR/owl-features/). It does not say it in the presentation, but I am assuming that this is taken from the SKOS Simple Knowledge Organization System standards (http://www.w3.org/2004/02/skos/). As you classify data assets, there will be multiple mapping exercises where the mapping will not be exact. The linking concepts defined within SKOS will allow you to relate items that are not precise or consistent in their relationships.

Demonstrating that the data problem grows faster than the number of data sources (Page 17). This sounds so simple. However, I am still surprised when I hear people say – “we have the data mapped between the sources I want, so no need to spend time mapping to standards”. Well ok that works if you have a small number of static data sets, but … that happens rarely. In the presentation Dan refers to this as the O(N2) problem versus the O(N) problem. I have included his graphics as these tell the story better than words.

network Problem

Tags: , , , , ,

Dealing with Fragmented Metadata

15 Jan

The overall challenge is one of sorting out the metadata management within the organization. Presumably this is creating data quality problems of some sort.

It is somewhat hard to evaluate the metadata question in the absence of a few other data points, as metadata is everywhere, and is influenced by numerous factors: overall data management policies, the information architecture, the system architecture, and the security policies to name a few. Additionally, there is often a mis-alignment between these factors and how the organization’s management structure supports data management best practices.

First step – define the data quality issue. This breaks out along three dimensions: 1. how users find and retrieve data assets; 2. How data is delivered; and, 3. how data is managed.

Metadata_Framework

Breaking data down along these dimensions will expose the function that is creating the duplication and the kind of metadata involved.

Additionally viewing the problem this way will address how the organization is thinking about the process. How is the library function addressed? Is there a cataloging function that keeps an inventory of all data assets available to the organization? Is there a role of data scientist – perhaps data “packager” is a better term for most organizations. Is there an organizational element to why metadata is getting fragmented? An example of this would be that the Librarian function is organizationally part of the Research Division. Data collection and packaging is an IT function. The Library software that manages the search and cataloging functions is managed separately from the metadata management system supporting the data warehouse. Anything done by Research creates fragmentation, and possibly duplication of metadata.

Once this perspective is applied, one gets a better idea of the impact related to the data quality problem. Issues of a cumbersome search interface, can be addressed in the short term through better training, or increased resources on the help desk. However, security or audit related shortcomings might result in a security breach which would have much greater impact.

So what to do? Clearly this is not just a metadata problem. In many organizations, the challenge of duplicated metadata is created by legacy environments that have grown fragmented over time. The roadmap that defines the path forward will be unique for each organization. However, a few best practices are called for.

  1. Creation of an information architecture / framework that all can agree on.
  2. Creation of an agreed on set of “states” that can be applied to data assets that are flowing though this information architecture.
  3. Creation of a policy/process/standards matrix that defines how these are applied to data assets based on where they are in the information architecture
  4. The above implies that there is a data governance component within the organization’s management structure – if this does not exist, it needs to be established.

From a platform perspective there are two best practice considerations:  a metadata registry should be evaluated; and metamodels created by which all metadata are viewed. The registry is most important, as it imposes a discipline on the metadata management process.

Think of a metadata registry as the “reference data” or the “controlled vocabulary” for metadata. For any given data asset, it defines what metadata one should expect to see. If it is not in the metadata registry, it should not exist. All of the IT tools that are engaged in moving, enhancing or creating derived works from data assets should use the metadata registry as the reference source for all activity related to metadata. This reduces the likelihood that metadata will need to be duplicated, and if it is duplicated, it reduces the likelihood that it is duplicated incorrectly.

This is managed by the ISO 11179 specification. My sense is that the development of a metadata registry is something that comes with Big Data. Just like reference data, and master data, metadata is a management challenge that is supported by IT capabilities NOT the other way around.

Tags: , ,