![Page 1: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/1.jpg)
1
A Metadata ArchitectureFor
Enterprise-Wide Data Sharing
![Page 2: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/2.jpg)
2
Same Data RequirementsDifferent Functional Needs, Same Descriptions, Different Names
Department of Defense (DoD)Data Interoperability Challenge
Logistics
Components/Services
Medical
Procurement
Finance
Transportation
PersonnelCommand& Control
![Page 3: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/3.jpg)
3
MIMMS
ATLAS II
CAMS
AMMRL
IMRL
NAOMISATAC
NSIPS
NALDA II
U2
SAUDI ARABIA
QATAR
KUWAIT
UAEMEXICO
JAPANFRANCE
UKCANADA
DVD Prime Vendors
(personnel?)
ARAMIS
CMIS
SUPPLY MGMT
SCS
CRIM
CAIMS
GATES
GDSS
WPS
IBS
CFM
DITTS
RFT-E
RFT-K
COP CSE
GCCS - C2
MTMCAMS
CMOSTCACCIS
G081BROKER
GOPAX
ADANSJALIS
MC-TFS
LIF
DSS
CC SS
AWRDS APS
SAILS
SAMS
TAMMISMEDSUP
ALP
JL ACTD
DOD CAV
TAMMISMEDASM
MEDSILS
UDR
VMI
NAV
NAC
MEDLOG
DBSS
MUFFIN
SNAP/SUADPS/FIMAR
FUELS - NEURS
CAIMS
ATAC
MANPERS
SBSS
CAS A
GO81
AFEMSDO35
MAARS II
MPS (BIC)
SCS
SASSY
FIMAR
CASEMIS
ISRAEL
OAS
ASIAN
SEATO
NATO
TCAIMS II
SARSS
SAAS-MOD
ROAMS
TAPDB
TAMMIS
ATAV (LIDB)
WARS
AMMIS
ARMS
PMIS
ATEMS
SPS/SDW
SAMMS
FLIS
JECPO
URD
DAAS LOTS
IC3
DoD JointApplicationsGCCS & GCSS
(IDEs)
CLASPENTAGON
CLASPACOM
CLASUSFKJTAV
CLASJFCOM
CLASEUCOM
CLASCENTCOM
UNCLASPENTAGON
UNCLASPACOMServer
UNCLASUSFK JTAV
Server
UNCLASJFCOMServers
UNCLASEUCOMServer
UNCLASCENTCOM(SOUTHCOM
SOCOM)Server
DISA
TRANSCOMGTN
DLA
USA
COALITION
USN USMC
USAF
MILSEALIFT
Commercial
JMAR
DARPA
GSA
TAPETAPETAPEISSE GUARDISSE GUARD
Freight Links
SCCR
CAIMS
USCG
DoD TARGET DATA SHARING ENVIRONMENTDoD TARGET DATA SHARING ENVIRONMENTA Logistics PerspectiveA Logistics Perspective
ATAC
MRP II
GCCS = Global Command & Control System
GCSS = Global Combat Support System
Essential AIS
Contributing AIS
Legend:
![Page 4: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/4.jpg)
4
METADATAREPOSITORY
Defense DataDictionary System
(DoD Standardized Data Elements {SDE})
17000+ SDEs
The DDDS: The Current DoD Repository of DoD Standardized Data Elements
Intended to be the DoD Repository of Data Elements toSupport DoD Enterprise-wide Interoperable Data Sharing
![Page 5: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/5.jpg)
5
PART I:The Problem
![Page 6: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/6.jpg)
6
Current Problems
1. Incorrect data architecture abstraction level for representing Enterprise Level Data Elements for interoperable data sharing
2. Numerous redundant representations of Standardized Data Elements (SDEs) (DIFFERENT NAMES – SAME DEFINITION)
3. Incomplete, non-existent and/or, non-current SDE metadata
4. Inadequate categories of SDE metadata
5. Inadequate support / enforcement of data administration processes for data management
![Page 7: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/7.jpg)
7
A Data Element Name and Data Element Definition Refresher:Two Data Element Metadata Attributes
• Data Element Name is a label given to establish Data Element identity
• Data Element Definition is a description providing complete, unambiguous meaning represented by a Data Element
• Name and Definition together provide the semantic context for the data item values represented by a Data Element
• Some key concepts/principles/facts about Data Element Naming and Definition:o NAME and DEFINITION are inseparableo NAME is a unique identifier for DEFINITION
o A NAME can be viewed as a kind of very short DEFINITION
o In order of precedence, DEFINITION creation should always precede NAME creationo Impossible to correctly NAME a data element with precision without a DEFINITION
o NAME and DEFINITION are at the core of the data integration / sharing process o Many data sharing issues arise from “bad” data element NAMING and DEFINITION practices
![Page 8: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/8.jpg)
8
A Proposed Metadata Architecturefor Shared Enterprise Data Elements
• Focuses on solutions for:o Problem 1 – Incorrect data architecture abstraction levelo Problem 2 – Differently named data elements for the same data element concepto Problem 4 - Inadequate categories of standardized data element metadata
• Not a solution for every kind of impediment to interoperable data sharing
• Problems 3 and 5 require quality improvements in process execution and in data management governance and will be addressed in Part II.
Will begin by looking more closely at the fundamental metadata architecture levels…….
![Page 9: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/9.jpg)
9
Design Layers of a Business Information System Database Architecture
• Specified Context Data Model Layer Roughly analogous to high level conceptual Entity-Relationship (E-R) data models of functional area/business domain, or Community of Interest (COI) data depicting the structure and relationships of entities and their attributes.
• Implemented Technology Data Model Layer Database schemas represented in a particular technology (SQL, COBOL, etc) based on fully attributed 3rd normal form logical data models
• Operational (Vendor) DBMS Data Model Layer Roughly analogous to a physical data model representing a particular vendor’s version of a technology based schema, i.e., Oracle SQL DBMS vs IBM DB2 SQL DBMS vs Sybase SQL DBMS, etc, etc
• Business Application View Data Model Layer Represents the application system access interface to a DBMS which preserves the separation and integrity of the database data from systems that operate on and manipulate the data
![Page 10: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/10.jpg)
10
ATTRIBUTEENTITYSUBJECT
“PERSONNEL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES
ATTRIBUTEENTITYSUBJECT
“LOGISTICS” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES
Design Layers for a Business Information System Database Architecture
ATTRIBUTEENTITYSUBJECT
“FINANCIAL” FUNCTIONAL AREA DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD FDAd Domain)
• May be represented in one, a combination of, or all of the following views:
o Entity w/attributes; no key designations; un-normalized; unresolved many – to – many relationshipso Key based entities; un-normalized; un-resolved many – to – many relationshipso Fully attributed; un-normalized; resolved or un-resolved many – to – many relationshipso Fully attributed; 3rd normal form; developed sub-typing; resolved many – to manys
• Current DDDS / DDA functional / subject area domains map to domains represented by designated DoD Functional Data Administrators (FDAds):
o Logistics DUSD(L)o Personnel USD(P&R)o Comptroller USD(C)o Health Affairs ASD(HA)o Etc, etc
• Each DoD “Subject Area” DAd should have a “specified” data model that represents the data element structures and relationships of functional area data element requirements for their respective functional area or community of interest.
Layer 1
Specified Context (Community of Interest) Data Model Layer
![Page 11: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/11.jpg)
11
Layer 1
Implemented Technology Data Model Layer
TECHNOLGY DEPENDENT (e.g.,IDMS,)FULLY ATTRIBUTED, LOGICAL DATA MODEL
COLUMNTABLESCHEMATECHNOLGY DEPENDENT (e.g.,COBOL,)
FULLY ATTRIBUTED, LOGICAL DATA MODELCOLUMNTABLESCHEMA
TECHNOLGY DEPENDENT (e.g.,SQL,)FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Architect / Modelers Domain)
COLUMNTABLESCHEMALayer 2
• 3RD Normal form ERD logical data model
• Represented in a technology dependent data architecture schema
• Technology driven / constrained data element naming
• Subject area entity and attribute templates are deployed into schema tables and columns that must conform to a particular chosen technology such as COBOL or SQL.
• Layer 1 attribute metadata is inherited by Layer 2 columns
• The relationship between Layer 1 and Layer 2 is one – to – many. That is to say that any attribute from Layer 1 may be deployed as a column into many Layer 2 schemas.
Design Layers for a Business Information System Database Architecture
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD COI DAd Domain)
![Page 12: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/12.jpg)
12
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD COI DAd Domain)
Layer 1
Operational Vendor DBMS Data Model Layer
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Architect / Modelers Domain)
COLUMNTABLESCHEMA Layer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (e.g., Sybase)DBMS DATA MODEL
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (e.g., DB2)DBMS DATA MODEL
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (e.g., Oracle)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs))
Layer 3
• Roughly analogous to a physical data model
• Vendor’s versions of particular technology based schema such as SQL, i.e., Oracle SQL DBMS vs Informix SQL DBMS, vs Sybase SQL DBMS, etc, etc.
• Data element naming is bound by vendor’s implemented DBMS business rules for a particular technology based schema.
• Again, the relationship between Layer 2 and Layer 3 is one – to – many. That is to say that any column from a Layer 2 schema may be deployed as a DBMS column in many Layer 3 DBMSs.
Design Layers for a Business Information System Database Architecture
![Page 13: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/13.jpg)
13
Business Application View Data Model Layer
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA DATA MODEL LAYER
(DoD COI DAd Domain)
Layer 1
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Architect / Modelers Domain)
COLUMNTABLESCHEMA Layer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs))
Layer 3
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEMVIEW DATA MODEL (Command & Control)
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL (Personnel App)
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL (Logistics App)
(Domain of Application System
Managers (SMs and/or PMs)
Layer 4
Design Layers for a Business Information System Database Architecture
• Data element naming in conformance with functional area common business language terms
• Finally, with respect to a single DBMS, the relationship between Layers 3 and 4 is also one– to – many from 3 to 4. That is, a DBMS column may be deployed as view columns in many applications that may interface with a particular DBMS.
![Page 14: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/14.jpg)
14
Layer 1 Person Given Name
Layer 2 Salesman First Name
Layer 4 Name
Layer 3 EFN
Standalone Database #1
Enterprise CommonData Element Concept “A”
Layer 1 Sailor Given Name
Layer 2 Sailor First Name
Layer 4 First Name
Layer 3 Sail_Frst_Nm
Standalone Database #2
Enterprise CommonData Element Concept “A”
Enterprise Registry ofStandardized Data Elements (SDE)To Represent Common Enterprise
Data Element Concepts
Enterprise Data Element Concept “A”
Layer 1 Employee Given Name
Layer 2 Employee First Name
Layer 4 Given Name
Layer 3 Emp_Gv_Nam
Standalone Database #4
Enterprise CommonData Element Concept “A”
SDE “A”: Person Given Name
SDE “A”: Employee First Name
SDE “A”: Sail_Frst_Nm
SDE “A”: Legal First Name
Effective Result: Four differently named versions, or, representations of the Enterprise common Data Element Concept, “A”, that will exist in the Registry as Standardized Data Elements.Redundancy and ambiguity is the consequence.
The Problem: Sourcing Enterprise “Context Independent” Data Element Standards from Enterprise “Context Dependent”
Data and Information Systems and Databases
Enterprise Registry ofStandardized Shared
Data Elements:The DDDS
Standalone Database #3
Enterprise CommonData Element Concept “A”
Layer 1 Legal Given Name
Layer 2 Authoritative First Name
Layer 4 Legal First Name
Layer 3 Lg_Auth_Frst_Nm
![Page 15: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/15.jpg)
15
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD COI DAd Domain)
Layer 1
Database System Architecture Design Steps
• Specified Context Data Model Layer
• Implemented Technology Data Model Layer
• Operational Vendor DBMS Data Model Layer
• Business Application View Data Model Layer
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODELLAYER(Data Architect / Modelers Domain)
COLUMNTABLESCHEMA Layer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs))
Layer 3
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL
(Domain of Application System
Managers (SMs and/or PMs)
Layer 4
METADATAREPOSITORY
Defense DataDictionary System
(DoD Standardized Data Elements)
17000+ SDEs
The DDDS RepositoryIntended to represent DoD globally shared enterprisestandard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept.
Design Layers for a Business Information System Database Architecture
![Page 16: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/16.jpg)
16
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD COI DAd Domain)
Layer 1
Database System Architecture Design Steps
• Specified Context Data Model Layer
• Implemented Technology Data Model Layer
• Operational Vendor DBMS Data Model Layer
• Application View Data Model Layer
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Architect / Modelers Domain)
COLUMNTABLESCHEMALayer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs))
Layer 3
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL
(Domain of Application System
Managers (SMs and/or PMs)
Layer 4
METADATAREPOSITORY
Defense DataDictionary System
The DDDS RepositoryIntended to represent DoD globally shared enterprisestandard data elements. Thus, the repository should contain only one named data element standard for each unique enterprise level data element concept. In reality, the repository contains many cases of differently named data elements that represent the same data element concept. The result is uncontrolled redundancy and ambiguity incapable of supporting seamless and interoperable data sharing.
Design Layers for a Business Information System Database Architecture
A source for DDDS SDEs
A source for DDDS SDEs
A source for DDDS SDEs
A source for DDDS SDEs
One GIGANTICsemantic mess
17000+ SDEs
![Page 17: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/17.jpg)
17
….But, Where’s the Beef ??
.…the Enterprise Data Element “Beef” ??
TheDDDS
“Big Bun”
![Page 18: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/18.jpg)
18
Design Layers for a Business Information System Data Architecture
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
Layer 0
The Enterprise Data Element Layer
• ISO 11179 Naming and Definition
• Context Independent Data Elements
o Uniform Naming
o Uniform Semantics
o Uniform Value Domains
![Page 19: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/19.jpg)
19
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD FDAd Domain)
Layer 1
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Architects / Modelers Domain)
COLUMNTABLESCHEMA
Layer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs)) Layer 3
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL LAYER
(Domain of Application System
Managers (SMs and/or PMs) Layer 4
The DDDS:
17000+ SDEs
Design Layers for a Business Information System Data Architecture
One giganticsemantic mess-redundancies &
ambiguities
DDDS Source
DDDS Source
DDDS Source
DDDS Source
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
Layer 0
![Page 20: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/20.jpg)
20
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
ATTRIBUTE
COLUMN
DBMS COLUMN
ENTITYSUBJECT
TABLESCHEMA
DBMS TABLE DBMS SCHEMA
ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION
FUNCTIONALLY DEPENDENT & TECHNOLOGYINDEPENDENT DATA MODEL TEMPLATESATTRIBUTE INHERITS DATA ELEMENT
“SPECIFIED” DATA MODEL (DoD FDAd Domain)
TECHNOLGY DEPENDENT &DBMS INDEPENDENT MODEL / SCHEMA
COLUMN INHERITS ATTRIBUTE“IMPLEMENTED” DATA MODEL)
(Data Architects / Modelers Domain)
DBMS DEPENDENT &APPLICATION VIEW INDEPENDENT
DBMS COLUMN (Oracle, DB2, etc) INHERITS COLUMN“OPERATIONAL” DATA MODEL
(Domain of Database Administrators (DBAs))
VIEW
VIEW COLUMN
BUSINESS INFORMATION APPLICATION
SYSTEM
VIEWCOLUMN
STRUCTURE
VIEW COLUMNSTRUCTURE
PROCESS
VIEWCOLUMN
STRUCTURETYPE
APPLICATION VIEWS OF DBMS TABLES & COLUMNS
“VIEW” DATA MODEL(Domain of Application System
Managers (SMs and/or PMs)
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA
Data sharing occurs at the “operational andapplication” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.
METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model ISO 11179
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
![Page 21: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/21.jpg)
21
ATTRIBUTEENTITYSUBJECT
FUNCTIONAL AREA / SUBJECT MATTER DEPENDENT DATA MODEL TEMPLATES
“SPECIFIED” DATA MODEL LAYER
(DoD FDAd Domain)
Layer 1
TECHNOLGY DEPENDENT (SQL, COBOL, ETC),FULLY ATTRIBUTED, LOGICAL DATA MODEL
“IMPLEMENTED” DATA MODEL LAYER(Data Modelers Domain)
COLUMNTABLESCHEMA
Layer 2
DBMS COLUMN
DBMS TABLE
DBMS SCHEMA
DBMS DEPENDENT (Oracle, DB2, Sybase, etc)DBMS DATA MODEL
“OPERATIONAL” DATA MODEL LAYER
(Domain of Database Administrators (DBAs))
Layer 3
APPLICATION VIEW COLUMN
APPLICATIONVIEW TABLE
BUSINESS INFORMATION
SYSTEM
BUSINESS APPLICATION SYSTEM“VIEW” DATA MODEL LAYER
(Domain of Application System
Managers (SMs and/or PMs)
Layer 4
Design Layers for a Business Information System Data Architecture
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
ISO 11179 BUSINESS CONTEXT INDEPENDENT DATA ELEMENT
REPRESENTATION
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
Layer 0
DoD CORE DATA ELEMENT METADATA
REPOSITORY
Implemented Model Layer
Operational DBMS Layer
Application View Layer
Specified Model Layer
ISO 11179 Model Layer
![Page 22: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/22.jpg)
22
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
ATTRIBUTE
COLUMN
DBMS COLUMN
ENTITYSUBJECT
TABLESCHEMA
DBMS TABLE DBMS SCHEMA
ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION
FUNCTIONALLY DEPENDENT & TECHNOLOGYINDEPENDENT DATA MODEL TEMPLATESATTRIBUTE INHERITS DATA ELEMENT
“SPECIFIED” DATA MODEL (DoD FDAd Domain)
TECHNOLGY DEPENDENT &DBMS INDEPENDENT MODEL / SCHEMA
COLUMN INHERITS ATTRIBUTE“IMPLEMENTED” DATA MODEL)
(Data Architects / Modelers Domain)
DBMS DEPENDENT &APPLICATION VIEW INDEPENDENT
DBMS COLUMN (Oracle, DB2, etc) INHERITS COLUMN“OPERATIONAL” DATA MODEL
(Domain of Database Administrators (DBAs))
VIEW
VIEW COLUMN
BUSINESS INFORMATION APPLICATION
SYSTEM
VIEWCOLUMN
STRUCTURE
VIEW COLUMNSTRUCTURE
PROCESS
VIEWCOLUMN
STRUCTURETYPE
APPLICATION VIEWS OF DBMS TABLES & COLUMNS
“VIEW” DATA MODEL(Domain of Application System
Managers (SMs and/or PMs)
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA
Data sharing occurs at the “operational and application” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.
METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model ISO 11179
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
![Page 23: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/23.jpg)
23
Supply ItemResourceQuantity
Supply ItemResourceQuantity
Supply ItemResourceQuantity
Supply ItemResourceQuantity
Supply ItemResourceQuantity Supply Item
ResourceQuantity
Functional/OrganizationalContext Dependent “Specified” Model
Army LogisticsManagement
Navy LogisticsManagement
Technology Dependent“Implemented” Model
ANSI SQL
ANSI SQL
Vendor DependentSQL DBMS
“Operational” Model
“Oracle” DBMS
“Sybase” DBMS
Business ApplicationInformation System (AIS)
“View” Model
Navy UADPS (AIS)
Army SAMS (AIS)
Concepts
Data Element Concept
ValueDomain
Data Element
View ColumnNames
DBMS ColumnNames
SQL ColumnNames
AttributeNames
Business Fact SemanticTemplate Name
Metadata Repository Architecture of RelatedRepresentations of DoD Enterprise Shared Data Elements
in Support of Data and Information Sharing
The quantity of each type ofFederal Supply System materielitem contained in an identifiableinventory of materiel objects.
Data Element Definition:
Additional Data Element Structural Metadata:
Supply ItemResourceQuantity
MaterielResource
Physical ItemBalance
Quantity
PhysicalMeasure
ConceptualValue Domain
Data type characteristics,local definition, enumerated values ( if specific ), etc.
ISO 11179 Context Inde pendentData Element Representation Meta Model
An Optimal Application of an ISO 11179 Based Data Element Architecture for Resolving Disparate Representations of Shared Enterprise Data Elements
Supply ItemResourceQuantity
Supply ItemResourceQuantity
METADATAREPOSITORY
Defense DataDictionary System(DoD Standardized Data Elements)
17000+ SDEs
![Page 24: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/24.jpg)
24
The “Optimal” Solution
![Page 25: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/25.jpg)
25
Mat_Itm_Inv_Qt
MaterielInventoryQuantity
Materiel ItemInventoryQuantity
Mat_Inv_Qty
Materiel UnitInventoryQuantity
Supply UnitQuantity
Stocked MaterielQuantity Ships Stores
Quantity
Functional/OrganizationalContext Dependent “Specified” Model
Army LogisticsManagement
Navy LogisticsManagement
Technology Dependent“Implemented” Model
ANSI SQL
ANSI SQL
Vendor DependentSQL DBMS
“Operational” Model
“Oracle” DBMS
“Sybase” DBMS
Business ApplicationInformation System (AIS)
“View” Model
Navy UADPS (AIS)
Army SAMS (AIS)
Concepts
Data Element Concept
ValueDomain
Data Element
CORE METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model
ISO 11179 Model
View ColumnNames
DBMS ColumnNames
SQL ColumnNames
AttributeNames
Business Fact SemanticTemplate Name
Metadata Repository Architecture of RelatedRepresentations of DoD Enterprise Shared Data Elements
in Support of Data and Information Sharing
The quantity of each type ofFederal Supply System materielitem contained in an identifiableinventory of materiel objects.
Data Element Definition:
Additional Data Element Structural Metadata:
Supply ItemResourceQuantity
MaterielResource
Physical ItemBalance
Quantity
PhysicalMeasure
ConceptualValue Domain
Data type characteristics,local definition, enumerated values ( if specific ), etc.
ISO 11179 Context Inde pendentData Element Representation Meta Model
Example Logistics Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
![Page 26: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/26.jpg)
26
Sail_Rat_Cde
SoldierRank Code
SailorRating Code
Sold_Rnk_Cd
Unit MemberRank Code
Squad MemberRank Code
Crew MemberRating Code Launch Team
MemberRating Code
Functional/OrganizationalContext Dependent “Specified” Model
Army PersonnelManagement
Navy PersonnelManagement
Technology Dependent“Implemented” Model
ANSI SQL
ANSI SQL
Vendor DependentSQL DBMS
“Operational” Model
“Oracle” DBMS
“Sybase” DBMS
Business ApplicationInformation System (AIS)
“View” Model
Navy (AIS)
Army (AIS)
Concepts
Data Element Concept
ValueDomain
Data Element
CORE METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model
ISO 11179 Model
View ColumnNames
DBMS ColumnNames
SQL ColumnNames
AttributeNames
Business Fact SemanticTemplate Name
Metadata Repository Architecture of RelatedRepresentations of DoD Enterprise Shared Data Elements
in Support of Data and Information Sharing
The code that represents the level ofauthority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level issubordinate to levels above andsuperior to levels below.
Data Element Definition:
Additional Data Element Structural Metadata:
Person Grade Code
HumanResource
Personnel Classifi- cation
Grade Code
PersonnelRankingMeasure
ConceptualValue Domain
Data type characteristics, etc.
ISO 11179 Context Inde pendentData Element Representation Meta Model
Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
![Page 27: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/27.jpg)
27
Part II:
Implementing theMetadata Architecture
For Enterprise-wide DataSharing in a Legacy System
Environment
![Page 28: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/28.jpg)
28
Table of Contents
1. A Realistic and Practical Approach to Achieve the Ideal Solution
2. How Do We Get to the Ideal?
3. Find Our Metadata
4. Perform Smart Meta-Data Mining
5. Find the Right Starting Layer
6. Reverse Engineer to Build the Upper Layers
7. Overall Forward Engineering Process
8. Process Statistics
9. Lessons Learned
![Page 29: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/29.jpg)
29
CONCEPT CONCEPT STRUCTURE
CONCEPT STRUCTURE TYPE
VALUEDOMAIN
VALUE DOMAINSTRUCTURE
VALUE DOMAINSTRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE TYPE
DATA ELEMENT CONCEPT STRUCTURE
DATA ELEMENT CONCEPT
DATAELEMENT
ATTRIBUTE
COLUMN
DBMS COLUMN
ENTITYSUBJECT
TABLESCHEMA
DBMS TABLE DBMS SCHEMA
ISO 11179BUSINESS CONTEXT INDEPENDENTDATA ELEMENT REPRESENTATION
FUNCTIONALLY DEPENDENT & TECHNOLOGYINDEPENDENT DATA MODEL TEMPLATESATTRIBUTE INHERITS DATA ELEMENT
“SPECIFIED” DATA MODEL (DoD FDAd Domain)
TECHNOLGY DEPENDENT &DBMS INDEPENDENT MODEL / SCHEMA
COLUMN INHERITS ATTRIBUTE“IMPLEMENTED” DATA MODEL)
(Data Architects / Modelers Domain)
DBMS DEPENDENT &APPLICATION VIEW INDEPENDENT
DBMS COLUMN (Oracle, DB2, etc) INHERITS COLUMN“OPERATIONAL” DATA MODEL
(Domain of Database Administrators (DBAs))
VIEW
VIEW COLUMN
BUSINESS INFORMATION APPLICATION
SYSTEM
VIEWCOLUMN
STRUCTURE
VIEW COLUMNSTRUCTURE
PROCESS
VIEWCOLUMN
STRUCTURETYPE
APPLICATION VIEWS OF DBMS TABLES & COLUMNS
“VIEW” DATA MODEL(Domain of Application System
Managers (SMs and/or PMs)
Functionally Independent Business Fact Semantic Templates (Globally Shared Data Elements)
(Domain of DoD Data Administration)
1.0 META MODEL ARCHITECTURE SUPPORTING ENTERPRISE WIDE SHARED DATA
Data sharing occurs at the “operational and application” view layers. Made possible throughthe relationships between all layers represented by metadata in a repository that enables relating syntax,structure, and semantics from any layer to a common ISO 11179 standard representation.
METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model ISO 11179
CONCEPTUAL VALUE DOMAIN
CONCEPTUAL VALUE DOMAIN STRUCTURE
CONCEPTUAL VALUE DOMAIN STRUCTURE TYPE
![Page 30: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/30.jpg)
30
Sail_Rat_Cde
SoldierRank Code
SailorRating Code
Sold_Rnk_Cd
Unit MemberRank Code
Squad MemberRank Code
Crew MemberRating Code Launch Team
MemberRating Code
Functional/OrganizationalContext Dependent “Specified” Model
Army PersonnelManagement
Navy PersonnelManagement
Technology Dependent“Implemented” Model
ANSI SQL
ANSI SQL
Vendor DependentSQL DBMS
“Operational” Model
“Oracle” DBMS
“Sybase” DBMS
Business ApplicationInformation System (AIS)
“View” Model
Navy (AIS)
Army (AIS)
Concepts
Data Element Concept
ValueDomain
Data Element
CORE METADATAREPOSITORY
Implemented Model
Operational DBMS
Application View
Specified Model
ISO 11179 Model
View ColumnNames
DBMS ColumnNames
SQL ColumnNames
AttributeNames
Business Fact SemanticTemplate Name
Metadata Repository Architecture of RelatedRepresentations of DoD Enterprise Shared Data Elements
in Support of Data and Information Sharing
The code that represents the level ofauthority and responsibility occupied by Person in a hierarchy of levels ranging from most superior to most subordinate in which each level issubordinate to levels above andsuperior to levels below.
Data Element Definition:
Additional Data Element Structural Metadata:
Person Grade Code
HumanResource
Personnel Classifi- cation
Grade Code
PersonnelRankingMeasure
ConceptualValue Domain
Data type characteristics, etc.
ISO 11179 Context Inde pendentData Element Representation Meta Model
1.1 Example Personnel Application of an ISO 11179 Based Data Element Architecture for Relating Disparate Representations of Shared Enterprise Data Elements
![Page 31: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/31.jpg)
31
2. How Do We Get to the Ideal?(or the least un-ideal)
• Find our metadata
• Perform smart metadata mining
• Pick the right starting layer
• Reverse engineer to build the upper layers
• Forward engineer to build standard-data based applications
![Page 32: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/32.jpg)
32
3. Find Our Metadata
• Existing schemas within running applications as that’s the only place where data-truth resides
• Extract Cobol FDs within running applications for the same truth reason
• Finally, research metadata libraries like ERwin models
![Page 33: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/33.jpg)
33
3.1 Where We Started
• DoD had 493 (Erwin) data models that were developed in the 1990s. There were 5709 tables and 16921 columns in these tables.
• We did not inventory each DoD Agency, but the key investigator (Hank Lavender) is very much aware of what, where, and how much all the schemas overlapped.
• This effort was to “prove the process”. We will soon start real Enterprise-wide data sharing projects.
![Page 34: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/34.jpg)
34
4. Perform Smart Meta-Data Mining
• Pick backbone and rib-cage (HR, Finance, Inventory Customer Management, Sales) Applications
• Pick the most commonly used schemas across the enterprise that support the backbone and rib-cage applications
• Pick the subset of schemas that have the most commonly used tables (note: commonly used is different from exactly the same as…)
• Make Where-Used and Frequency-Used Matrices
![Page 35: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/35.jpg)
35
4.1 Where Used & Frequency Matrices
IDM Data Model Counts Data Model Description Tables Columns Relationships
C-03 Budgets & Currency 56 178 53
C3-12 Command & Control 28 276 65
ES-07 Environmental Hazards 41 464 46
ES-08 Environmental Projects 28 185 40
LG-06 Transportation Operations 19 91 26
LG-23 Materiel Documentation 36 272 51
LG-28 Materiel Characteristics 45 225 48
PR-22 Training & Instruction 20 135 23
PR-31 Person Characteristics 36 118 41
Totals 309 1944* 393*542 Unique Data Element Concepts
Basic Types and Populations
![Page 36: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/36.jpg)
36
4.1 (cont) Subject Areas Use Across IDM Schemas
SDM IDM SchemasSubject Areas C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31
EnvironmentalManagement X X X
HealthManagement X X X X
LogisticsManagement X X X XLogisticsOperations X X X X X
LogisticsPlanning X X X
MaterielMaintenance X X
MaterielManagement X X X X X X X
TransportationOperations X X X
PropertyManagement X X X
PersonnelManagement X X X X X X
ManagementAdministration X X X
![Page 37: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/37.jpg)
37
4.1 (cont) Where Used & Frequency Matrices
Organization X X X X X X X
Person X X X X X X
Country X X X X X
Location X X
Task X X
Facility X X
Plan X X
Guidance X X
Geolocation X X X
SDM Subject Area Entity C-03 C3-12 ES-07 ES-08 LG-06 LG-23 LG-28 PR-22 PR-31
LogisticsManagementPersonnelManagementLogisticsOperations
LogisticsOperationsManagementAdministrationPropertyManagementLogisticsPlanningEnvironmentalManagement
PropertyManagement
IDM Schemas & Tables
Frequency of Use Matrix
![Page 38: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/38.jpg)
38
5. Find the Right Starting Layer
Data Modeling Layer Description Start Here ?
Data Elements Context independent business factsemantic templates
NO, these are not databasemodels and have no context
Specified Data Model Technology independent datamodel templates
NO, as these are just templates and not database models
Implemented Data Model
DBMS independent databasedata models and hosts fordatabase object classes
OK- if you have “Erwin” like data models that can beresearched, tabulated, andextracted via Excel orSQL DDL
Operational DataModel
DBMS dependent and OperatingSystem specific
YES! This is the best as itmatches the reality ofoperating databases andapplications
View Data ModelDatabase application specificSQL views
No, as this is tooapplication-use specific, and not data model centric.
![Page 39: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/39.jpg)
39
6. Reverse Engineer to Build The Upper Layers
• Import to appropriate layer
• Promote to higher data modeling layer
• Re-engineer the Specified Data Model layer
• Analyze to discover the Data Elements
• Build Data Element Model Metadata layer
![Page 40: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/40.jpg)
40
6.1 Import to Appropriate Layer
IDM
Importing SQL DDL
![Page 41: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/41.jpg)
41
6.1 Import to Appropriate Layer
IDM Tables
IDM
![Page 42: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/42.jpg)
42
6.2 Promote to Higher DataModeling Layer
Promote IDM to SDM
![Page 43: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/43.jpg)
43
6.2 Key Promotion Issues
Key Difference Between Subject-Entity-Attribute vs Schema-Table-ColumnModel of a subject area. Intellectual boundaries, notdata processing boundaries. Not a conceptual versionof a logical database. It’s a subject based data modeltemplate. Define once, use many times, differently inIDM models.
Subject-Entity-Attribute (SDM)
Model of a database schema that may involve attributesfrom multiple entities in one table, or attributes ofentities across multiple tables. Intended to be implemented within a DBMS as an operational database.
Schema-Table-Column (IDM)
Subject-SchemaNot related. This would then mean TransformationalRelationship.
Entity-Table Not related. This would mean TransformationalRelationship
Attribute-ColumnYes, Related. This allows define once, use many timesmodeling.
![Page 44: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/44.jpg)
44
• Assign Entities to different Subjects• Reassign Entities to within Entities (sub-typing)• Reassign Attribute’s Semantics• Conform Attribute Names to Subject Area Scope• Reassign Attributes to different Entities• Reassign Attributes to different Data Elements
6.3 Re-engineer the SpecifiedData Model
Reassign Entity to Subject
SDM
![Page 45: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/45.jpg)
45
6.3 Re-engineering the SpecifiedData Model
Reassign Entity to Subject Assign Attribute Meta Category Values
Reassign Attribute to Data Element Reassign Attribute to Entity
BASIC PROCESSES SDM
![Page 46: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/46.jpg)
46
6.4 Reallocate Foreign Keys toEncapsulate Subject’s Entities
For Each Subject Area:
• Make a List of Entities
• Make a Subject Area Based E-R Model Diagram
• Delete Unnecessary Foreign Keys from Existing Entities
• Make New Foreign Keys Where Needed
• Export to E-R Diagrammer to Verify Result
• Recycle if Necessary
SDM
![Page 47: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/47.jpg)
47
SDM
6.4 (cont) Reallocate Foreign Keys toEncapsulate Subject’s Entities
Validate/Create SDM Foreign Keys
![Page 48: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/48.jpg)
48
SDM
6.4 (cont) Reallocate Foreign Keys toEncapsulate Subject’s Entities
Modify SDM Foreign Keys
![Page 49: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/49.jpg)
49
6.5 Discover the Data Element
Promote SDM Attributes to Data Elements
![Page 50: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/50.jpg)
50
6.6 Build Data Element ModelLevel Metadata
![Page 51: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/51.jpg)
51
6.7 Data Element, Attribute,Column Differences
Key Differences Among Data Element, Attribute, Column
Characteristic Data Element Attribute Column
Context
Reason forexistence
Frequency ofuse
Example
Stand-alone independentbusiness fact template.
A characteristic of anentity that exists withinthe context of a subject.
A characteristic of atable that exists withinthe context of a schema.
Source of semantics andcommon general meaningfor classes of attributesand columns.
Source of value basedrefinement of the intentof the entity. The set ofall attributes fully definean instance of an entity.
Source of value basedrefinement of the intent of the table. Theset of all columns fullydefine an instance ofa table.
Defined once within thecontext of an entity.source of business factsacross one or more columns within one ormore tables.
Define once, use manytimes to provide semanticsto attributes or columns.
Defined once withinthe context of a table.
Identifier Asset IdentifierPerson IdentifierCustomer Identifier
Invoice Line Item• Part Number• Salesman Identifier• Customer Identifier
![Page 52: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/52.jpg)
52
7.0 Overall Forward Engineering Process
• Import from higher level to lower level
• Map IDM to ODM legacy schemas to preserve existing systems environment and/or Generate new ODM schemas to replace legacy systems
• SQL Views can support legacy names or new names
• Generate Application
![Page 53: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/53.jpg)
53
7.1 Import From Higher LevelTo Lower Level
Subject Area Data Model to Implemented Data Model
• Start Metabase IDM• Make the Target Schema• Pick an SDM Subject• Select the Root Entity• Create the Data Model Entity Tree• Perform Import (from SDM to IDM)• “Prune” Schema-Table Set to Just Those Needed• “Prune” Table-Column Set to Just Those Needed• Move Columns Among Tables as Needed• Import Next SDM Model and Perform “Pruning” Steps
• Mapping to New IDM from SDM Preserved-----Of Course!
![Page 54: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/54.jpg)
54
7.1 (cont) Import From Higher LevelTo Lower Level
Subject Area Data Model to Implemented Data Model
Import SDM Entities to IDM Tables
![Page 55: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/55.jpg)
55
Implemented Data Model to Operational Data Model
• Start Metabase ODM• Make the Target DBMS Schema• Pick an IDM Schema• Select the Root Table• Create the Data Model Table Tree• Perform Import (from IDM to ODM)• “Prune” DBMS Schema-DBMS Table Set to Just Those Needed• “Prune” DBMS Table-DBMS Column Set to Just Those Needed• Move DBMS Columns Among DBMS Tables as Needed• Import Next IDM Model and Perform “Pruning” Steps
• Mapping to New ODM from IDM Preserved-----Of Course!
7.1 (cont) Import From Higher LevelTo Lower Level
![Page 56: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/56.jpg)
56
Implemented Data Model to Operational Data Model
7.1 (cont) Import From Higher LevelTo Lower Level
Import IDM Schema Tables to ODM DBMS Tables
![Page 57: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/57.jpg)
57
ODM
7.2 Generate SQL
Generate DBMS SQL DDL
![Page 58: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/58.jpg)
58
7.3 Generate Application
ODM
![Page 59: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/59.jpg)
59
Task Name Notes Effort Metric
1 Find the Right Starting Point
With MS/Access based repository of data models. Close to about 100 models
2 days
2 Import to appropriate layer
Had to fix a number of data modeling errors in source CASE tool
40 hours for 10+ data models
3 Promote to Higher Data Modeling Layer
Required several cycles of distilling subjects
40 hours for 10+ models
4 Re-Engineer Had to re-engineer Fkeys, rename entities and some attributes. Also had to reconnect new attributes to “old columns.”
240 hours for 10+ models
8. Process Statistics
![Page 60: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/60.jpg)
60
Task Name Notes Effort Metric
5 Abstract to Data Element
Required review of each attribute, and creation of MCVs, etc.
0.25 hours per attribute for 1000 attributes. So, 250 hours.
6 Build Data Element Model Level Metadata
Required generation of higher level concepts, value domains, etc.
80 hours
7 Import from higher level to lower level
Required design of new data models for new databases from data model templates, and/or just re-mapping to existing models
8 hours per model for 10 existing models and for 2 new models. Thus, 100 hours
8 Generate SQL Required specification of data types and lengths
20 columns per hour for 80 hours.
9 Input to and thenGenerate Application
Export of one Model from IDM 1 hour to export,1 hour to generate1st cut application
8. (cont) Process Statistics
![Page 61: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/61.jpg)
61
9. Lessons Learned
• It can be done. However, it is not a walk in the park!
• It requires clear understanding of separation of Data Models. Data Element from Specified DMs, from Implemented DMs, and from Operational DMs. These are NOT transformations (conceptual to logical to physical). These are different data models.
• Subject Matter Experts are Essential, Critical, and Absolutely Necessary.
• It’s not top down. It’s bottom-up. But once built, use it top-down.
• You must have a metadata repository and data modeling tool that works at the enterprise level, and not just at the database or data model level.
• We made changes to the metadata repository system along the way. So, being able to change the meta model, entry and update and reports, is essential.
• Given that Entity reuse for just these ODS models was about 4x, the value for the data model template reuse in data warehouses and data marts is incalculable.
![Page 62: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/62.jpg)
62
THANK YOU
Michael M. GormanFounder and President
2008 Althea LaneBowie, Maryland 20716-1518Phone: +1.301.249.1142Fax: +1.301.249.8955Email: [email protected]: <http://www.wiscorp.com>
Hank LavenderSenior Information Engineer
1310 Braddock PlaceAlexandria, Virginia 22314-1648Phone: +1.703.836.5900Fax: +1.703.836.8691Email: [email protected]: <http://www.amerind.com>
nI c
![Page 63: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/63.jpg)
63
Back-ups
![Page 64: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/64.jpg)
64
DoD CORE DATA ELEMENT METADATA
REPOSITORY
Implemented Model Layer
Operational DBMS Layer
Application View Layer
Specified Model Layer
ISO 11179 Model Layer
DoDEnterprise
InteroperabilityMetadata Repository
Core DoD Enterprise
Data ElementMetadata
Repository
Proposed DoD Metadata Repository
Complete Representationsof Data Element Metadata
Data Element Metadata Relationshipsto Multiple Categories of Metadata
Today
The Future
The Payoff
Seamless & TransparentInformation Interoperability
![Page 65: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/65.jpg)
65
Mat_Itm_Inv_Qt
MaterielInventoryQuantity
Materiel ItemInventoryQuantity
Mat_Inv_Qty
Materiel UnitInventoryQuantity
Supply UnitQuantity
Stocked MaterielQuantity Ships Stores
Quantity
Functional/OrganizationalContext Dependent “Specified” Model
Army LogisticsManagement
Navy LogisticsManagement
Technology Dependent“Implemented” Model
ANSI SQL
ANSI SQL
Vendor DependentSQL DBMS
“Operational” Model
“Oracle” DBMS
“Sybase” DBMS
Business ApplicationInformation System (AIS)
“View” Model
Navy UADPS (AIS)
Army SAMS (AIS)
View ColumnNames
DBMS ColumnNames
SQL ColumnNames
AttributeNames
The quantity of each type ofFederal Supply System materiel item contained in an identifiable inventory of materiel objects.
Data Element Definition:
Additional Data Element Structural Metadata:
Data type characteristics,local definition, numerated values ( if specific), etc.
The Current DoD Architecture for Defining Standard Data Element Representations of Shared Enterprise Data Elements
METADATAREPOSITORY
Defense DataDictionary System(DoD Standardized Data Elements)
16000+ SDEs Business Rule: Only one named representation is permitted to exist in the repository as an Enterprise SDE.
(SDE Access Name)
![Page 66: 1 A Metadata Architecture For Enterprise-Wide Data Sharing](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e5d5503460f94b555cc/html5/thumbnails/66.jpg)
66
METABASEREPOSITORY
Implemented Data Model (IDM)
Specified Data Model (SDM)
ISO 11179 Data Element Templates
Operational DBMS Model (ODM) SQL DBMS
ODMLAYER
XML Schema Info Tables
OUTPUT ODM LAYER
HUMAN RESOURCESDATABASE
Output – XML Wrapped Metadata
Output Schema Information
Feedback
DoDGlobal Information
Grid (GIG)