national information exchange model (niem) building a niem iepd: the niem iepd life cycle and tools...

National Information Exchange Model (NIEM)

Building a NIEM IEPD: the NIEM IEPD Life Cycle

and Tools

6 September 2006

John Wandeltand

Mark Kindl

Georgia Tech Research Institute

2

NIEM Builds on GJXDM

NIEM = GJXDM + Expanded Scope + Lessons Learned

• Incremental changes to GJXDM to address specific issues and expanded scope

• A few examples:– Expanded Domains (Intel, International Trade, Emergency Mgt,

Infrastructure Protection, etc.)– Better Modularity– Improve Reusability - Leveraging work of others – Discovery of IEPDs

and components– Capturing Business Context– Incorporating feedback into the model - Model Refinement through use– Support for components from external standards– IEPD consistency

3

NIEM Use Cases

• To effectively exchange information, there must be a common semantic understanding of data among participants, and the data must be formatted in a consistent manner.

• NIEM provides reusable components and IEPDs for exchanges• Between NIEM domains (inter-domain).

• Within NIEM domains (intra-domain) (where multiple sub communities exist within a single NIEM domain, e.g. justice).

4

IEPDLife

Cycle

0 Scenarios,BRM taxonomies

2 Map +model

3 Build +validate

4 Assembleto IEPD spec

1 Info exchgrequirements

5 Publish +implement

IEPD Development vs the Model

IEPD Library(virtual)

NIEM

The Model – NIEM reusablecomponents

IEPD Development• National priorities (e.g. NDEx, TWPDES)• Authoritative sources (e.g. Rap Sheet)• Local requirements

NIEMNDR

reuse /revise

add /replace

reuse

add /replace

5

NIEM Naming and Design Rules

• Provides a normative technical reference for building NIEM data model components and IEPDs.

• Draft NIEM NDR will be available for review and comment with the 1.0RC release.• Specifies some of the following:

– Use of types, elements, attributes, groups– Enumerations (code lists)– Annotations (documentation and appInfo)– Namespaces– ISO 11179 naming– External standards use (e.g. Geospatial)– Creating new entities (objects)

• Composition• Specialization w/ inheritance• Associations (relationships)• Roles

– Adding properties to existing entities• Type augmentation• Metadata

6

NIEM CoreNIEM Core

UniversalUniversal

CommonCommon

StructuresStructures

NIEM Architecture (Model View)

NIEM

CodeLists..

Externaldomains or standards

External domains or standards

External Domain 2External Domain 2

Y

Wrap / Refer

Y

NIEM DomainsNIEM Domains

ScSc

EMEM

ImIm

IPIPITIT

JJ IntelIntel

ExternalDomain 3ExternalDomain 3

Z

(a) Translate / Map

(b) Extract / Use

ZNZN

Create (boot strap)

Justice (GJXDM)Justice (GJXDM)

Justice specific

Total Migration(special case)

External Domain 1External Domain 1

XN

Adopt

candidate

7

What is a NIEM IEPD?

Documentation and specification for describing

NIEM information exchanges.

Contains:• A set of NIEM conforming XML schemas that

– (Re)use or correctly extend NIEM components.– Define a class of XML exchange instances.

• Documentation detailing– How to implement the IEP with the schemas.– Exchange model, business rules, use cases, etc.– Example instances, stylesheets

• Metadata for indexing, search, discovery, maintenance, registration, etc.

8

NIEM XML Reference Architecture

Exchange xmlExchange xml

*Exchange xsd*Exchange xsd

*Extension xsd*Extension xsd

*Code listxsd

Namespace

* Optional

XML Schema

NIEM xsd subsetNIEM xsd subset *Constraint xsd*Constraint xsd

IEPDs have their own target

namespaces

IEPDs have their own target

namespaces

ConstraintValidationConstraintValidation

ConformanceValidation

ConformanceValidation

9

NIEM

IEPD Component Perspective

IEPDIEPD

SubsetSubset

External 2External 2External 3External 3

1

1

22

2

3

3

3

NIEM IEPD schemas are NIEM conforming.

ExtensionExtension External 1External 1External 0External 0

66544

External domains or standardsExternal domains or standards

1, 5 – create2, 4 – wrap / refer3 – translate / map6 – adopt

10

The IEPD Life Cycle

• General framework for building IEPDs in logical steps.• Primary process from the practitioner perspective.• Guide to understanding how IEPDs are ideally built

and published. • Not prescriptive, IEPD builder …

– Adjusts the scope of the framework to the IEPD level of effort. – Enters the life cycle at appropriate step.

• The value of formal specifications for input/outputs of the IEPD Life Cycle. – Ensure consistency across IEPDs.– Capture business context to facilitate search and discovery

of NIEM components and IEPDs.– Machine readable and process-able to automate support

for the IEPD Life Cycle.– Encourage and facilitate commercial tool development

and value added capabilities.

NIEMNDR

NIEMNIEM

NIEM IEPDLife Cycle

0 Scenarios,BRM Taxonomy


2 Map +model

2 Map +model

3 Build +validate

3 Build +validate

4 Assembleto IEPD spec4 Assemble

to IEPD spec

1 Info exchangerequirements


5 Publish +implement5 Publish +implement

Executive orders, mission statements, business requirements, business context, policies/procedures


ExistingIEPD artifacts


Submission of candidate

NIEM components, component business context



Specific information exchange, high-level business

requirements, existing IEPDs



IEPD artifacts to share/reuse,

metadata for discovery



Gap analysis, specs for new componentsGap analysis, specs for new components

Valid IEPD schemas, example instances, documentation, metadata


Data requirements, business context,exchange model


IEPD for reuse library


NIEMNDR

Complete IEPDComplete IEPD

NIEM IEPDLife Cycle



2 Map +model

2 Map +model

3 Build +validate

3 Build +validate

4 Assembleto IEPD spec4 Assemble

to IEPD spec



5 Publish +implement5 Publish +implement

















Gap analysis, specs for new componentsGap analysis, specs for new components







NIEMNDR

NIEMNDR

NIEMNIEM

Complete IEPDComplete IEPD

13

IEPD Life Cycle Feeds the Data Model Maturity Life Cycle

NIEM

Staging AreaNIEM Governance

NBAC / NTAC:vet, harmonize,conformance

Groups with common data interests

Early Awareness& coordination

IEPDs

New component candidates & business context

New component candidates & business context

NIEM IEPDLife Cycle





3 Build +validate

2 Map +model

Topdown

Bottomup

14

From the IEPD Life Cycle Perspective

GJXDM Tools• Justice Information Exchange Model (JIEM) Tool• JIEM-GJXDM direct mapping capability (through SSGT)• Schema Subset Generation Tool (SSGT)• Moving forward• GJXDM Spreadsheet• IEPD Clearinghouse

NIEM Tools• Component Mapping Template (CMT)• Graphical Browser • Schema Subset Generation Tool (SSGT)• Information Exchange Package Documentation (IEPD) Tool• Toward NIEM 1.0RC1 (30 Sep 2006)

GJXDM Tools

16

Justice Information Exchange Model(JIEM) Modeling Tool

• JIEM is a Web application that facilitates analysis of business context and content (payload) requirements for justice information exchanges.

• The analysis is represented in a site database.

17

JIEM Exchanges and Dimensions

A site database defines exchanges in 5 dimensions: process, event, agency, condition, document (information).

18

JIEM Reference Model

• There is a special site database called the JIEM Reference Model, which defines exchanges common to most jurisdictions.

• A site database can be created from scratch or copied from another site database, or the Site Database Builder can be used to create one from the JIEM Reference Model. Using the Site Database Builder typically reduces the effort of site database creation by about 75%.

19

JIEM Data Elements

A site database includes data elements, which can be simple or may be composed of other data elements.

20

JIEM Data Element Usage

Documents contain data elements and define their typical usage.

21

JIEM Reports

• A variety of textual and graphic reports can be generated.• Reports generally relate to content of the database or usage of data

(comprehension or conceptualization of the data, data admin, etc.)

22

How JIEM and SSGT Interoperate

Schema Subset Generation Tool (SSGT)

JIEM SSGT Mapping Tool

ArgoUML

Data exchange requirements (local vocabulary) output in XMI

(i.e. UML-XML)

Data exchange requirements (local vocabulary) output in XMI

(i.e. UML-XML)

.XMI

23

SSGT Mapping Tool

Enables user that has built an AS-IS or TO-BE model with JIEM to map local data components directly to GJXDM data components through the SSGT.


24

SSGT Mapping Tool – Search / Discover

• User local data components are displayed (left) with convenient access to the SSGT (right).

• User can search with SSGT for GJXDM data components that match local data components.

Schema Subset Generation Tool (SSGT) Schema Subset Generation Tool (SSGT)

25

SSGT Mapping Tool – Select / Build

User maps by – selecting matching GJXDM data components (check box). – confirming (clicking) the match to build the map.

Schema Subset Generation Tool (SSGT) Schema Subset Generation Tool (SSGT)

26

SSGT Mapping Tool – Generate Subset and Report

When all local data components have been reviewed and mapped or not, the Mapping Tool can generate:

1. Subset for mapped elements (also contains the wantlist).2. Report detailing what mapped and what did not.


27

How JIEM and SSGT interoperate



.XMI

ArgoUML

Wantlist

Mapping Report

SSGT

28

Schema Subset Generation Tool(SSGT)

• SSGT is a Web application that allows a user to browse the GJXDM and create GJXDM schema subsets and their wantlists.

• Supports basic and advanced search for GJXDM components.

29

SSGT – Expand aType or Property

• A complex type can be expanded to see its properties.• A property can be expanded to see its type’s properties.

30

SSGT – Types

User clicks a type to see its: – definition– base types– derived types– properties– properties that it is the type of

31

SSGT – Properties

User clicks a property to see its: – definition – types in which it’s contained– its types.

32

SSGT – SelectProperties and Types

Properties and types can be selected for inclusion in the GJXDM schema subset.

33

SSGT – Generate Subset.zip

Generating a GJXDM schema subset, creates a .zip file containing both the subset and and its corresponding wantlist.

34

SSGT – Save and Load Wantlists

Wantlists can be saved and loaded so that work can be suspended and resumed.

35

How JIEM and SSGT interoperate



.XMI

ArgoUML

Wantlist

Mapping Report

SSGT

Subset.zip

36

Moving Forward

• Export other dimensions beyond information dimension in standard open specification (e.g. BPMN, etc.).

• Evaluate applicability to other domains and NIEM.

• Capture additional metadata (security, privacy, etc.) to support the implementation of an SOA.

• Support generation of extension and constraint schemas.

• Support sharing/reuse of mappings, wantlists, and schemas across JIEM exchanges and site DBs.

• Support sharing/reuse with other non-JIEM users through standard import/export specifications (the NIEM IEPD specification is well suited for this).

37

GJXDM Spreadsheet

38

GJXDM Spreadsheet

39

GJXDM IEPD Clearinghouse

40

GJXDM IEPD Clearinghouse

41

GJXDM Tool Summary

• Justice Information Exchange Model (JIEM) Tool:– Define/model exchange context and content requirements.

– Generate XMI (UML-XML) representation of local data exchange requirements.

• GJXDM Spreadsheet:– Off-line textual representation of GJXDM (the model and dictionary).

– Search, navigate, browse, discover GJXDM data components.

• Schema Subset Generation Tool (SSGT):– On-line textual representation of GJXDM (the model and dictionary).

– Search, navigate, browse, discover GJXDM data components.

– Select and map GJXDM data components to local components (Mapping Tool).

– Generate mapping report spreadsheet (Mapping Tool).

– Select data components and generate schema subset and associated Wantlist.

• Manually produce other artifacts required for an IEPD. – Examples: exchange, extension, and constraint schemas, and example instances.

• IEPD Clearinghouse (CH):– Register and share a GJXDM IEPD.

42

IEPDLife Cycle


2 Map +model

3 Build +validate




JIEM

SSGT

IEPD CH

Spreadsheet

SSGT Map

Available Tools: GJXDM

SSGT

NIEM Tools

44

Component Mapping Template (CMT)

A spreadsheet for mapping local data elements to NIEM elements.• Developed to expand GJXDM to NIEM by identifying data requirement gaps.• Developed as submission format for NIEM pilots.• Reflects experience gained from working with key GJXDM customers.• Structures the process of using NIEM for exchange content requirements.• Identifies new data components for addition to NIEM.• Structured format allows for automated processing.• Structured method to capture:

– Local data requirements (vocabulary) and map them to existing NIEM data components

• exact matches• partial mapping details • no mapping

– Information that can provide a metric for modeling refinement and improvement– Information that can be used for keywords, search hints, mapping assistance, etc.

45

CMT – General Tab

General information on exchange(s) being mapped (business context)

46

CMT – Template Tab

Information on how local components are mapped to NIEM

47

CMT – Codes Tab

Information on how codes are mapped to NIEM

48

NIEM Spreadsheet

49

NIEM Tools for Browse, Navigate, Search, Discover, Generate/Build

Graphical BrowserSchema Subset Generation Tool

IEPD Tool

50

NIEM Graphical Browser

Java-based GUI to browse, navigate, and generally familiarize with NIEM structure.

8 differentstarting locations

3 different views

51

NIEM Graphical Browser –Starting Points and Views

• User may start from one of 8 different locations in NIEM (others can be added). • Select from 3 different views of the NIEM:

(1) Data model, (2) Inheritance, (3) Associations

• The 3 views always correlated to the same location in the model.

Data Model (types + properties)

Inheritance

Associations

52

NIEM Graphical Browser –Entities and Relationships

• Model is displayed as a set of nodes and edges corresponding to entities and fundamental data modeling relationships between them: – parent_of (IS-A)– contains (HAS-A)– type_of

• Double-clicking an entity (node) moves focus to that entity (by centering and highlighting) and displays all entities and relationships to/from it.

53

NIEM Graphical Browser – Relationships

Relationships (edges) are directional; to expose a relationship label, hover the cursor over its corresponding edge.

54

NIEM Graphical Browser – Screen Management

When an entity has >10 relationships, only the first 10 entities are exposed; next 10 can be seen by clicking the “next 10” button (lower right).

55

NIEM Graphical Browser – Focus

The entity of focus can be dragged around to expose entities that may be outside the window.

56

NIEM Schema Subset Generation Tool (SSGT)

• A repository of all NIEM data components with their associated metadata linked as a consistent, cohesive data model

• Functions: search, find, and select components for, build and generate a subset schema for an IEPD.

• Linkage of data components is key to the tool’s ability to provide model navigation and efficient generation of schema subsets.

57

NIEM SSGT – Search

Allows both simple and advanced search for data components using various means and criteria including:

– Type of component (property, type, facet, external, or association)– With string, substrings, wildcard, exact/partial, etc.– Within component name or definition or both

58

NIEM SSGT – Select / Add

• User finds and selects components required by checking boxes to build a NIEM subset schema.

• A property component can be added to a subset – as a content bearing element or as a reference to content bearing element.

– as an element within its Subject type or as an independent element.

59

NIEM SSGT – Dependencies

• SSGT adds all other components that the user requirements depend on.

• User may delete components from the subset only if he added them.

• Components that user requirements depend on can only be deleted by SSGT (when corresponding user requirements have been deleted).

60

NIEM SSGT – Subset Schema

When user is done adding components, SSGT can automatically generate a complete NIEM NDR conformant subset schema for a given version of NIEM.

Wantlist

Subset.zip

61

NIEM SSGT – Wantlist

• A Wantlist is an open spec XML file representing user requirements (without dependencies) and the state of an SSGT session.

• A Wantlist can be saved and reloaded later to resume work or to reconstitute or modify a subset schema.

62

NIEM IEPD Tool (IEPDT)

• Enables user to easily assemble, document, generate, and share complete or partial IEPD per NIEM IEPD specification.

• User registers for account and corresponding work space.

63

NIEM IEPDT – Entering an IEPD

User creates an IEPD through a 5 step process:1. Upload artifact files (optional)2. Enter basic IEPD metadata.3. Enter additional IEPD metadata.4. Enter authoritative source metadata.5. Verify all information and create the IEPD.

64

NIEM IEPDT –Validation and Review

• Validates package completeness against NIEM IEPD content specification and reports results (does not have to pass validation to share).

• Provides a review of metadata and artifact files.

65

NIEM IEPDT –Auto Format and Assembly

• Formats metadata file per NIEM IEPD spec (metadata.xml).• Formats catalog file of artifacts per NIEM IEPD spec (catalog.xhtml).• Assembles IEPD as compressed archive (filename.zip).• Installs IEPD in user work space and provides option to share.

IEPD Catalog.html

IEPD Catalog.html(source view)

xhtml

74

NIEM IEPDT – Download / Upload

• Download IEPD as single, compressed, self-contained archive (<filename>.zip).• Upload IEPD archive file (created manually or by IEPDT per IEPD spec).• IEPDs created manually or by any other tool can be uploaded by IEPDT.

75

NIEM Tool Summary

• Component Mapping Template (CMT): – Map off-line data requirements for content submissions to NIEM.

– Capture information – metrics, business context, POC data, mapping statistics, etc.

• Graphical Browser– Visually navigate and familiarize with NIEM.

• Schema Subset Generation Tool (SSGT): – Repository of NIEM data components and metadata.

– Textually navigate, search, discover NIEM data components.

– Build and generate subset schemas for NIEM IEPDs.

• Information Exchange Package Documentation (IEPD) Tool: – User work space – assemble, document, download, upload

IEPDs per NIEM IEPD specification.

– Share complete IEPDs or their artifacts.

76

IEPDLife Cycle

2 Map +model

3 Build +validate




SSGT

CMT

Graph BrwsrIEPDT

Spreadsheet

IEPDTSSGT

Available Tools: NIEM


77

IEPDLife Cycle

2 Map +model

3 Build +validate




JIEM

SSGT

SSGT

CMT

Graph Brwsr

IEPD CH

IEPDT

Spreadsheet

Spreadsheet

SSGT Map

IEPDT

GJXDMNIEM

Available Tools: NIEM + GJXDM

SSGT

SSGT

+


78

NIEM 1.0RC1 (30 Sep 2006)Refinements and Improvements

• URI for each NIEM data component. – A universally unique identifier; can be a URL that resolves to a documentation page defining semantics, structure, and relationships with other data components.

• Selective Sharing of IEPD artifacts and metadata. – IEPD Tool will allow user to selectively share parts of an IEPD (rather than simply the entire IEPD or nothing).

• Support the building of a new IEPD from an existing IEPD. – Capability to build IEPDs directly by selecting components to be used within the IEPD, generate, and optionally register/share.

• Improved integration between IEPD Tool, Graphic Browser, and SSGT. – Intent is to remove extra manual steps between tools (e.g., download SSGT subset and upload to IEPD Tool).

• Version support for IEPDs. – For bug fixes and changes; will allow for multiple versions of the base IEPD to exist to avoid possible conflicts. Track version history and change log in IEPD metadata.

• Basic reporting feature. – usage statistics and management metrics, for example: components used most often in IEPDs, common search terms, commonly selected results, etc.

• Export NIEM as a MS Access database

Discussion & Questions