peter doolan, gvp & chief technologist, oracle public sector david webber, information...

<Insert Picture Here>

Peter Doolan, GVP & Chief Technologist, Oracle Public Sector

David Webber, Information Architect, Oracle Public Sector

IJIS Industry Summer Briefing – July 2011

NIEMTest Model Data

Deploy Requirements

Build Exchange

GenerateDictionary

Exchange Development

Using NIEM EIEM Dictionaries forBetter Information Alignment

Copyright ©2011, Oracle. All rights reserved. Oracle Draft Materials – Limited Circulation

Overview

• The NIEM NTAC has introduced the use of Enterprise

Information Exchange Models (EIEM) and dictionaries

• How can your organization leverage these, what are the

technical advantages, business benefits and challenges?

• Understand lessons learned from existing experience

creating EIEM dictionaries and models and what are the

resources and assets needed to do this for an enterprise.

• Creating IEPD exchanges using EIEM assets can enable

rapid leveraging of NIEM for organizations; however,

managing the process and stakeholders for the EIEM then

provides additional needs.


Introduction

• What is the big picture of EIEM for federal enterprises?

• How does government acquire and manage these assets?

• How can you leverage your existing infrastructure?

• What roles do semantics and rules play?

• EIEM collaboration and alignment

• Lessons learned and examples

• What tools are available?

• Initial support provided in GTRI dictionary server

• CAM toolkit dictionary support


EIEM – THE BIG PICTURE

Transitioning from NIEM past/present to NIEM future


NIEM Past and Present Challenges

• Vertical domain vocabulary for DHS and DOJ

• One dozen domain dictionaries

• Technology limited - built using XSD schema

• Core components highly contextual to DHS/DOJ

• Surprisingly – no actual logical models of information!

• Gap with semantic technologies integration

• Limited data content rules and code lists

• Mapping automation support missing

• Multi-year development life cycles


NIEM Future?

• Abstract dictionary representations with strong

semantics, rules and code lists support

• Schema, models, documentation are generated from

dictionary automatically

• Contextual mechanisms allow linear number of true

core components – not exponential growth

• Easier for practitioners to leverage with more

predictable and repeatable results

• Enable domain use horizontally across government

• ETL mapping automation support

• Continuous collaboration environment


EIEM – EXPLORED

Understanding EIEM today


Why do we need EIEM dictionaries?

• Allows domains to manage their components libraries• Provides consistency for project development teams

• Sets of NIEM consistent XML exchange components• Aligned to enterprise data stores• Optimized for reuse and interoperability

• Save time and effort across the enterprise• Perennial question for developers – when should I use NIEM

components, and when our own local ones?• Are there components already available for that purpose?

• Provide formal mechanisms and procedures to share components and collaborate across SDLC process

• Provide external parties consistent data views


How to create EIEM dictionaries

• Top down - directly from information models

• Enterprise modelling tools

• SQL database physical table definitions

• Bottom up

• Harvesting from existing data assets

• Harvesting from XML exchange structures and schema

• Using neutral abstract standards based representations


Conceptual Information Model

Child(BBIE)Item

Child(BBIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Follows

Naming and Design Rule (NDR) principles and guidelines

Canonical Components

DictionaryXML

Canonical Components

DictionaryXML

ebXML CCTS terms (ABIE, BBIE, ASBIE)

Parent = Aggregate Business Information Entity

Child = Basic Business Information Entity

Attribute = Association Business Information Entity

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item

Parent (ABIE)Item . . . . .

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Child(BBIE)Item

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

* CCTS – Core Components Technical Specification

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Attribute(ASBIE)

Each compound component

Each atomic component

Optional attributes of component


Example – Person Name

• Person Name (ABIE)• Verified Details? (ASBIE)

• First Name (BBIE)• Middle Name (BBIE)• Last Name (BBIE)

• Previous Name? (ASBIE)• Language Code (BBIE)

Language Code may exist independently of Person Name

Verified Details and Previous Name are flags that denote additional information about the entity they are associated with

There are three component items aspects:

structure relationships; content rules; definitions


Top Down - ICE EIEM dictionary generation

ICE Enterprise Data Model (EDM)

OASIS CAM

template

OASIS CAM

template

Model Components

XSD schema

Model Components

XSD schema

ERwin Refactoring,Compress,

Renaming ToolNIEM NDR

Evaluations

ERwin Refactoring,Compress,

Renaming ToolNIEM NDR

Evaluations

GenerateStandard

Components Dictionary

XML

GenerateStandard

Components Dictionary

XML

55

Canonical Dictionary

of ICE exchange

components

XML

+

XSD

XML

+

XSD

33

Analyst Review

ebXML CCTS compatible (ABIE, BBIE, ASBIE)

22

EleEle

DefDef

DDLDDL

Production Rules

44

11

Completed CAM

template

Completed CAM

template

Excel Spreadsheet

66

77

BENEFITS: Will reduce ICE development time and effort

Pre-built NIEM consistent XML components

XML consistent with ICE enterprise SQL data

Removes scope for definition errors

Maximizes reuse potential across ICE

Open public standards based approach

Open technology infrastructure

Code

Lists

Code

ListsCode

Lists

Code

Lists

ERwinClient

ERwin model

Write

ApplyImportExport


Renaming Tool

• Expands abbreviations (has built-in lists with context)• Corrects common typos and spelling errors (200+)• Adjusts names to camel case with no separators (_)• Retains acronyms• Adds name suffix according to NIEM representation terms

and type of content, e.g. date, number, code, etc• Optionally refactor rules by their type for interoperability;

make nillable optional; relax DB length restrictions• Can optionally insert namespace prefix• Can be configured by editing the lists in: word-utils.xsl


Bottom Up – OASIS/NIST EML

Option 2 – Derive from existing exchange XSD schemaImport each XSD and merge into CAM dictionary

Exchange XSD

schema

Exchange XSD

schemaExchange XSD

schema

Exchange XSD

schemaExchange XSD

schema

Exchange XSD

schema

CAM template

CAM templateCAM

templateCAM

templateOASIS CAM

template

OASIS CAM

template

NDR Evaluation,Refactor,Renaming

Tool

NDR Evaluation,Refactor,Renaming

Tool

33

Merge & Generate

DictionaryXML

Merge & Generate

DictionaryXML

44

XMLXMLImport

Import

ImportDictionary of

canonical components

22

Analyst Review

ebXML CCTS compatible (ABIE, BBIE, ASBIE)

11

EML Core XSD

schema

EML Core XSD

schema

OASIS CIQ schema

OASIS CIQ schema

OASIS GIS schema

OASIS GIS schema

EML – Election Markup Language

State and Local voting management systems data standard

Generate model

diagramsUML/XMI

Mind maps

Generate model

diagramsUML/XMI

Mind maps

55


N-DEx Dictionary and Model

(Harvested from N-DEx schema – ndexia.xsd and ndexibp2.xsd)

Freemind Interactive ModelDictionary viewer with N-Dex components


Leveraging EIEM dictionaries

• Where can we use EIEM assets?

• What formats do we need to use?

• Are there applicable standards / guidelines?


Visual Designer with Dictionary Collection

Drag n’ Drop Tool

Drag n’ Drop Tool

Industry dictionaryIndustry dictionaryDomain dictionaryDomain dictionary

Component Definitions

44

Component Definitions

Search

Tools

Search

Tools

22

Exchange

Designer

Exchange

Designer

11

Insert

Dictionary

Parent

Components

Insert

Dictionary

Parent

Components

33

Completed Exchange Template

Completed Exchange Template

55

Collection


Alien Transfer Manifest – Mock Up

Technical Note: Inline child elements for ice: components shown for illustrative purposes only; actually are type reuse references (paperclip icon on parent node).

dictionariesdictionaries

XMLXML

XMLXML

XMLXML

• ICE EDM dictionary

• NIEM Immigration dictionary

• EID extensions dictionary

• ICE EDM dictionary

• NIEM Immigration dictionary

• EID extensions dictionary

ICE components referenced

EID pre-defined collections

based on NIEM + ICE + new

Immigration component

Namespaces of dictionary components

Definitions stored as syntax

neutral canonical XML

NIEM components

Collection


Challenges and Opportunities

• Learning curve for application development teams

• Teams used to manually crafting XML Schema and Java code

in developer XML IDE tools

• Paradigm shift

• Integration into SDLC and IEPD processes

• Integration with existing SOA stack environments

• Currently working with SAIC and State Department

• Automatic generation of WSDL and Java code assets

• Unit test XML instance generation

• XML test framework and validation services


Vision: Top Down Exchange Assembly

Canonical XML Components DictionaryCanonical XML Components Dictionary11

Component Associations and CouplingsComponent Associations and Couplings22

Exchange Templates and RulesExchange Templates and Rules33

W3C Schema and Model RepresentationsW3C Schema and Model Representations44

Delivery Control, Messaging, SecurityDelivery Control, Messaging, Security55

Presentation

Collaboration S

ervices

Artefact relationships

Implementation Artifacts and ExamplesImplementation Artifacts and Examples66

Navigation and Query

Tools

Domains Reuse Library

Components

Automatic Rendering Tools

Deployment Environments and Middleware

XML

User Inserts

Relationship Lookups

TemplateCatalog

Testing Workbench, Rules Engine, Data Samples, Integration ETL

SchemaXML

Models Components


Exchange Delivery Lifecycle

Monitor

Deploy

Design

Author

Test

Component Dictionary

CAM

Template

Compiled

Templates

XSD Schema

XML Samples

XMI / UML Models

Production

Log Results

Template

Updates

Documentation

Drag and Drop Visual Editor


htt

p:/

/ww

w.c

am

ed

ito

r.o

rg

CAM Toolkit


CAM V2.0 Release Features

• Dictionary “Drag and Drop” of components to build exchanges

• Enhanced visual editor and performance - 3x faster, smaller

memory footprint

• Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit

• Mind map models – integration with Freemind tool

• UML/XMI export – import to UML tools / ArgoUML

• Code list table lookups – rule based and selectable

• This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema.

23 NIEM Team - Public Sector Sales


Available XML Dictionaries

• LEXS 3.1.4 dictionary• OASIS EDXL dictionary• OASIS EML dictionary

• NIEM 2.1 dictionaries• CBRN dictionary• Emergency dictionary• Family dictionary• Immigration dictionary• Infrastructure dictionary• Intelligence dictionary• Justice dictionary• Maritime dictionary• Screening dictionary• Trade dictionary• Immigration blueprint• NIEM core dictionary

Available from download site

direct link:http://sourceforge.net/projects/camprocessor/files

XMLXMLXMLXML

XMLXMLXMLXML

XMLXMLXMLXML

+ includes spreadsheets and sample models

Packaged with CAM editor

see dictionary folder of install

+ spreadsheet

+ sample control files

XMLXMLXMLXML

XMLXML

Note: Those marked in bold are model style dictionaries with recursive components.


EIEM Next Steps

• Align CAM dictionary tools with GTRI dictionary server

• Integrate to other modelling tools – not just ERwin• Provide extended reporting tools for analysis across

dictionary of components and between dictionary collections – report on • Inconsistency• Common components• Extensions

peter doolan, gvp & chief technologist, oracle public sector david webber, information...

Documents

asbie parent

niem components

niem eiem dictionaries

eiem assets

niem future slide

eiem collaboration

bbie middle

bbie previous