niem and xml for architects and developers

<Insert Picture Here>

Delivering NIEM-aligned data services

David Webber, Information Architect, Oracle Public Sector

NIEM and XML

for Architects

and Developers

The following is intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Disclaimer Notice

Slide 2

© 2012/13 Oracle Corporation – Restricted Distribution 3

Agenda

•Background - What are the business drivers and needs?•NIEM today – Technical Introduction•The Delivery Life Cycle – Lessons Learned•Software Delivery - Anatomy of a NIEM “IEPD”•Tooling to the Rescue!•Key Pillar of Dictionaries•Summary

BACKGROUNDBusiness Drivers and Needs


XML is everywhere…

•PMIX – Prescription Drug information sharing•Elections Results Reporting•Pharmacy and Vaccines locating•COBie Buildings Information sharing•PESC – Student Post-Secondary Education•City 311 Services•Vehicle spare parts inventory•Public Safety information sharing•Healthcare services delivery coordination•eDevice control templates


Public Sector Challenges

• Cost of building information exchanges

– Reduction of development sunk costs through minimizing coding tasks

– Allowing rapid prototyping and proof of concepts (agile development)

– Flexibility - dynamic adaptable templates instead of rigid fixed code

– Reuse – templates for common systems / solutions easily shared and adapted

• Government transparency and public information sharing

– Structured content from data stores – e.g. election results reporting, monthly cost

reports, scheduled road maintenance; obvious metadata and semantics

– Simple search requests on data – e.g. part numbers, codes, licenses lookup

– Not intended for unstructured and textual content document delivery

• Practical real world data sharing anywhere

– Direct simple tools with short learning curve and plug and play deployment


Technology Capability Requirements

• Delivering on the Open Data Digital Strategy Vision for government

• Open Standards and Open Source based

• Plug and play with code-free templates and rapid development

• Supports NIEM and Open XML exchanges

• Leverages deployed technology today

• Workflow integration

• Secure messaging delivery and partner management

• Delivered across-platform and device

• End-to-end security & governance

NIEM TODAYTechnical Introduction

The NIEM FrameworkNIEM connects communities of people who share a common need to exchange information in order to advance their missions, and provides a foundation for seamless information exchange between federal, state, local, and tribal agencies. Much more than a data model, NIEM offers an active user community as well as a technical and support framework.

Support FrameworkTechnical FrameworkCommunity

Formal Governance Processes

Online Repositories

Mission-Oriented Domains

Self-Managing Domain Stewards

Data Model

XML Design Rules

Development Methodology

Predefined Deliverables (IEPD)

Tools for Development and Discovery

Established Training Program

Implementation Support

Help Desk & Knowledge Center

The NIEM Data Model

NIEM’s data model is a set of common, controlled, and approved XML data structures and definitions vetted through the Federal, State, Local, Tribal and Private Sectors.

Data elements are organized into core and domain-specific components

Core components are used by multiple

domains and can be described by structure,

semantics, and definition universally

Domain-specific components are

continually updated by subject matter experts that are actual NIEM

participants and industry experts for

their particular domain

NIEM Naming and Design Rules (NDR) specify how each of

these components are defined and utilized

NIEM’s Domains View

NEWEST DOMAINS

NIEM Government Resource

Management Domain

NIEM Health Domain

NIEM Human Services Domain

DOMAINS IN PLANNING

NIEM Agriculture

NIEM Education

NIEM Transportation

11


Available XML Dictionaries

•NIEM 2.1 dictionaries– CBRN dictionary– Emergency dictionary– Family dictionary– Immigration dictionary– Infrastructure dictionary– Intelligence dictionary– Justice dictionary– Maritime dictionary– Screening dictionary– Trade dictionary– Immigration blueprint– NIEM core dictionary

Available from download site

direct link:http://sourceforge.net/projects/camprocessor/files

XMLXML XMLXML XMLXML XMLXML XMLXML XMLXML

+ includes spreadsheets and sample models

Note: Those marked in bold are model style dictionaries with recursive components.


NIEM uses W3C Schema Collection

XSD Collection

*Code list xsd

Namespace* OptionalXML Schema

xsd subsetxsd

subset *Constraint xsd

*Constraint xsd

*Extension xsd

*Extension xsd

*Restriction xsd

*Restriction xsd

Industry Domain

schemas

Code list xsd

LEGEND:

Exchange xsdExchange xsd

NIEM Exchange Schema


NIEM Past and Present Challenges

•Vertical domain vocabulary for DHS and DOJ

•One dozen domain dictionaries

•Technology limited - built using XSD schema

•Core components highly contextual to DHS/DOJ

•Surprisingly – no actual logical models of information!

•Gap with semantic technologies integration

•Limited data content rules and code lists

•Mapping automation support missing

•Multi-year development life cycles


NIEM is still improving…

•Collections of complex XML Schema•Verbose components•Embedded context in names•Currently facing significant scaling challenges

– Inconsistencies; too much manual management; slow lifecycles

•Dictionary technology incubating•Enhanced code lists mechanism incubating•UML profile is evolving initiative with OMG

– Attempt to marry modelling techniques and XSD Schema syntax

•Policy and Security mechanisms incubating•Need to embrace enabling tools for rapid expansion of

domains and community collaboration

DELIVERY LIFECYCLE

How to build NIEM solutions

Lessons learned

Foundation Pillars


The 8 “D”s and NIEM

•Design•Develop•Deploy•Document•Dictionaries•Discovery•Differentiate•Diagnose

Repeatable, Reusable Process(Exchange Specification Lifecycle)

NIEM IEPD Process

*IEPD - Information Exchange Package Documentation

©2012 Oracle Corporation 18

NIEM IEPD Reports

DictionariesDiscovery

NIEM IEPD / Exchange Delivery Lifecycle

Diagnose

Differentiate

Design

Develop

Validated

Templates /

Schema

XSD Schema

XML Samples

XMI / UML Models

Production

Results

Requirements

Updates

Documentation

Drag and Drop Visual

Designer

DocumentDeploy

Exchange

Templates 1

2

3

4

5

Required IEPD artifacts


Integrating into your existing processes and control• Critical to manage and steer use of NIEM

• Ensure business outcomes match requirements• Lessons learned with NIEM

• The earlier in the software development process that NIEM is introduced the better the overall outcomes

• Each step has deliverables – business plan, delivery architecture, data model, alignment reports, schema, test results that can be monitored in your SDLC*

• Monitoring products reinforces reuse as best practice• Tools essential in review processes

• Collaboration tools can coordinate artifacts across projects and participants

• Test bed tools and open APIs reduce integration alignment costs *SDLC – Software Development Life Cycle

SOFTWARE DELIVERY

Anatomy of a NIEM “IEPD”

It is not just all about the XSD schema


IEPD Components & Requirements

<Exchange_Schema/>

<Extension_Schema/>

<Subset_Schema/>

IEPD IEM

IEPD MPD

NIEM Core Schema(s)

Domain Schema(s)

Main Document

Catalog

Change Log

Sample XML Instance

In order to be NIEM-conformant, the IEPD must adhere to:1. NIEM Conformance Document

2. NIEM Naming and Design Rules (NDR) v1.3

3. NIEM Model Package Description (MPD) Specification v1.0

Formal NIEM IEPD Packaging Details

IEPD Artefact Description

Exchange Files

Subset Schema (SSGT) Subset of the full NIEM schema—a compressed directory of schemas to distinguish from other schema sets.

Wantlist User requirements -distinguishes user-required data components from components they depend on for conformance. Generated by and up-loaded to the Schema Subset Generation Tool (SSGT). This is an open specification and the SSGT is not required to create a wantlist, though it is easier.

Exchange Schema Base document schema that defines the XML root element and is generally named after the IEPD itself. Also known as the document schema, reference schema, or root schema.

Constraint Schema Constraints for separate constraint validation path—a compressed directory of schemas to distinguish from other schema sets.

Extension Schema Specification for extended components—separate local name-spaces of components not contained in NIEM.

Sample XML Instance Example instance(s) – may reference optional stylesheet.

Stylesheet (generic) Example stylesheet for display of instance(s).

Documentation

Master Documentation The Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirements

Exchange model XMI Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling Language (UML) model.

Business Rules Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3) Schematron or other formal business rule language, or (4) generated by a development tool.

Mapping to NIEM Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to NIEM components ; preferably the Component Mapping Tool (CMT).

Extended Components Components created because they were not in NIEM—may be part of mapping spreadsheet and include structure and definitions of new components. Prefer CMT.

Change log Record of cumulative changes from previous IEPD versions. The initial IEPD simple records its creation date.

Catalog

Catalog XML file A machine-readable list of artifacts in the IEPD in an open, portable and browser-displayable format (simple manual edit).

Metadata XML file All metadata of owner and domain to be registered with the IEPD (simple manual edit).

Generated using CAM

Never Used

Word Document

Manual Entry


XML Technology ComponentsNIEM: need to manage complete aspects of exchange delivery, not just the structure and content data model


Swim Lane View

UML Technologies XML TechnologiesW3C Schema

This is what NIEM exchanges are delivered with on-the-wire!

UMLXMI

XSD Namespace

EnumerationsXSLT

XQueryXPath

XML

Patterns

Associations

CardinalityStereotypes

InheritanceFacets

Elements

Attributes

ContentComplex Types

Audience:

Data Modellers / Designers

Audience:

Software Developers and Middleware Engineers

Audience:

General content delivery and applications; content engineers; rules representation systems; multimedia delivery; geospatial systems

Objects

Web

HTML

JavaScript

JSON

REST

SQLSOAP

Audience:

Web application developers

Open-XDX

24

TOOLING TO THE RESCUEAccelerating and Improving the NIEM process


IEPD Delivery Tasks / Tools Matrix

CAM CAMVOpen-XDX

(OPX)Exchange Schema Development X

Canonical Model Development / Generation X

Exchange Schema Generation X

NIEM Compliance Checking / Reporting X

WSDL / JAXB Binding Generation X

Operation Message Test Case Generation X

Production Services Message Generation X

SQL Schema Mapping X X

NIEM IEPD Artifact Generation X

- Cross-reference spreadsheet X

- UML model X

- Business Rules Report X

- XSD schema package X

Conformance Test Suite X

Run time XML instance validation X

In order to be NIEM-conformant, the IEPD must adhere to:

1. NIEM Conformance Document

2. NIEM Naming and Design Rules (NDR) v1.3

3. NIEM Model Package Description (MPD) Specification v1.0


CAM Toolkit for NIEM

•Concepts behind the Content Assembly Mechanism

approach

•Understanding the steps involved in top down /

bottom up exchange design, testing and delivery

•Applicability to NIEM and IEPD / MPD delivery

* NIEM – National Information Exchange Model

* IEPD – Information Exchange Package Documentation

*MPD – Model Package Description


CAM Templates – WYSIWYG Exchange

Structure Rules

Documentation

CAM Template

http://www.oasis-open.org/committees/cam


INGESTINGEST

CAM and W3C Schema Wizards

EXPORTEXPORT

XSLT tools

process

entire XSD

collection

XSLT tools

process

entire XSD

collectionXSD Collection

Structure

Rules

Documentation*Code list xsd

Namespace* OptionalXML Schema

xsd subsetxsd

subset*Constraint

xsd*Constraint

xsd

*Extension xsd

*Extension xsd

*Restriction xsd

*Restriction xsd

Industry Domain

schemas

Code list xsd

LEGEND:

Exchange xsdExchange xsd

Exchange Schema

CAM Template

htt

p:/

/ww

w.c

amed

ito

r.o

rgCAM Editor – Main Screens


CAM Toolkit Features

• Dictionary “Drag and Drop” of components to build exchanges

• Visual editor and structure editing

• Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit

• Mind map models – integration with Freemind tool

• UML/XMI export – import to UML tools / ArgoUML

• Code list table lookups – rule based and selectable

• This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema.

31


XML Exchange Package Artifacts

•Template Evaluation Report

– Naming and Design Rules assessment

– Interoperability factors review

– Spell checking

•Comparison to Dictionary

– Reuse report, cross-reference spreadsheet, subset “wantlist”

•Generate Schema Bundle with layout styles (e.g. NIEM)

•Realistic test samples generation

•Testing with runtime validation engine (CAMV)

•Business information exchange documentation

•Models – XMI and mind maps


CAM Template Evaluation Report

•The purpose of the CAM template evaluation report is

to provide a suite of checks against a template that

are difficult, tedious or otherwise to determine by

manual inspection of a template structure and its

associated rules and annotation details

•The report covers the following areas:– alerts users to potential issues with their template and

provides suggestions for improving the functional details

– addresses logical and conceptual issues that simple syntax

checking alone does not flag

– can be used as part of a formal review process to ensure a

baseline for quality control


CAM Toolkit Capabilities


Online Tutorials – www.niemtrainingvideos.org

KEY PILLAR OF DICTIONARIESIntroduction and Concepts


The Semantic Framework and Vision

XML/NIEM Components in

Neutral Dictionary

Representation

UML Technologies

XML Technologies

W3C Schema Tools

Semantic Representations

Rules Technologies

HTML 5 rendering tools

Security solutions

Middleware solutions


Dictionaries: Key Pillar of Information Sharing

Canonical XML Components DictionaryCanonical XML Components Dictionary11

Component Associations and CouplingsComponent Associations and Couplings22

Exchange Templates and RulesExchange Templates and Rules33

W3C Schema and Model RepresentationsW3C Schema and Model Representations44

Delivery Control, Messaging, SecurityDelivery Control, Messaging, Security55

Presentation

Collaboration S

ervices

Artefact relationships

Implementation Artifacts and ExamplesImplementation Artifacts and Examples66

Navigation and Query

Tools

Domains Reuse Library

Components

Automatic Rendering Tools

Deployment Environments and Middleware

XML

User Inserts

Relationship Lookups

TemplateCatalog

Testing Workbench, Rules Engine, Data Samples, Integration ETL

SchemaXML

Models Components

Canonical Dictionary Collections


Dictionaries Semantics Approach• Abstract dictionary representations with strong

semantics, rules and code lists support

• Schema, models, documentation are generated from

dictionary definitions automatically

• Contextual mechanisms allow linear number of true

core components – not exponential growth

• Easier for practitioners to leverage with more

predictable and repeatable results

• Enable domain use horizontally across government

• ETL mapping automation support

• Continuous collaboration environment


Developing domain dictionaries (EIEM)

• Allows domains to manage their components libraries• Provides consistency for project development teams

• Sets of NIEM consistent XML exchange components• Aligned to enterprise data stores• Optimized for reuse and interoperability

• Save time and effort across the enterprise• Perennial question for developers – when should I use NIEM

components, and when our own local ones?• Are there components already available for that purpose?

• Provide formal mechanisms and procedures to share components and collaborate across SDLC process

• Provide external parties consistent data views

*EIEM – Enterprise Information Exchange Model


EIEM/BIEC

• Business Information Exchange Components (BIEC)• NIEM-conforming XML schema data

component definition that meets a particular recurring business requirement for an enterprise

• Enterprise Information Exchange Model (EIEM)• NIEM-conforming schemas that

define data components to be reused in IEPDs developed by an enterprise; collection of enterprise BIECs organized into a subset and one or more extension schemas


Managed Dictionaries – Authoritative Sources

Slide 42

Centralized

Distributedschema

schema

schema

SQL

SFIS Values Library Service, NIEM, Metadata Registry

AL

LO

WE

D V

AL

UE

Sschema

Managed Dictionaries

Publish

SUMMARYNext Steps


Why CAM and NIEM?

“Can you give us a brief overview of how you expect IT professionals and line of business managers to use CAM*?”

– Van Ristau, CTO DLT Inc.

*CAM editor toolkit – http://www.cameditor.org and OASIS Content Assembly Mechanism standard – http://docs.oasis-open.org/cam

http://www.cameditor.org/

http://docs.oasis-open.org/cam


Response

•Line of business managers can use CAM to drive better

alignment to NIEM and faster more consistent and

predictable delivery of NIEM information exchanges

•CAM provides tools that allow business analysts to work

collaboratively with software developers to build exchanges

•Teams use CAM to perform the 8 “D’s” tasks when

implementing NIEM exchanges

•CAM automates the delivery of key XML artifacts for NIEM

exchange services

•XML validation can be deployed using the CAMV engine

•Open-XDX provides rapid flexible open data web services


Public Sector Open Data Services DeliveryOpen Platform Solutions

• Visual Editor

• SQL Browser

• Dictionaries

• Drag and Drop

• Query/Response

• Generate Data

• Validate Data

• Send Data

• JAX-WS

• soapUI

• CAM Editor

• Content Models

• OER Integration

• Web Forms UI

Open Data Test Bed Services Dictionaries

DB

Web

Open XDX

CAMV

Open-XDX / CAM Editor

Dictionary

Security

SQL

XML

XML

SQL

Repository

• CAM Editor

• Templates

• Policy Rules

• XACML

Templates

XACML


Technology Components and Features

•Providing open data sharing– Open-XDX component accelerates deployment of interfaces

•Aligning meaning and definitions– Collaboration using XML component dictionaries and NIEM

•Formal information verification and content rules– CAMV rules engine provides runtime handling– Industry domain test bed facilities– Rules documentation and sharing

•Business process integration– Oracle BPM and Forms combined with CAMV and Open-XDX

• In progress – Security profiles and policy templates– Upcoming release of CAM Editor and XACML integration


CAMV interactive

testing

Cloud Services Deployment

Open-XDX web

services

XML Technologies

W3C Schema Tools

Semantic Representations

Rules Technologies

Security solutions

Middleware solutions

CAMV test bed for

industries

Registry for Dictionaries

Security Profile

Templates

Web site and collaboration tools

Policy Rules

CAMeditor.ORG Project StatisticsSNAPSHOT OF

PROJECT ACTIVITIES

135,000+ CAMeditor.org page

visits to site

165+ countries have downloaded tools; 30% of visitors are

from U.S.;800+ downloads

weekly

2000+ student views of online video training

resources

8 languages now available

www.cameditor.orgwww.niemtrainingvideos.org

niem and xml for architects and developers

Technology

niem past

niem frameworkniem

public information

oracle public sectorniem

open xml exchanges

niem iepd tooling

information architect

data stores