DataShare

Kate Fitta NYC Mayor’s Office of the Criminal Justice Coordinator

Jonathan Natarajan NYC Department of Technology and Telecommunications (DoITT)

Doug Mamroth iWay Software

New York City’s

Enterprise

Integration

Platform

Agenda

• DataShare Overview – Progress to Date

• DataShare Architecture

• iWay Software the Role of Adapter Technology

• DataShare - Future Phases

Kate Fitta

NYC Mayor’s Office of

the Criminal Justice Coordinator (CJC)

DataShare Overview – Progress to Date

DataShare Overview

The Mayor’s Office of the Criminal Justice Coordinator (CJC) and the Department of Technology and Telecommunications (DoITT) partnered in early 2004 to deliver DataShare 2.0 – the replacement for a mainframe system utilized by criminal justice agencies (DataShare 1.0) and NYC’s new integration platform.

DataShare was designed for citywide integration, but deployed initially for 17 agencies (city, state, not-for-profits).

The Public Safety Portal, which is the user-facing dimension of the City’s integration solution for criminal justice agencies, includes both administrative and business user functions.

The initial phase concluded in July 2006 with the deployment of the Public Safety Portal and a set of GJXDM exchanges.

This undertaking was a Mayoral initiative, but agencies did not have a mandate to participate

Phase 1 – Facts & Figures

12 months for requirements gathering, solution analysis, systems integration procurement and solution selection

18 months to complete implementation – 2 major deployments

17 city, state and nonprofit agencies integrated via hub-and-spoke architecture

35 FTEs comprised the dedicated project team – City staff and consultants combined

60+ agency-side resources brought to bear during 1st phase

$12M spent on professional services, HW, SW

100% of post prod support provided by City of New York upon Phase 1 completion

Data Exchanges/Services

Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal

Criminal justice data that is shared between and among participating agencies via the hub and portal

The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare

Agency Participation

Public Safety Portal Functionality and Links to Applications

Criminal Justice Integration Platform (Hub)

Phase 1 Providers• NYPD• NYC DOC• Criminal Justice Agency (CJA)• OCA

Phase 1 Receivers• NYPD• Department of Probation• Department of Correction• District Attorneys (5)• Special Narcotics Prosecutor• CJA• Law Department• Dept. of Juvenile Justice (DJJ)• OCA• Legal Aid Society• Assigned Counsel Plan• HRA• DoITT (on behalf of CJIS)

The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT

Core Hub Features & Functionality• Push functionality• Pull functionality• Query application• Notification application

Phase 1 Business Deliverables• NYPD arrest data push (legacy & GJXDM)• NYPD arrest notification (GJXDM)• CJA ROR interview push and pull (GJXDM)• OCA supreme and criminal court actions push (legacy & GJXDM)• DOC inmate information (legacy & GJXDM)• DOC release notification (GJXDM)• DOC jail status query (GJXDM)• CJA NYSID – SSN match (legacy & GJXDM)

Core Portal Features & Functionality• Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data• System administration for agencies • Reduced sign-on (RSO) for DNA HITS application• e911 application link • webCRIMS application link from OCA• e-Justice portal link from DCJS

Phase 1 – Scope and Deliverables

• Decisions about DataShare architecture needed to consider the autonomy of agencies

• DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases

• DataShare was intended to serve as a pass-through system, with minimal data stored centrally

• The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies.

• The City decided to adopt GJXDM as its transport standard for criminal justice services

Guiding Principles & Strategic Decisions

GJXDM Strategy & Approach

• City decided to adopt GJXDM for all future criminal justice exchanges

• City decided to implement both legacy and GJXDM versions of existing exchanges via new platform to facilitate agency transition and minimize risk

• GJXDM SME consultants worked with provider and receiver agencies to develop schemas and leverage national model as appropriate.

• 30 representatives from 14 agencies attended GJXDM training in 2005

• City-State collaboration a priority for 2007

• GJXDM workgroup taking shape among stakeholder agencies to drive strategy going forward, especially collaboration with state and national initiatives; DOP, NYPD, DOC to play key leadership roles.

• City will post IEPDs for Phase 1 exchanges this fall and continue to stay coordinated with national efforts

Public Safety Portal Services

DOC Jail Status Query – This function enables users to enter a NYSID, book and case number or a combination of first name, last name and date-of-birth via the Public Safety Portal and retrieve the incarceration status and history for the individual(s) that match the criteria. Jail status data includes information about the inmate's current incarceration status, incarceration history, warrant history and DOC housing history from 1999 - present.

CJA Interview Pull – This function enables users to enter a docket number or arrest number via the Public Safety Portal and retrieve the details of the corresponding interview the Criminal Justice Agency conducted with the defendant prior to their criminal court arraignment. Interview data includes information about the defendant's employment, residence, education and criminal history and is only available for interviews conducted from 2005 - present.

Subscriptions for NYPD Arrest Notification (Basic) - This function enables business users to enter a NYSID, name information or location information via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the individual or location specified by the criteria.

Subscriptions for NYPD Arrest Notification (Advanced) - This function enables administrators to enter criteria corresponding to up to 150 arrest fields via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the specified criteria.

Subscriptions for DOC Release Notification - This function enables users to enter a NYSID or arrest number or book & case number via the Public Safety Portal and subscribe to be notified via email when the corresponding inmate is released from or admitted to City correction custody.

Links to Applications - DNA HITS, eJustice, e911, WebCRIMS

Public Safety Portal Services

New Exchanges

CJA Interview Push - The CJA Interview Push is a new, GJXDM data exchange that was first deployed through DataShare 2.0 in July 2006. In conducting interviews with criminal court defendants prior to arraignment, the Criminal Justice Agency collects information about the defendant's employment, residence, education and criminal history and submits it as a report to judges to help them make their bail decisions. CJA sends this information to DataShare 2.0 post-arraignment as a single daily batch (Tuesday – Saturday at 2am EST) of all interviews.

NYPD Arrest Push - The NYPD Arrest Push is a new, GJXDM version of a legacy exchange that had available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains information about arrest incidents on a near real-time basis.

DOC Inmate Push -The DOC Inmate Tracking Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains data about DOC's entire inmate population: basic assessment information gathered at intake, information about inmate movement among DOC facilities, and a record of discharge. The DOC Inmate Tracking Push is a daily batch provided 7 days/week (2am - 6am EST).

OCA Court Actions Push - The OCA Criminal Court Action Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains criminal case information from OCA's CRIMS system including court dates, disposition, sentence information, judge name, and final information about case charges. The OCA Criminal Court Action Push is a daily batch provided 7 days/week (12am - 2am EST).

CJA NYSID/SSN Push - The CJA NYSID/SSN Mapping Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. The exchange was first deployed through DataShare 2.0 in July 2006 and contains a mapping of defendants' NYSID, arrest number, and social security number. CJA sends the NYSID/SSN Mapping Push to DataShare 2.0 as a weekly batch (Wednesdays at 7am EST).

New GJXDM Exchanges

Juvenile Case Management System Law Department Dept. of Probation Dept. of Juvenile Justice (DJJ)

Richmond County DA

Richmond County DA

Public Safety PortalProviders

Special Narcotics Prosecutor

Queens County DA

Kings County DA

Bronx County DA

Legal Aid Society

Manhattan DA

Criminal Justice Agency

Dept. of Corrections

NYPD

18B Web (Assigned Counsel)

Office Court Administrator

DataShare – Current Landscape

ReceiversPublic Safety Portal

Richmond County DA

DataShare

DataShare Architecture

Jon Natarajan

NYC Department of

Technology & Telecommunications (DoITT)

Information Sharing Capabilities

The DataShare & Public Safety Portal solution facilitates the sharing of information across city agencies providing business process integration for the participating agencies.

DataShare will provide the following services related to enterprise applications integration (EAI)– Business process routing (via publish/subscribe)– Data standardization– Web-based portal that can offer truly distributed business

applications – Business process auditing capabilities– Field level filtering for messages – Security

– 24/7 Monitoring

Existing Systems Installed Base

MVS VAX/VMS LINIX UNIX Microsoft

FTP SMTP CICS Transactions Flat Files

DB2 ADABASE DATACOM VSAM ADMINS/VAX RDB CACHE MUMPS SQL SERVER ACCESS ORACLE EXCEL

Standards-based (J2EE, LDAP, SQL)

Enterprise Service Bus (WebSphere MQ and Broker)

GJXDM (iWay Transformation)

Public Safety Portal (Vingette & BEA WebLogic )

Involve the agencies that know the data

Involve the agencies that know the processes

Use existing City application implementation standards

Meeting The Needs – New Environment

Technical Architecture Requirements

Security – Authentication, Authorization, Encryption, LDAP

Performance – Handle 100’s of messages/second

Scalability – Able to add new data exchanges easily

Availability – 99.9% availability as ‘crime takes no holiday’

Extensibility – Expose functionality via different methods like Web services

Standards – J2EE, LDAP, GJXDM

DataShare: The Solution

The Solution consists of 6 key components

Criminal Justice End Users

Vignette Application Portal

J2EE Applications

BEA Weblogic Application Server

NotificationsWebSphere Everyplace

Access Server

WebSphereMessage Broker

(Enterprise Service Bus)

JMS-MQ

MQ

DataShare Data Base

RDBMS File Drop

Agency Adapter Platform

WebSphere MQ

iWay Adapter

MQ

RDBMS File Drop

Agency Adapter Platform

WebSphere MQ

iWay Adapter

MQ

Agency A Agency B

Mail ServerSMTP

Novell eDirectory

SSL

SSL SSL

SSL

Architecture & Components

Enterprise Service Bus (DataShare 2.0 integration message broker)

Integration layer provides message brokerage/routing and integration business services for citywide agencies.

Public Safety Portal Portal provides front-end Web applications (such as notifications

and queries), user administration, and reports to the agency user community.

Notification Services The DataShare 2.0 notification services provide the Public Safety

Portal users and the criminal justice agency backend systems to subscribe to key transactions traversing the Enterprise Service Bus as notifications.

Agency Adapter Platform Agency iWay adapters and MQ used as the gateway to and from

the criminal justice agencies and the enterprise service bus hosted at DoITT

Database DataShare 2.0 database is used to store application and

transactional metadata.

Novell eDirectory (DoITT’s enterprise directory server) Enterprise directory server stores citywide agency users and

groups. The eDirectory server will be used to provide services such as distributed provisioning, authentication and authorization.

Architecture & Components

• Agencies opted for one of two delivery methods:– Flat File– Database

• Agencies that opted for flat file delivery then chose the method for inserting data in the file:– Populate data elements in the XML format, which preserves

intact the format and contents of the XML– Populate data elements following a comma separated value

(CSV) format or other appropriate convention

• Agencies that opted for database delivery then chose the method for inserting data into the database:– Insert the entire XML as is in the database (e.g. as a character

large object or CLOB), which preserves intact the format and contents of the XML

– Convert the XML elements into columns in a relational or flat table and insert each XML record as a row in the table

GJXDM Adaptor Approaches at Agencies

DataShare Leverages DOITT’s CityNet

CityNet

Adapter Platform· Staging · Queue· Monitoring· Adapter

Providing System

WBI Message Broker

Adapter Platform· Queue· Monitoring· Adapter

Configuration Database

Public Safety Portal

Adapter Platform· Staging · Queue· Monitoring· Adapter

Receiving System

Directory

DoITT

Providing Agency

Receiving Agency

iWay Software & the Role of Adaptor Technology

Doug Mamroth

iWay Software

Enables connectivity to the integration server from applications, application environments, and/or databases to an integration server platform

Provides the flexibility to integrate with the presentation business logic database

Exposes agencies systems for exchanging data in both directions

Provides an architecture that enables separate applications to work together

Allows agency applications to be enhanced, added or removed without affecting others

The Role of Global Justice XML

What iWay Initially Saw In DataShare

Each Agency needed to provide the ability to: Make requests (“Pull”) – Services oriented

This requires Request/Reply

Exchange data (“Push”) – Data centricThis requires Data send/receive

Take part in process – Process centricThis requires Event listener/emitter

The combination of the above is the EAI It demanded a new form of B2B Protocol not commonly available - GJXML

Talk to the adapter using a variety of services (SOAP, FTP, HTTP, . . . ) not the system

Transforms information from an existing format to GJXDM and publishes as a Web Service

Creates WSDL in the process Stores all services in a common directory GJXDM built around an B2B EDI model

Transaction Type Standard Elements Provides “Wiggle Room” Best Implemented Using Trading Partner Agreements

iWay’s Approach – Adaptive Framework

Trading Partners

Every Agency Is Different Different Agencies Want Different Versions (GJXDM 3.0, 3.1,

NIEM) Different Agencies Have All Of The Data (In Complete Records) Different Agencies Have Different Security Requirements Different Agencies Have Different Privacy Policies Different Agencies Have Different Backend Systems Different Errors Must Be Handled In Different Ways

Thousands Of Messages Will Be Coming From Many Partners, Many Different Sources

Think Of Every Agency You Exchange Information As A Trading Partner

Always Consider Your Partners Capabilities

Real-Time Processing To Handle The Differences

Message Manager: the history of a message

Correlation Manager: expectations for a message

Names of transformsProcess flowsPipelinesAccess keys (i.e. ISA06)

Basic MOUProfile

Formal namesError handler contacts

Adapter Technology

Doug’s material

Agency Adapter Platforms

Enables connectivity from applications, application environments, and/or databases to an integration server platform

Provide the flexibility to integrate with the presentation, business logic or database tier of an enterprise application

Act as an extension to an enterprise application, providing a flexible solution for exposing external GJXDM messaging interfaces which can be leveraged for exchanging data in both directions

Provide an architecture that enables separate applications to work together in a de-coupled fashion such that applications can be easily added or removed without affecting others

Adapter is based on iWay Software

– Phase 2 Objectives

– Related Integration Initiatives

– Technical Enhancements

DataShare – Future Phases

1) Expand List of Participating DataShare Agencies

NYS Division of Criminal Justice Services (DCJS)

NYS Office of Children & Family Services (OCFS)

NYC Department of Investigation (DOI)

2) Expand List of Services Available Via DataShare & Public Safety Portal

Integrated Juvenile Justice (IJJ) exchanges

eArraignment exchanges

Assorted other exchanges and queries

3) Enhance Existing Shared Applications:

DNA HITS

18B Web

e911

4) Develop New Shared Application - eArraignment

NYC Objectives for Integration in P2

DataShare & Related Integration Initiatives

Application/

InitiativeDescription Agencies Involved

Future Integration via DataShare

eArraignment

New shared application to automate assembly of information between arrest and arraignment of defendants

NYPD, DA’s, Courts, Criminal Justice Agency, NYS Division of Criminal Justice Services (DCJS)

Application and relevant exchanges via DataShare (arrest info, criminal history, complaints, defendant interviews) to be developed in 2007

DNA HITS

Shared application that facilitates case management across multiple agencies when DNA profiles match City or State databases

NYPD, DA’s, Chief Medical Examiner, NYS Division of Criminal Justice Services (DCJS)

Services provided via DataShare (jail status query, arrest and release notifications, criminal history, PSI, defendant interviews) will be integrated with the DNA HITS application

Integrated Juvenile Justice

Targeted initiative aimed at enhancing integration among agencies that deal with juvenile delinquents and offenders

Dept. of Juvenile Justice, Law Department, Dept. of Probation, Family Court, NYS Division of Criminal Justice Services (DCJS), NYPD

A set of new exchanges and services will be developed and deployed via DataShare to support this initiative in 2007.

e911

Shared application that allows DA’s and Law Department to request recordings of 911 calls from NYPD

NYPD, DA’s, Law Department

XML requests will be submitted by some agencies via DataShare in 2007.

NYC Integration - Future Landscape

Integrated Juvenile Justice Law Department Dept. of Probation Dept. of Juvenile Justice (DJJ)

Queens County DA

Richmond County DA

Manhattan DA

Assigned Council Plan (ACP)

Public Safety Portal

Shared Criminal Justice Applications eArraignment E911 18B Web DNA HITS

Dept. of Correction (DOC)

NYPD

Criminal Justice Agency (CJA)

Office for Court Administration (OCA)

Bronx County DAKings County DA

Legal Aid Society (LAS)Special Narcotics Prosecutor

Human Resource Authority (HRA)

ACS

OCFS

New York State Agencies (e.g., DCJS)

DataShare 2.0(DoITT Hub that Brokers Data

Exchanges Among Agencies and Applications)

DataShare 2.0(DoITT Hub that Brokers Data

Exchanges Among Agencies and Applications)

Data Exchanges (all GJXDM)

Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal

Criminal justice data that is shared between and among participating agencies via the hub and portal

The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare

Phase 1 Phase 2

Agency Participation

Public Safety Portal Functionality and Links to Applications

Criminal Justice Integration Platform (Hub)

Phase 1 Providers• NYPD• NYC DOC• Criminal Justice Agency (CJA)• OCA

Phase 1 Receivers• NYPD• Department of Probation• District Attorneys (5)• Special Narcotics Prosecutor• Law Department• Dept. of Juvenile Justice (DJJ)• OCA• Legal Aid Society• Assigned Counsel Plan• HRA• DoITT (on behalf of CJIS)

The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT

New Providers• Department of Probation• District Attorneys (5)• Special Narcotics Prosecutor• Law Department• Dept. of Juvenile Justice• Legal Aid Society• 3-1-1• Department of Finance• NYS DCJS

New Receivers• NYS DCJS• NYS OCFS• NYC DOI

Core Hub Features & Functionality• Push functionality• Pull functionality• Query application• Notification application

Platform Enhancements• Workflow pilot to support juvenile justice process• Federated query• SOA• Geo-coding service

Business Deliverables• NYPD arrest data• NYPD arrest notification• CJA ROR interview• OCA supreme and criminal court actions• DOC inmate information• DOC release notification• DOC jail status query• NYSID – SSN match

Business Deliverables• Phase 1 exchanges sent to additional receivers• DOP’s Pre-sentencing investigation (PSI) report for NYPD, DAs• Juv. Just. exchanges between Law, DOP, DJJ, OCFS, Fam. Court• DAs’ declined to prosecute data for DCJS and NYPD• DAs’ protected parties data for OCA• Family Court UCMS data for 18B Web• Legal Aid assigned attorney data for OCA• NYS DCJS criminal history (rap)

Core Portal Features & Functionality• Functionality for agencies to subscribe to notifications • Functionality for agencies to perform queries • Functionality for agencies to pull data• System administration for agencies • DNA HITS application link• e911 application link • webCRIMS application link • e-Justice portal link from DCJS

Portal & Supported Applications• DNA HITS application enhancements and RSO• 18b Web application enhancements• NYPD Photo application link• NYS DOCS application link• Integration with e-Justice

Jan ‘05 – July ’06 Aug ’06 – Dec ’07

NYC Integration – P2 Scope and Deliverables

Phase 2 (targeted)Phase 1

Technology Enhancements

The Integration Service Offering solution has been built on flexible infrastructure, software & hardware, with expandability in mind. As technologies and user needs evolve, the Enterprise Service Bus and its supporting J2EE infrastructure will enable new levels of integration through the use of solutions such as: Web Services (centralized web services registry) Distributed / federated applications Business activity monitoring (BAM) Workflow Business process choreography (BPC)

As various City agencies see the true value in this solution their needs and requirements will evolve. This constant evolution process will inform / guide DoITT in its decisions to provide the above mentioned solutions or functionality.

Q&A