data transport standard

Data Transport Standard

Nathan Chitty

Gary Allen

Session 51

2

Data Transport Issues in Higher Ed

• E-mail is not reliable or flexible enough for confidential

and time-sensitive data

– No guarantee of delivery

– No guarantee of order of delivery for sequence

dependent data

– No automatic confirmation of receipt or facility for

retransmit

– No synchronous response available

– Email size limitations

3


• FTP data exchange has own challenges

– Possible to overwrite earlier files

– No confirmation of receipt

– No synchronous response

4


• Encryption today is always separate and subject to

its own

– Issues

– Maintenance

– Failures

5

DTS Addresses These Transport Issues

• DTS addresses

– The confirmation issue with a send-receive

protocol – confirmation is built in

– The order of delivery problem by actively

delivering and receiving the data – no

unconfirmed hand-offs

6


• DTS addresses

– The size problem through data compression

– The FTP overwrite problem by not using

filenames

7


• DTS addresses

– The lack of a synchronous response by building

in a required synchronous response, even if

only for handling status

– The encryption issue by using standard HTTPS

for encryption – the same technology as for

online banking

8

What is DTS?

• DTS – Data Transport Standard – is a

specification for an adjunct to or a replacement for

existing data transport mechanisms

– PGP / GnuPG encryption

– SecretAgent w/ Email

– FTP and SecureFTP

9

What is DTS?

• DTS is a specification not a product

• DTS was developed by the Postsecondary

Education Standards Council (PESC) for

exchanging data for:

– Inquiries

– Reports

– Transactions

10

What is DTS?

• DTS uses Internet technologies to facilitate real

time data exchange and transaction processing

• DTS builds on stable technologies, not specific

products

• DTS, once implemented, reduces programming

and per-transaction costs through standardization

11

DTS Benefits

• A Web Services implementation

– Delivery confirmation included – no guessing

– All requests get a response

– All submissions get an answer of some kind

• Facilitates real time data exchange

12

DTS Benefits

• Includes automatic data encryption

• Uses digital signature standards

• Platform independent

• Strong authentication with non-repudiation

13

Benefits To “System Providers”

• Adds value to schools’ systems

– Schools want transport added to systems and are

generally not concerned with the technologies

• Easier to build one transport protocol for all

recipients

– Just as CommonRecord created the drive to build

one XML format

14

Benefits to “Service Providers”

• As everyone implements DTS, the need to support

other transports will drop

• If any school implements DTS, service providers

will have to support it

• Also provides a single communication

infrastructure option for internal systems

15

DTS Specification

• Specification covers

– Technical interchange rules and processes

– Recommended best practices

• Technical Specification is the pure Simple Object

Access Protocol (SOAP) interface

• Implementation Guide is for both .Net and Java

reference implementations

16

DTS Specification

• Reference implementation examples are available

• Specification does not cover

– Business rules for transaction processing

– Operational oversight, monitoring or escalation

17

DTS Technical Workgroup

• Task: Create a written specification for real-time

exchange of data between organizations

– Meets business requirements

– Standards based

– Standard technologies (Java, .Net)

– Payload Insensitive

– Secure and reliable

18

DTS Technologies

• Global XML Web Service Architecture (GXA),

generally accepted as the foundation for building

Web Services

– WSDL (Web Service Definition Language)

– SOAP (Simple Object Access Protocol)

– WS-I (Web Service Interoperability)

– WS-S (Web Service Security)

19

DTS Technologies

• WS-Security (Digital Signatures)

– Strong authentication with non-repudiation

– X.509 encryption keys and certificate

authorities

• SSL encryption of HTTP streams

20

Anticipated Architectures

• Immediate processing

– Request and processed Result Response

• “Push/Push” deferred processing

– Request and Acknowledge Response

– Request with Result and Acknowledge Response

• “Push/Pull” deferred processing

– Request and Acknowledge Response (just send)

– Request for Result and Result Response

21

Immediate

Trading Partner BDTS Service

Trading Partner BProcessing Application

Trading Partner AData Application

Trading Partner ADTS Client

Internet(https)

37

1 8

5 6

1. Data application creates data to be sent and hands off to DTS Client to send.2. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository.3. DTS Client makes request to DTS Service sending the data.4. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.5. DTS Service extracts data from DTS Request and hands off to a processing application.6. Processing application returns processed results to the DTS Service.7. DTS Service responds to the DTS Request with the processed results.8. DTS Client uses public key (step 2) to verify signature and then hands processed results back to the data application.

Repository Repository

2 4

22

“Push/Push” (part 1)




Internet(https)

1C1F

1A 1E1A. Data application creates data to be sent and hands off to DTS Client to send.1B. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository.1C. DTS Client makes request to DTS Service sending the data.1D. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.1E. DTS Service extracts data from DTS Request and stores it for guaranteed processing later.1F. DTS Service responds to the DTS Request acknowledging the data was received.


1B 1D

Trading Partner BDeferred Storage

23

“Push/Push” (part 2)


Trading Partner ADTS Service

Trading Partner BDTS Client

Internet(https)

2C2F

2A. Trading Partner B finishes processing the dat and hands off to DTS Client to send.2B. DTS Client looks up Endpoint URL (and public key) for Trading Partner A from their repository.2C. DTS Client makes request to DTS Service sending the data.2D. DTS Service looks up public key of Trading Partner B from their repository and verifies signature of payload.2E. DTS Service extracts data from DTS Request and stores it to process later.2F. DTS Service responds to the DTS Request acknowledging the data was received.

Repository

Repository

2D

2B

2A

2E

24

“Push/Pull” (part 1)




Internet(https)

1C1F

1A 1E1A. Data application creates data to be sent and hands off to DTS Client to send.1B. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository1C. DTS Client makes request to DTS Service sending the data.1D. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.1E. DTS Service extracts data from DTS Request and stores it for guaranteed processing later.1F. DTS Service responds to the DTS Request acknowledging the data was received.


1B 1D

Trading Partner BDeferred Storage

25

“Push/Pull” (part 2)




Internet(https)

2B2E

2D

2A. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository2B. DTS Client makes request to DTS Service asking for data.2C. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.2D. DTS Service looks for processed data for Trading Partner A.2E. DTS Service responds to the DTS Request with any data available, if no data is available the returns “Nothing to Provide”.2F. DTS Client verifies signature from Trading Partner B, and extracts payload for processing.

Repository

Repository

2A

2C

2F

26

DTS Analogy

• DTS is the definition of the “Pipe” and the

structure of its contents

– The “Pipe” is the internet

– The content is SOAP

– The end points/junctions are Web Services

– The sources are Web Service enabled clients

27

DTS Analogy

• DTS defines how others can connect to the “Pipe”

already installed

– Any connections must have certain “threads”

– Any connections must handle two way traffic

independent of how the traffic will be used

28

DTS Analogy

• By knowing about the

pipe and the type of

connections, any

“plumber” can use his/her

own tools to make

connections; just so long

as the threads match

29

How Did We Do It?

• Created basic HelloWorld service and client

– Worked interoperable

• Added simple Headers to HelloWorld

– Was not interoperable

• Added complex Header to HelloWorld

– Was not interoperable

30

Why SOAP Headers

• To answer routing and processing expectations

without opening the payload

• Remain payload insensitive

• Allow extensibility for new processes

31

DTS SOAP Headers• DTSRequestRouting

• DTSRequestServiceExpectation

• DTSRequestPayloadType

• DTSRequestPayloadBytes

• DTSRequestSignature

•DTSResponseRouting

•DTSResponseAcknowledge

•DTSResponsePayloadType

•DTSResponsePayloadBytes

•DTSResponseSignature

32

Convoluted Filename vs Header Elements

• A [B] <X.Y.Z:M>

– A = File Type, B = Encrytption, X.Y.Z = key

identifier, M = Unique message ID

• Encryption unnecessary because using HTTPS

• DTSRequestPayloadType = A

• DTSRequestRouting

– SourceIDSubCode = X, SourceID =Y(.Z)

– UUID = M

33

Interop Problem with SOAP Headers

• xsi:type attribute in Header elements

– Java includes and requires this attribute

– .Net does not

34

All about SOAP

<soap:Header><DTSRequestPayloadType xsi:type="DTSRequestPayloadType" xmlns="http://www.datatransportstandard.com">

<value>CRC01Request</value>

</DTSRequestPayloadCode>

35

SOAP is the Key

• The SOAP transmitted across the wire is of

primary importance

– Element names

– Type attribute

– Not Namespace moniker (Java uses one by

default, .Net does not)

• How you get the correct SOAP is not important

36

Java:<ns1:DTSRequestSignature

soapenv:mustUnderstand="0" xsi:type="ns1:DTSRequestSignature"

xmlns:ns1="http://www.datatransportstandard.com"><ns1:value>SignatureValue</ns1:value>

</ns1:DTSRequestSignature>

.Net:<DTSRequestSignature

xsi:type="DTSRequestSignature" xmlns="http://www.datatransportstandard.com">

<value>SignatureValue</value></DTSRequestSignature>

SOAP Differences That Do Not Matter

37

Reference Implementation Architecture

• Client Application

• Client Core

• Service Core

• Service Application

38

Client Application

• Knows nothing of SOAP or Web Services

• Implements Client Core Interface

– “Setters” and “Getters” of DTS specific

elements

• Houses specific business logic

39

Client Core

• Knows nothing of business logic

• Uses properties set to construct the SOAP

• Interface for “setting send” and “getting returned”

elements

• Handles the communication to Service Core- DTS

Specification

40

Service Core

• Accepts transmissions from Client Core

• Implements Service Application Interface

– “Setters” and “Getters” of DTS specific elements

• Creates return SOAP

– Format return acknowledgement or data from

Service Application

– Construct SOAP faults

41

Service Core (con’t)

• Isolated business logic

– Examples

• Invoke Service Application based on payload

• Place payload in “queue”

42

Service Application

• Interface for “setting sent” and “getting to be

returned” elements

• Houses specific business logic

• Knows nothing of SOAP or Web Services

43

Connecting the layers

Internet

Client Application

Client Core Service Core

Service Application

DTS SpecDTS Spec

Entity A Entity BPlatform specific communication

Platform specific communication

44


Internet

Client Application


Service Application

DTS SpecDTS Spec



Client Application

45


Internet

Client Application


Service Application

DTS SpecDTS Spec



Client Application

Client Application

46


Internet

Client Application


Service Application

DTS SpecDTS Spec


Client Application

Client Application

Service Application


47


ESB

Internet

Client Application


Service Application

DTS SpecDTS Spec

Point A Point B

Client Application

Client Application

Service Application

Platform, but DTS interface Platform, but DTS interface

48

Connecting the Layers

ESB

Internet

Client Application


Service Application

DTS SpecDTS Spec

Point A Point B

Client Application

Client Application

Service Application


ExternalClient Application

49

DTS Version 2

• New tools

• No changes to interfaces

• Reorganization of SOAP Headers

• Full WS-Security integration

50

New Tools

• Axis 1.4 removed requirement for xsi:type

attribute

– Reduces complexity from .Net side

• Implemented WSS4J and WSE

– WS-Security integration

51

Reorganization of Headers

<DTSRequestRouting xsi:type="DTSRequestRouting" xmlns="http://www.datatransportstandard.com">

<UUID> <transmissionDateTime> <sourceID> <sourceIDCode> <recipientID></DTSRequestRouting><DTSRequestPayloadType

xsi:type="DTSRequestPayloadType" xmlns="http://www.datatransportstandard.com">

<value>CRC01Request</value></DTSRequestPayloadType><DTSRequestPayloadBytes

xsi:type="DTSRequestPayloadBytes" xmlns="http://www.datatransportstandard.com">

<value>53</value></DTSRequestPayloadBytes><DTSRequestServiceExpectation

xsi:type="DTSRequestServiceExpectation" xmlns="http://www.datatransportstandard.com">

<value>Immediate</value></DTSRequestServiceExpectation><DTSRequestSignature

xsi:type="DTSRequestSignature" xmlns="http://www.datatransportstandard.com">

<value>someting</value></DTSRequestSignature>

<DTSRequestHeader xmlns:dts="urn:org:pesc:datatransport" soap:mustUnderstand="1" soap:actor="urn:org:pesc:datatransport/dts" xmlns="urn:org:pesc:datatransport">

<dts:DTSRequestRouting> <dts:DTSUUID> <dts:DTSSourceID> <dts:DTSSourceIDSubCode> <dts:DTSRecipientID> <dts:DTSRecipientIDSubCode> </dts:DTSRequestRouting> <dts:DTSRequestPayloadType> <dts:DTSRequestPayloadBytes> <dts:DTSRequestServiceExpectation> </DTSRequestHeader>

<wsse:Security soap:mustUnderstand="1"> <wsu:Timestamp> <wsu:Created> <wsu:Expires> </wsu:Timestamp>... <SignatureValue>…</wsse:Security>

Version 1 Version 2

52

WS-Security

• Version 1 utilized portions

– Authenticity of data via Digital Signatures

– However, proprietary Signature header element

• Version 2 implements WS-S Header construct

– Removal of proprietary signature element

– Inclusion of Security block

– Inclusion of Binary Security Token element

53

WS-Security (con’t)

• Binary Security Token Element

– Specification allows for the public X.509

certificate to be included in the transmission

– Since, Certificate Authority signed X.509

certificate

• No need to exchange public key

• Trust the certificate chain (CA)

54

WS-Security

• .Net

– Implementing WS-S via WSE in .Net adds WS-

Addressing block

– Creates interoperability problem between .Net

and Java

– SOAP Output handler created to remove it

55

Technical Wrap Up

• Version 1 approved by PESC (5/2006)

• Electronic Exchange Advisory Team (EEAT) within NCHELP- Electronic Standards Committee is currently writing FFEL Commonline, CAM, CR:C implementation rules

• DTS Technical group is currently writing change proposal for PESC

– No changes to interfaces, Just Better!

56

Connecting the Layers

ESB

Internet

Client Application


Service Application

DTS SpecDTS Spec

Point A Point B

Client Application

Client Application

Service Application


ExternalClient Application

57

Additional DTS Information

• Visit PESC at www.pesc.org

• Materials available include

– Executive summaries

– Specifications

– Reference (proof of concept) implementations

58

We appreciate your feedback and comments. We can be reached at:

Name: Nathan Chitty, Nelnet, Inc.Phone: 904-281-7235Email: [email protected]

Name: Gary Allen, ELM ResourcesPhone: 510-903-0674Email: [email protected]

Contact Information

data transport standard

Documents

dts data transport standard

transport protocol

timesensitive data

data compressionthe

higher edftp data exchange

protocol confirmation

required synchronous

higher edencryption