beyond messaging
DESCRIPTION
The OMG Data Distribution Service (DDS) has brought a disruptive wave of innovation into what is commonly referred to as “Messaging” technology. DDS raises the abstraction level from unstructured, or loosely structured, messages to sound data types that allow architects and application developers to deal with concepts close to their problem space as opposed to having to worry about “plumbing” -- as it happens as with many messaging technologies. As it is common for disruptive innovations to take some time to be digested, many people still see DDS as an ultra-fast real-time messaging -- yet this is only the tip of the iceberg.This presentation will explain how Data Distribution is different from Messaging and as a result how DDS differs from messaging technologies such as JMS. The webcast will cover the architectural benefits brought by data centricity as well as how the features introduced by the recently adopted DDS specification on Dynamic and Extensible Topic Types will support the design of efficient, extensible and evolvable systems.TRANSCRIPT
Angelo Corsaro, Ph.D.Chief Technology OfficerPrismTechOMG DDS SIG [email protected]
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Beyond Messaging....
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./messaging
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
What is Messaging about?
‣ Messaging is about distributing messages generated by a producer to its consumers
‣ Messaging technologies often supports P2P and Pub/Sub interactions
‣ The producer/consumer exchange are often mediated by brokers To
pic
Queue
P2P Messaging
Pub/Sub Messaging
‣ A message is the mechanism provided to messaging applications for exchanging information
‣ A Message is usually composed by a header and an opaque body
‣ In other terms, a message is an untyped (or weakly typed) information holder
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
What is a Message?
Header Body
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
An Example: AMQP Messages
‣ AMQP define a message as made by a header and a body
‣ The body is an opaque sequence of bytes
‣ Thus, data serialization is delegated at the application
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
An Example: JMS Messages
‣ JMS define a message as made by a header and a body
‣ The body can be of five different types
‣ Text‣ Map‣ Array of Bytes‣ Stream of native types‣ Serialized Object
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
ISO/OSI view of Messaging
‣ Messaging operates at the Session layer but does deal with the presentation layer (e.g. data representation).
‣ Messaging applications have to deal directly with data presentation
Messaging
Messaging App
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Messaging in JMS‣ The user data has to be
represented in a format supported by the JMS message body
‣ Object serialization has some shortcomings:
‣ not interoperable with other programming languages‣ Not the most time/space efficient way of encoding your data
// Create a TextMessageTextMessage message =
queueSession.createTextMessage();
// Create and Set the Message BodyString msgText = "body";message.setText(msgText);
// Send the Message queueSender.send(message);
Sending Message
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Messaging in JMS
‣ Application have to figure-out the actual type of the “Message”
‣ Data has to be extracted from the message
‣ Lot’s of potential for introducing errors that a sounder type-system could prevent
// Receive the MessageMessage m = queueReceiver.receive();
// Figure-out what it isif (m instanceof TextMessage) { TextMessage message = (TextMessage) m; String body = message.getText(); System.out.println("Received: " + body); }else if (m instanceof MapMessage){ // do something else}else { // Handle Error}
Receiving Message
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
In Summary
‣ Messaging provides a mechanism for shipping messages from producers to consumers
‣ The Message is a first class citizen, application data is not.
‣ The conversion of user data-types into the types supported by the messaging has to be addressed at the application level
‣ Byte sequences for AMQP ‣ Text/Map/Bytes/Stream/Object for JMS‣ etc..
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./data-distribution
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
What is Data Distribution?‣ Data distribution is about
making application defined data available where needed and when needed
‣ Data is a first class concept, it can be created, updated, read and eventually disposed
‣ The last value (or last N-values) of a Data is available
VehicleSpeedVehicleSpeedVehicleSpeed
plate
“A123”
“B456”
“C789”
dx dy
50 0
0 45
10 30
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
101 202
303 404
505 606
[1/2]
TrakCo.VehiclePosition
struct VehiclePosition { string plate; long x, y; }
TrakCo.VehicleSpeed
struct VehicleSpeed { string plate; long dx, dy; }
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
What is Data Distribution?
‣ The details of how application data is encapsulated and propagated are dealt by the data distribution technology
VehicleSpeedVehicleSpeedVehicleSpeed
plate
“A123”
“B456”
“C789”
dx dy
50 0
0 45
10 30
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
101 202
303 404
505 606
[2/2]
TrakCo.VehiclePosition
struct VehiclePosition { string plate; long x, y; }
TrakCo.VehicleSpeed
struct VehicleSpeed { string plate; long dx, dy; }
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDS
‣ The definition of Data in DDS is tied to the definition of a Topic
‣ A Topic is the association between: ‣ Unique Name‣ Type‣ QoS
‣ A Topic can have associated a key in order to express data instances identities
VehiclePosition
TrackCo.VehiclePosition
{ Reliability Deadline, Priority Transient, ...}
TopicType QoS
Name
[1/4]
Topics
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDS
‣ A Topic key can be made of an arbitrary number of Topic Type attributes
‣ Each unique key value identify a Topic Instance
‣ Topic Instance, can be Created, Read, Updated, and Disposed (CRUD)
[2/4]
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
101 202
303 202
101 606
TrakCo.VehiclePositionTopic
Topic Type
Topic Instances
Topic Key
Topic Instances
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDS‣ An update for a Topic Instance is called Sample
‣ DDS can maintain a configurable history of samples per Topic Instance
[3/4]
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
101 202
303 202
101 606
TrakCo.VehiclePositionTopic
Topic Type
Topic Instances
Topic KeyVehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
110 202
333 202
80 606
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
“B456”
“C789”
x y
150 222
366 202
50 606
Samples
Time
Samples
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDSCreation‣ A Topic Instance is automatically created the first time it is
written
[4/4]
Instances Lifecycle
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
101 202
TrakCo.VehiclePositionTopic
Topic Type
Topic InstancesTopic Key
Time
NEW
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDS
Update‣ Subsequent writes will simply update the value of the instance
[4/4]
Instances Lifecycle
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
101 202
TrakCo.VehiclePositionTopic
Topic Type
Topic InstancesTopic Key
Time
NEW
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
170 202
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data in DDS
Dispose‣ An instance can be explicitly disposed
[4/4]
Instances Lifecycle
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
101 202
TrakCo.VehiclePositionTopic
Topic Type
Topic InstancesTopic Key
Time
NEW
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
170 202
VehiclePositionVehiclePositionVehiclePosition
plate
“A123”
x y
170 202Disposed!
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
ISO/OSI View of Data Distribution
‣ Data Distribution operates at the Session+Presentation layers
‣ Data Distribution Applications transparently read/write application defined data
Data Distribution
Application
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data Distribution in DDS
‣ User defined types can be written with a simple write operation. Encapsulation and distribution are taken care by DDS
[1/2]
// Create dataVehiclePosition pos(“A101”, 101, 202);
// Write DatadataWriter.write(pos);
Writing Data
[Syntax compliant with the DDS-PSM-C++ upcoming standard as currently defined at http://code.google.com/p/dds-psm-cxx ]
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Data Distribution in DDS
‣ User defined types can be written with a simple write operation. Encapsulation and distribution are taken care by DDS
[2/2]
// Create data+info containersstd::vector<VehiclePosition> vpos;std::vector<SampleInfo> info;// Read DatadataReader.read(vpos, info);
Reading Data
[Syntax compliant with the DDS-PSM-C++ upcoming standard as currently defined at http://code.google.com/p/dds-psm-cxx ]
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
In Summary
‣ Data Distribution provides a mechanism for sharing user-defined distributed data types and transparently reading and writing instances of these types
‣ Data types and data instances are first class citizens. This implies that data-life-cycle transitions are also dealt-with
‣ User defined Data-Types are preserved end-to-end. Their encoding is transparently dealt by the data distribution service
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./vehicles-tracking
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
A Vehicle Tracking SystemImagine to develop a s system that has to deal with the following:
‣ Identify when a car enters and leaves a city
‣ Track the position of each identified car and draw a path of the last N positions
‣ Track the instantaneous as well as the window average speed. Fine if the window average exceeds the speed-limit
NEW
BYE
NEW
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Vehicle Tracking with Messaging‣ How are you going to manage the presence/absence of cars?‣ This concern has to be implemented in your application logic.
‣ How are you going to easily distinguish between cars?‣ Your application will have to deal with it.
‣ How are you going to maintain a window of vehicle speed so to enforce speed limits?‣ Again, your application will have to explicitly maintain a last K-values cache of the speed for each care and make sure that this is updated each time you receive an update.
Is there a better way to do this?
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Vehicle Tracking with DDS
‣ How are you going to manage the presence/absence of cars?‣ DDS provide that information through entity life-cycle management!
‣ How are you going to easily distinguish between cars?‣ DDS allow to access samples on a per-instance basis.
‣ How are you going to maintain a window of vehicle speed so to enforce speed limits?‣ DDS provide support for data history.
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Vehicle Tracking with DDS‣ Dynamic Discovery‣ Cars will be discovered as soon as there is network connectivity
‣ QoS‣ Local as well as End-to-End resource usage and temporal characteristic can be controlled via DDS QoSes
‣ Filtering‣ Filter can be performed on the whole content and is not limited by the attributes that are attached to the header by data publishers
‣ DDS is wire-interoperable, language, and OS independent, thus it is easy to deploy it anywhere
Much More with DDS
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Tracking System Topics
‣ Three different topics are used to represent the Vehicle, its position and the Speed
‣ The three topics define the same key so that information associated to the same car can be easily correlated
TrakCo.VehiclePositionTopic
Topic TypeTopic Key
struct VehiclePosition { string plate; long x, y; }
TrakCo.VehicleSpeedTopic
Topic TypeTopic Key
struct VehicleSpeed { string plate; long dx, dy; }
TrakCo.VehicleTopic
Topic TypeTopic Key
struct Vehicle { string plate; string model; }
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Tracking System Topics QoS
TrakCo.VehiclePosition
struct VehiclePosition { string plate; long x, y; }
TrakCo.VehicleSpeed
struct VehicleSpeed { string plate; long dx, dy; }
TrakCo.Vehicle
struct Vehicle { string plate; string model; }
Reliability History Durability Destination Order
RELIABLE KEEP_LAST TRANSIENT SOURCE_TIMESTAMP
Reliability History Durability Destination Order
BEST_EFFORT KEEP_LAST(3) VOLATILE SOURCE_TIMESTAMP
Reliability History Durability Destination Order
BEST_EFFORT KEEP_LAST(5) VOLATILE SOURCE_TIMESTAMP
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./extensible-topics
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Dealing with Type Evolutions
‣ A DDS-based System might need to evolve its information model without inducing any global impact on required upgrades nor breaking interoperability between new and existing part for the system
‣ As an example let’s assume that we have now some cars that can provide more information on the vehicle, how do we capture this evolution?
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
DDS ExtensibleType System
‣ DDS is now (see Extensible and Dynamic Topic Types Specification) equipped with a structural type system
‣ This type system support type extension (evolution in general) relying on a structural sub-type operator and a set of annotations and QoS that allow to control and constraint the type system
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./xtype-in-action
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Type Extensibility
Problem
‣ What if we need to extend the VehiclePosition type to add the error estimate on the x and y direction?
‣ How can we do this with no impact on running applications?
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Type ExtensibilitySolution‣ Simply extend the existing VehiclePosition and properly annotate new
attributes
‣ Existing application will detect that the new type is an extension and will “project/extend” it to the known type
‣ Type “projection/extensions” can be controlled via QoS
struct VehiclePosition { string plate; //@key long x, y; }
struct VehiclePosition { string plate; //@key long x, y; float xe, ye;}
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Type Extensibility
Problem
‣ OK, that was good. But now I need to add an optional image to my Vehicle
‣ How can I add an optional field? Furthermore, I don’t want the figure to be copied into the topic, I’d like it to be a reference. Can I do this to limit the number of copies and reduce he memory utilization?
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Type ExtensibilitySolution
‣ Simply extend the existing Vehicle and properly annotate new attributes
struct Vehicle { string plate; //@key string model; }
struct Vehicle { string plate; //@key string model; sequence<octet> photo; //@optional @shared}
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
./efficiency
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Encoding Efficiency
‣ DDS serialization is several times more efficient than JSON, XML and YAML
‣ Differences can be quite extreme when dealing with complex types
Size for Serialized CoFlight Flight Data Plan
see Esposito et al. OMG RTWS 2008
DDS encoding is 10x more space efficient than XML!
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Encoding Impact on RTT
see Esposito et al. OMG RTWS 2008
DDS RTT with different Serialization Format
‣ Inefficient encoding can easily dominate RTT
‣ DDS native encoding is the most efficient among extensible encodings!
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
Size (bytes)
Lat
en
cy (
use
c)
‣ DDS implementations are capable of delivering very low and predictable latencies
‣ No other Pub/Sub technology can approach similar level of performance!
DDS is Very Efficient!
© 2
010
Prism
Tech
. All
Rig
hts
Res
erve
d.
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
SO
MG
DD
S
In Summary
‣ DDS provides a strongly typed extensible type system
‣ Types can be evolved and extended without breaking backward compatibility nor loosing type-safety
‣ Type “Projection/Promotions” can be fully controlled via QoS
‣ This flexibility does not come at the cost of introducing the time+space overhead typical of XML-based approaches
DDS provides extensibility without compromising efficiency!
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
./summing-up
Ope
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
SOpe
nSpl
ice
DD
S
Concluding Remarks
‣ Data Distribution give you far more than plain Messaging: ‣ higher level of abstraction and more built-in functionalities‣ end-to-end type-safety‣ extensibility‣ performance
‣ Data Distribution is well beyond messaging, it is the most effective way of designing loosely coupled distributed applications.
http://www.opensplice.com/
http://www.opensplice.org/
emailto:[email protected]
http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com
http://bit.ly/1Sreg
http://www.slideshare.net/angelo.corsaro
http://twitter.com/acorsaro/
D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m
OpenSplice DDSReferences