extensible information models with dds

Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech [email protected]!

海賊天使

DDS Overview

Copyrig

ht 2010, PrismTech – A

ll Rights Reserved.

Data Distribution in a Nutshell

¨  Data distribution is about making application defined data available where needed and when needed, while preserving and end-to-end type contract

¨  Data is a first class concept, it can be Created, Read, Updated, and eventually Disposed (CRUD)

¨  The last value (or last N-values) of a Data is always available to applications

Id Temp Scale

101 25 C

202 78 F

303 299 K

DW

DW

DW

DR

DR

DR

struct TempSensor { ! @Key long Id; ! float temp; ! TScale scale; !} !

Fully Distributed Global Data Space

enum TScale { ! C, F, K !}; !

DW: DataWriter DR: DataReader

Copyrig


ll Rights Reserved.

DDS Topics ¨  A Topic defines the subject of

publications and subscriptions

¨  A Topic has associated a user defined type and QoS

¨  The Topic name, type and QoS have a well defined role in matching subscriptions

¨  Topics can be discovered or locally defined

Topic

Name

QoS Type

DURABILITY, DEADLINE, PRIORITY, …

[1/2]


“com.myco.TTempSensor”

Copyrig


ll Rights Reserved.

DDS Topics

¨  DDS Topic types can have associated keys

¨  Each unique key-value identify a Topic Instance – a specific stream of values

[2/2]

Topic

Name

QoS Type


DURABILITY, DEADLINE, PRIORITY, …


DDS Type System

Copyrig


ll Rights Reserved.

DDS Type System The DDS Type System is defined by means of:

¨  Abstract Type System ¨  The definition of types supported by DDS

¨  Concrete Syntax(es) to express types ¨  Syntax for specifying DDS Topic Types

¨  Serialization Format ¨  Format for representing topics types as a byte-stream

Limitations in DDS v1.2

Copyrig


ll Rights Reserved.

DDS v1.2 Type System ¨  Abstract Type System

¨  Monomorphic Nominal Type System

¨  Not explicitly defined by the specification

¨  Concrete Syntax(es) to express types ¨  IDL Subset (e.g. struct, enum, union, primitive types)

¨  Serialization Format ¨  CDR (Common Data Representation)

Copyrig


ll Rights Reserved.

Limitations ¨  The monomorphic nominal type system makes it

hard to, transparently extend, information models ¨  For a given topic, DataWriters and DataReaders need to

use exactly the same type definition thus limiting independent evolutions of the reading/writing side

¨  The type system is not formally defined

[Abstract Type System]

Copyrig


ll Rights Reserved.

Example ¨  Suppose that a new Temperature

Sensor was available that produced humidity estimates along with temperature

¨  In DDS v1.2 it would be impossible to deploy new sensors along with old sensors while using the same topic

¨  So what could we do to incrementally update the system?


struct TempSensor { ! @Key long Id; ! float temp; ! TScale scale; ! float hum; ! !} !

Old Sensor

New Sensor

[1/2]

Copyrig


ll Rights Reserved.

TTempSensor

Inconsistent Topic Types ¨  The scenario below would lead, in DDS v1.2, to an Inconsistent Topic Type error

struct TempSensor { ! @Key long Id; ! float temp; ! TScale scale; ! float hum; !} !

TTempSensor struct TempSensor { ! @Key long Id; ! float temp; ! TScale scale; !} !

TTempSensor

DW DR

Copyrig


ll Rights Reserved.

Example ¨  The solution to the problem is to introduce

and Extension Topic/Type

¨  This extension topic/type shares the same key as the original topic and contains the new attributes

¨  This gets the job done. Yet it adds complexity to the applications that will want to take advantage of the capabilities provided by new sensors. These applications, will now have to deal with two topics, or rely on Multi-Topics


struct TempSensorExt { ! @Key long Id; ! float hum; !} !

Old Sensor

Sensor Extension

[2/2]

Copyrig


ll Rights Reserved.

struct TempSensorEx { ! @Key long Id; ! float temp; !} !

TTempSensorEx

TTempSensor

Topic Extension ¨  The Legacy DR continues to use the old topic, while new DR are made

aware of the extension topic



TTempSensor DR

struct TempSensorEx { ! @Key long Id; ! float temp; !} !

TTempSensorEx

DW DR

TTempSensorEx

Legacy DR

Copyrig


ll Rights Reserved.

Limitations ¨  DDS v1.2 specifies IDL as the only concrete syntax

for specifying Topic Types

¨  In addition, no standard/portable way of specifying keys is defined

¨  Not everyone is happy with using IDL. Some users would like to use UML to define topic types, other, XML, etc.

[Concrete Syntax]

Copyrig


ll Rights Reserved.

Limitations

¨  CDR is a very efficient binary serialization format, its only limitation is the lack of support for extensible types

¨  This stems from the fact that CDR pre-supposes the exact knowledge of types and assumes that the order of type attributes is invariant

[Serialization Format]

Copyrig


ll Rights Reserved.

Limitations Recap

¨  The main limitations present in the DDS v1.2 type system are the limited support for information model extensibility and evolution

¨  Under the current model, information model evolution or extensions are either non-transparent or require complete update-redeploy cycles

DDS Extensible and Dynamic Topic Types

[DDS-Xtypes]

Copyrig


ll Rights Reserved.

DDS-XTypes Overview The DDS-XTypes Specification defines:

¨  A polymorphic structural type system for DDS topic types – which formalizes extends in several ways the DDS v1.2 type system

¨  A set of standard concrete syntaxes for representing topic types

¨  A set of serialization formats supporting extensible types

¨  A Dynamic API for defining Topic Types and DR/DW operating over these types

DDS-XTypes Type System

Copyrig


ll Rights Reserved.

Type System

¨  The DDS-Xtypes type system defines the rules and the type constructors that can be used to create valid DDS Topic Types

¨  The type system also defines a structural sub-type relation that is used to determine assignability of topic types

Copyrig


ll Rights Reserved.

Primitive Types ¨  The DDS-XTypes defines the following Primitive Types

¨  The Type mapping defines how DDS Types are mapped to native Programming Language Types (e.g. C99 int32_t, float, etc.)

Primitive Types

Integral Types

Byte Int16 Int32 Int64 Char8 Char16

Floating Point Type

Float32 Float64 Float128

Boolean

Bool

Copyrig


ll Rights Reserved.

Constructed Types ¨  The DDS-Xtypes introduces some new constructed types not

originally available in DDS v1.2 (marked in red in the figure below)

Constructed Type

Collection

Map Sequence String Array

BitSet Enumeration Aggregation

Union Structure Annotation

Alias

Copyrig


ll Rights Reserved.

Annotations ¨  DDS-Xtypes introduces annotation as a way to

attach meta-data to , and control the semantics of, types and type members

¨  Annotations are aggregated types whose members are restricted to being: ¨  Primitive Types ¨  String

¨  Enumeration

Copyrig


ll Rights Reserved.

Annotations in IDL ¨  Some built-in annotations, such as

@ID, @Key, @Optional, etc., are defined by DDS-XTypes specification

¨  User can define their own annotations

¨  Annotation can also be added as //@ after the annotated type member

@Annotation!local interface MyAnnotation { ! attribute long attr1; ! attribute string attr2 default “zzz”; !} !!@MyAnnotation!struct MyStructA { ! long member1; ! string member2; !} !!struct MyStructZ { ! @MyAnnotation (attr1 = 10, ! attr2 = “yyy”)! long memberX; ! string memberY; //@Optional!} !!

Copyrig


ll Rights Reserved.

Structure ¨  Structures are the type used to associate user-defined types with

Topics

¨  Structures defined by the DDS-Xtypes Specification support Single Inheritance

¨  Structures members have associated some properties, namely: ¨  ID. To uniquely identify an attribute ¨  Key. To identify whether the member is part of the key ¨  Optionality. To express the fact that an attributed is optional and thus

might not be set ¨  Must Understand. To enforce that a given field is not discarded ¨  Shared. To allow for non-inline data-storage of the attribute

Copyrig


ll Rights Reserved.

DDS-XTypes Structures ¨  Built-in annotations can

be used to control the semantic associated with type members

¨  Annotations, as explained next, are also used to control the kind of extensibility/mutability supported

struct TempSensor { ! @Key @ID(10) ! long Id; ! @MustUnderstand @ID(11) ! float temp; ! @Optional @ID(12) ! float hum; !}; !!!Struct InfraRedTempSensor : TempSensor { ! @Shared @ID(20) ! byte spectrum[1024]; !}; !!

Copyrig


ll Rights Reserved.

Type Extensibility & Mutability To better support incremental system updates, the DDS-XTypes Specification defines 3 different kinds of types. The @Extensibility annotation controls the kind attached to a given type

¨  Final ¨  The type is sealed and nothing can be changed w/o breaking compatibility

¨  Extensible (Default) ¨  The Type is monotonically extensible, meaning that new attributes can be added

while preserving the compatibility previous versions of the same type

¨  Mutable ¨  The Type can be mutated by adding, reordering and removing fields. The type

compatibility depends on structural compatibility

Copyrig


ll Rights Reserved.

Assignability ¨  The compatibility between different types, is expressed in

terms of assignability

¨  Given two types, T1 and T2, then based on the kind of extensibility supported, T1 is-assignable-from T2 if the type T1 can be “projected/widened” from T2

¨  Example



T1 T2

Projection

Copyrig


ll Rights Reserved.

Final Types Assignability Rules ¨  Given two final types T1 and T2, we say that “T1 is-

assignable-from T2”, iff: ¨  T1 is-assignable-from T2 iff the two types have exactly the

same members – meaning exactly same names and types as well as @ID and @Key annotations

¨  Example:

@Extensibility(FINAL_EXTENSIBILITY) !struct TempSensor { ! @Key long Id; ! float temp; ! TScale scale; !} !

@Extensibility(FINAL_EXTENSIBILITY) !struct TSensor { ! @Key long Id; ! float temp; ! TScale scale; !} !

T2 T1

Copyrig


ll Rights Reserved.

Extensible Types Assignability Rule

¨  Given two extensible types, T1 and T2, we say that T1 is-assignable-from T2 iff: ¨  T2 is a monotonic extension of type T1, e.g., T1 is a “prefix”

of T2

¨  Example: @Extensibility(EXTENSIBLE_EXTENSIBILITY) !struct TSensor { ! @Key long Id; ! float temp; ! TScale scale; !}; !

T1 @Extensibility(EXTENSIBLE_EXTENSIBILITY) !struct TSensor { ! @Key long Id; ! float temp; ! TScale scale; ! float hum; !}; !

T2

Copyrig


ll Rights Reserved.

Mutable Types Assignability Rule

¨  Given two mutable types, T1 and T2, we say that T1 is-assignable-from T2 iff: ¨  T1 and T2 share the same keys (e.g. same members

sharing name and ID) ¨  Homonymous members in T1 and T2 share the same ID

and vice-versa ¨  For each member x of T1 (T1.x) and y of T2 (T2.y) sharing a

common ID then T1.x is-assignable-from T2.y ¨  All @MustUnderstand member of T2 also exist in T1

Copyrig


ll Rights Reserved.

Mutable Types Example

@Extensibility(MUTABLE_EXTENSIBILITY) !struct HumSensor{ ! @Key @ID(10) long Id; ! @ID(13) float hum; !}; !

T1 @Extensibility(MUTABLE_EXTENSIBILITY) !struct TempSensor { ! @Key @ID(10) long Id; ! @ID(11)float temp; ! @ID(12)TScale scale; ! @ID(13)float hum; !}; !

T2

T1 is-assignable-from T2

@Extensibility(MUTABLE_EXTENSIBILITY) !struct HumSensor{ ! @Key @ID(10) long Id; ! @ID(13) float hum; ! @ID(20) long status; !}; !

T1 @Extensibility(MUTABLE_EXTENSIBILITY) !struct TempSensor { ! @Key @ID(10) long Id; ! @ID(11)float temp; ! @ID(12)TScale scale; ! @ID(13)float hum; !}; !

T2

T1 is-assignable-from T2

Copyrig


ll Rights Reserved.

Type Consistency QoS

¨  The DDS-XTypes introduces a the Type Consistency Enforcement Policy to control legal type conversion

¨  This Policy defines the following possibilities: ¨  EXACT_TYPE. As DDS today (but with checks) ¨  EXACT_NAME. What DDSI requires today

¨  DECLARED. Compatible type signatures + assignability ¨  ASSIGNABLE. Assignability is what matters

Copyrig


ll Rights Reserved.

Topic Model ¨  The DDS-XTypes

specification extends the topic definition by introducing the concept of a type signature

¨  Type signature are used to constrain the set of types that might be considered assignable

Topic

Name

QoS

Type

Type Signature

TYPE_CONSISTENCY DURABILITY, DEADLINE, PRIORITY, … struct TempSensor { !

@Key long Id; ! float temp; ! TScale scale; !} !


“TempSensor, TSensor”

Information Model Evolution Example

Copyrig


ll Rights Reserved.

Example – Reloaded ¨  Suppose that a new

Temperature Sensor was available that produced humidity estimates along with temperature

¨  So what could we do to incrementally update the system?



Old Sensor

New Sensor

Copyrig


ll Rights Reserved.

TTempSensor

Assignable Topic Types ¨  With DDS-XTypes the system evolution does not introduce any burden

¨  Types are projected through the assignability rules



TTempSensor

DW

DR


TTempSensor

DR

Guidelines

Copyrig


ll Rights Reserved.

Picking your “Extensibility” ¨  Final Types should be used to express universal invariant of the system.

Something that is so fundamental it should never-ever change w/o requiring a full system update

¨  Extensible Types are the default in the specification because their semantic is pretty straightforward and safe. Thus, if you don’t want to get into type refactoring you might stick with this default

¨  Mutable Types give you maximum flexibility, however you should be designing your information model around them. When using Mutable Types ensure that you carefully use the @MustUndertand and @Optional annotations to set structural lower bounds on your type as well as nicely dealing with absence of members

Copyrig


ll Rights Reserved.

Type Signature ¨  If you design your information model properly, then

you are safe in ignoring the type signature and using the ASSIGNABLE Type Consistency policy

¨  However, when integrating existing systems, the type-signature is key to scope what you might match

Summing Up

Copyrig


ll Rights Reserved.

Concluding Remarks

¨  The DDS-XTypes specification provides a set of powerful extensions to deal with system extensibility and evolution

¨  The Specification is adopted and under finalization

¨  The OpenSplice DDS team is already implementing it!

Copyrig


ll Rights Reserved.

http://www.opensplice.com/

http://www.opensplice.org/

emailto:[email protected]

http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com

http://bit.ly/1Sreg

http://www.slideshare.net/angelo.corsaro

http://twitter.com/acorsaro/

D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m

OpenSplice DDS

extensible information models with dds

Technology

dds type systemcopyright

dds topics dds topic

type definition

defined topic

dds v1

abstract type system

user defined type

inconsistent topic type