Ope

nSpl

ice

DD

S

Extensible and Efficient Data Centric Architectures

Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig Co-Chair

PrismTechangelo.corsaro@prismtech.com

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Agenda☐ Data-Centric Architectures (DCA)☐ DDS and DCA☐ Extensible and Efficient DCA with DDS☐ Concluding Remarks

Ope

nSpl

ice

DD

S

Data-Centric Architectures

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Data-Centric Architecture (DCA)Abstract DefinitionA Data Centric Architecture consists of:☐ Abstract Data Types: A, B, C, ... ☐ Arrows: f, g, h, ...where arrows satisfy the following properties:

Given f: A → B, g: B → C, and h: C → D

then the following holds true:

∃ g ∘ f: A → C [Composition]

∃ 1A : A → A [Identity Arrow]

h ∘ (g ∘ f) = (h ∘ g) ∘ f [Associativity]

f ∘ 1A = f = 1B ∘ f [Unit]

A Data Centric Architecture is a Category

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Data-Centric Architecture (DCA)The abstract definition of a Data-Centric Architecture formally captures its key properties:

☐ The Abstraction Barrier is defined in terms of Abstract Data Types

☐ The System Behaviour is specified in terms of transformations (functions or arrows) over these Abstract Data Types

From this emerges a natural relationship between DCA and Functional Programming Languages

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Example: Space Traveller☐ Suppose we want to create a

distributed game in which we have (some) UFOs that are trying to destroy our spaceship

☐ UFO can be our allies/enemies and can be played by other players or the computer

☐ Some UFOs, might just be space travellers and should not be destroyed

☐ The goal of the game is to destroy the UFOs that have been classified as threats

[1/3]

Copyrig

ht  2011,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Example: Space Traveller

Data Types☐ FlyingObject☐ Dynamics☐ Classification☐ Collision

Arrows☐ animate: Dynamics → Dynamics☐ classify: FlyingObject x Dynamics → Classification☐ collide: [FlyingObject x Dynamics] → [Collision]☐ pilot: IO → Dynamics x FlyingObject☐ fire: IO → FlyingObject x Dynamics☐ display: FlyingObject x Dynamics x Classification x Collision

→ IO

☐ The DCA for this game could be described as follows (keeping it simple...)

[2/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Example: Space Traveller

pilot

fire

animate classify

collide display

IODynamics

Dynamics DynamicsDynamics

Classification

Dynamics

Classification[Dynamics][Collision]

Collision

IO

FlyingObject

FlyingObject

IODynamics

FlyingObject

FlyingObject

[FlyingObject]

[3/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Data-Centric vs. Service-CentricData-Centric and Service-Centric architectures, such as SOA, Distributed Objects, etc., differ in several dimensions:

☐ Abstraction Barrier:☐ Data-Centric architectures rely on data abstractions☐ Service-Centric architectures rely on service abstractions

☐ Level of Coupling☐ Data-Centric architectures rely solely on the shared knowledge of abstract data

types (agree on the “what”)☐ Service-Centric architectures rely on the shared knowledge of operational

interfaces along with the knowledge of the service provider and of its existence (agree on “what”, “who”, and “when”)

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

DCA Benefits☐ Loose Coupling

☐ Anonymicity (e.g. only things to be known are the data types and not who produces/consumes them)

☐ Composability (especially when combined with functional languages)

☐ Extensibility

☐ Ease of Integration

☐ Performance (easier to parallelize)

Ope

nSpl

ice

DD

S

DDS and DCA

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

DCAs with DDS

☐ DDS provides first class support for DCAs

☐ DCAs Abstract Data Types are captured in DDS via Topics

☐ DDS does not provide primitive support for specifying DCAs Arrows. These are defined by the architect using her/his favourite formalism

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

DDS Topics☐ A Topic defines a typed data

stream☐ A Topic has associated a data

type and QoS☐ The Topic name, type and QoS

defines the key functional and non-functional invariants

☐ Topics can be discovered or locally defined

DURABILITY,DEADLINE,PRIORITY,

…

“net.gva.VehiclePosition”

struct Position2D { long vid; //@Key long x; long y;};

TopicTypeName

QoS

Ope

nSpl

ice

DD

S

Extensible and Efficient DCA with DDS

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Back to the Space Traveller☐ In the next few slides,

we’ll explore the steps required to design extensible and efficient data models with DDS

☐ We’ll use the Space Traveller as our running example

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Space Traveller Topic Types//@Nestedstruct Bounds { long width; long height;};

struct FlyingObject { long oid; //@Key long kind; Bounds bounds;};

//@Nestedstruct Vector { long x; long y;};

struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

enum Classifier { UNKNOWN, FRIEND, THREAT};

struct Classification { long oid; // @key Classifier kind;};

struct Collision { long oid; long coid;};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

From 2D to 3D☐ Our original data model was designed for a 2D world

☐ Let’s assume that we want to be able to run our game in both 3D as well as 2D. In addition we don’t want to break interoperability between the 3D and the (older) 2D version

☐ Afterall the 2D visualization is simply a projection of the 3D visualization

☐ How can we do this?

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D w/o X-Types☐ If we don’t want to break interoperability with existing games, as we are not going

to be able to upgrade them all, we can’t change the existing topic types

☐ As a result we need to define another topic type to capture the 3rd dimension. This way the older version of the game will not notice the difference, while the new version will be made aware of the additional topic-type providing the 3rd dimension

//@Nestedstruct Vector { long x; long y;};

struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

struct DynamicsEx { long oid; //@Key long z; long dz;};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Extensibility w/o X-Types

☐ Extending and evolving a DDS data model w/o the use of X-Types introduces (1) fragmentation, (2) can potentially introduce update anomalies (e.g. inconsistency), and (3) adds complexity for both the architect and the developer

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

X-Types Background Info

☐ The DDS X-Types specification extends DDS with:☐ A polymorphic structural type system☐ A set of concrete syntaxes for expressing DDS Topic Types☐ Extensible Type Encoding☐ A Dynamic API for runtime creation and manipulation of

Topic Types, DataReaders and Writers

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Types Assignability☐ The main mechanism provided by the X-Types specification

to support interoperability among different types is the concept of “assignability”

☐ The rules of assignability depends on the kind of extensions/modifications allowed for the type:☐ FINAL => Types are identical☐ EXTENSIBLE => Types are one the monotonic extension of the other☐ MUTABLE => Types can be projected w/o violating attribute

invariants

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with X-Types☐ The first thing to observe from our current data model is that it does not

provide any extensibility annotations

☐ This means that by default it is considered “Extensible”, or to be more precise, monotonically extensible

//@Nestedstruct Vector { long x; long y;};

struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Extensible X-Types (EXT)

☐ Monotonically Extensible Types (EXT), as suggested by their name, imply extensions only at the “tail” of the type.

☐ Can we go from 2D to 3D using Monotonic Extensible Types?

☐ Let’s give it a try ...

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with EXT?☐ Let’s add explicit extensibility annotations first, so that

we don’t get confused later on

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y;};//@Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

[1/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with EXT?☐ Let’s now extend the Vector Type:

☐ Everything is fine as this extension is “adding to the tail” of our type

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y; long z;};

[2/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with EXT?☐ The resulting types are now:

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y; long z;};//@Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

☐ Wait a second... The monotonic-extension on the type Vector is leading to a non-monotonic-extension of the type Dynamics!

☐ We need a “more extensible” Dynamics Type!

[3/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Mutable X-Types (MXT)

☐ Mutable X-Types (MXT) provide the most generic form of extensibility. Attributes can be added in any order, and they can equally be removed

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with MXT?☐ As our extension to the Vector type is monotonic,

we can keep it as a EXT

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y;};

[1/3]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with MXT?☐ The Dynamics type needs to be MXT so to allow non-

monotonic extensions.

[2/3]

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y; long z;};//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with MXT?☐ The resulting type ensures that a DataReader of the old

version can safely project the new types on the old types

[3/3]

//@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY)struct Vector { long x; long y; long z;};//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

☐ Yet, there are still a few issues. Can you see them?

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Open Issues☐ The “legacy” DataWriter will not match DataReaders for the new type as

the Vector type would have to be “widened” as opposed to “projected”. This means that the 3D version of the game won’t see the flying objects on the 2D game. While the 2D will see a projection of the 3D objects

☐ If we change the extensibility of the Vector type to MUTABLE we still have an issue. How can we figure out, from the 3D application, when a “z” is not available? We need this information to be able to assign a arbitrary flight level to 2D UFOs as well as being able to distinguish when the “z” value is really “0”

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with MXT Revised☐ To address the issues just noted, we can simply change the Vector extensibility type, and

also tag the z attribute as Optional

☐ Optional attributes can have a special “nil” value that allow us to detect when the field was not set

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; long y; long z; //@Optional};//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key Vector pos; Vector speed;};

☐ As defined the types allow the level of interoperability required by our application. However, our type definition is a bit loose as we are not defining the strong invariants...

[1/2]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

2D to 3D with MXT Revised☐ For mutable types it is important to identify which attributes represent the

strong invariants of the type. These are the attributes that should always be present to allow assignability. These invariants are described via the MustUnderstand annotation

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) };//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3)};

☐ To allow attribute re-ordering/removal it is a good practice to explicitly assign attributes Ids, otherwise those will be deduced by the order of declaration thus impacting the extensibility of a type

[2/2]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Resulting Types//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Bounds { long width; //@MustUndertand @Id(1) long height; //@MustUndertand @Id(2) long depth; //@Optional @Id(3)};

//@Extensibility(MUTABLE_EXTENSIBILITY)struct FlyingObject { long oid; //@Key @Id(1) long kind; //@MustUndertand @Id(2) Bounds bounds; //@MustUndertand @Id(3)};

// @Extensibility(FINAL_EXTENSIBILITY)enum Classifier { UNKNOWN,FRIEND,THREAT};

// @Extensibility(FINAL_EXTENSIBILITY)struct Classification { long oid; // @key Classifier kind;};

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) };//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3)};

// @Extensibility(FINAL_EXTENSIBILITY)enum Classifier { UNKNOWN,FRIEND,THREAT};

// @Extensibility(FINAL_EXTENSIBILITY)struct Classification { long oid; // @key Classifier kind;};

NOTE: Some Final types to demonstrate their use

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Are we Done?

☐ Our game has become very popular and friends play with it across the internet

☐ However, some people are complaining that in some cases we are consuming a bit too much bandwith, especially when playing on narrow-band links

☐ How can we fix this problem?

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

N’ouvre la bouche que lorque tu es sur que ce que tu vas dire est plus beau que le silence.

[Proverbe Arabe]

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Is our Data Model too Chatty?☐ If we look at the Dynamics

type, it includes the position -- which will change regularly and relatively often -- as well as the speed vector -- which is subject to change less often (especially for UFOs)

☐ As a consequence of mixing the same topic-type attributes with different update rates, we are sending more data than we really should

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) };//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3)};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Frequency Normalization☐ To address this issue we can split the topic type into as many

topic types as necessary to ensure that attributes are updated at similar rates

//@Extensibility(MUTABLE_EXTENSIBILITY)struct FOPosition { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2)};

//@Extensibility(MUTABLE_EXTENSIBILITY)struct FOSpeed { long oid; //@Key @Id(1) Vector speed; //@MustUndertand @Id(2)};

☐ Notice that splitting the topic can lead to temporal inconsistencies, thus either (1) these are acceptable to your application or (2) we use coherent updates, or (3) we add sequence numbers to correlate the corresponding positions/speeds

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Delta Updates☐ This splitting does not feel

completely right, does it?

☐ It would be nice if the middleware could figure out what had changed and only send deltas

☐ Delta-updates are not currently available in DDS, these will be included in future versions of the standard

☐ Is there a way we can emulate delta-updates?

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) };//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3)};

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

Emulating Delta Updates☐ The @Optional annotation

can be used to emulate delta updates.

☐ This works since optional attributes set to nil are never sent on the wire

☐ This comes at a cost, as it requires some work -- ideally embedded into a layer sitting between the middleware and the application

//@Nested @Extensibility(MUTABLE_EXTENSIBILITY)struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) };//@Extensibility(MUTABLE_EXTENSIBILITY)struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Optional @Id(3)};

Ope

nSpl

ice

DD

S

Concluding Remarks

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

DCA Benefits☐ Loose Coupling

☐ Anonymicity (e.g. only things to be known are data types and not who produces/consumes them)

☐ Composability (especially when combined with functional languages)

☐ Extensibility

☐ Ease of Integration

☐ Performance (easier to parallelize)

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

My Advice on X-Types

Start using the X-Types annotation system immediately!

☐ DDS vendors are still working on the implementions of this specification, yet comment-style annotations //@ will be ignored by non-compliant IDL compilers

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

My Advice on X-TypesWhen in doubt, make outer types Mutable X-Types. Inner types are usually better candidates for Extensible X-Types (yet keep in mind our example)

☐ The main trade-off between Mutable and Extensible X-Types lies in the space/time efficiency. The latter is more efficient than the former.

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

My Advice on X-Types

Always identify @MustUnderstand attributes of Mutable X-Types. Then declare as @Optional whatever is not @MustUnderstand

☐ This will ensure that you’ll always be able to distinguish between default constructor values and nil

Copyrig

ht  2012,  PrismTech  –    A

ll  Rights  Reserved.

Ope

nSpl

ice

DD

S

My Advice on X-Types

Always explicitly define @Id for the attributes of Mutable X-Types

☐ This will ensure you’ll be to handle attribute re-ordering and removal

Ope

nSpl

ice

DD

S

Questions?