large-scale system integration with dds for scada, c2, and finance
Post on 18-Dec-2014
690 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Real-TimeMiddleware Experts
Large-Scale System Integration with DDS for SCADA, C2, and Finance
Rick Warren, Principal Engineerrick.warren@rti.com
© 2009 Real-Time Innovations, Inc. 2
What do I mean by “large system”?
Systems of systems– Modular, hierarchical design– Legacy components,
subsystems– Multiple technologies– Global scale
Decoupled subsystem lifecycles– Independent development– Independent deployment and
use– Independent management– Independent revision and
retirement
Multiple communities of interest– Different data interest,
entitlements– Non-uniform levels of trust
LANLAN
DDS #1(Initech)
DDS #1(Initech)
JMS(Dunder Mifflin)
JMS(Dunder Mifflin)
DDS #2(Acme)
DDS #2(Acme)
Legacy(COBOL ‘R’
US)
Legacy(COBOL ‘R’
US)
Satellite Links
© 2009 Real-Time Innovations, Inc. 3
What matters to integrators of large systems?
Governance (including over security)– What information will be exchanged?– Under what conditions will it be exchanged?– Who is allowed to exchange this information?– If these SLAs are violated, can the exchange be prevented?
Can I be notified?– (In the past, what has occurred wrt these SLAs?)
Isolation– When I connect A and B, ensure they don’t break (each other)– When I disconnect A, ensure B doesn’t break– When I connect C, don’t change A or B
Scalability (like in any system,
Fault Tolerance but stakes are higher)
© 2009 Real-Time Innovations, Inc. 4
Part 1: Architecture
(we’ll get to technology later)
© 2009 Real-Time Innovations, Inc. 5
SystemSystem
Component
Component
Component
Class
A Modest Proposition
State
Behavior
Class
Fundamental design principles scale– Abstraction: Provide interface based on relevant concepts– Encapsulation: Hide internal implementation, communication– Composition: Combine existing capabilities into new ones
Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Schematic of a Composed System
Subsystems may have different network environments
Integration may have different network environment than subsystems themselves
Data may need to be transformed / cleansed as it moves among subsystems
Routing / gateway services will adapt data types / formats / protocols
LAN LANWAN
Router/Gateway
Router/Gateway
Isolation
Additional Governance
Schematic of a Composed System
Wash, rinse, repeat
Subsystem 1
P P
PP P
Subsystem 2
P P
PP P
Router/Gateway
Router/Gateway
Same data model?Same network env.?
Same lifecycle?
Behavior unaffected?Understandable?
Apples and Oranges
P P
PP P
Subsystem 1 + Subsystem 2
P P
PP P
Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Router/Gateway
Router/Gateway
© 2009 Real-Time Innovations, Inc. 9
Nothing New Under the Sun
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
© 2009 Real-Time Innovations, Inc. 10
Nothing New Under the Sun
U.S. CombatData Space
U.S. CombatData Space
Allied CombatData Space
Allied CombatData Space
U.S. C4IData Space
U.S. C4IData Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Defense Industry
© 2009 Real-Time Innovations, Inc. 11
Nothing New Under the Sun
NYC TradingData Space
NYC TradingData Space
London TradingData Space
London TradingData Space
Tokyo TradingData Space
Tokyo TradingData Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Financial Services Industry
© 2009 Real-Time Innovations, Inc. 12
Nothing New Under the Sun
GenerationData Space
GenerationData Space
Substation #1Data Space
Substation #1Data Space
Substation #2Data Space
Substation #2Data Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Power Industry
© 2009 Real-Time Innovations, Inc. 13
Scalability and Fault Tolerance
Fault tolerance: Router/gateway can’t be single point of failure. 3 answers:
– Persistent data survives system failures– Redundancy allows continued service during failure– Segmented/load-balanced configuration limits scope of failures
Scalability: How big can a subsystem be?1. How big a subsystem can you understand, test, and debug?2. How well does infrastructure scale?
© 2009 Real-Time Innovations, Inc. 14
Part 2: DDS
© 2009 Real-Time Innovations, Inc. 15
How Does DDS Stack Up?
Governance– Structural SLA: data type– Behavioral SLA: delivery QoS– Security– Problem
detection/prevention/notification– System monitoring and recording
Isolation– Prevent data “leaks” and side effects– Allow dynamic (dis-|re-)connection– Support independent integration and
evolution
Fault tolerance– Data persistence– Segmented/load-balanced routing– Redundant routing
Scalability– WAN connectivity– Peer-to-peer communication– Brokered/managed communication
– Built in– Built in– Products available; future standards– Built in
– Products available
– Built in: domains, partitions
– Built in– Built in, esp. w/ DDS-XTypes
– Built in– Built in if router uses DDS– Built in if router uses DDS
– Products available; future standards– Highly scalable– Protocol support if necessary
© 2009 Real-Time Innovations, Inc. 16
DDS-RTPS Supports Redundant Data Routing
WriteMyData
ReadMyData'
Transform and Forward
Router 1
Router 2
Multiple readers:
no problem
Multiple writers:prevent
duplicates!
Fortunately, built into
DDS-RTPS
© 2009 Real-Time Innovations, Inc. 17
DDS-RTPS Supports Redundant Data Routing
ReadMyData'
DDS-RTPS Message
Inline QoS
Data
Inline QoS
Orig. Writer Info
Other Metadata
Original Writer Info
Orig. Writer GUID
Orig. Writer Seq. #
Identity of original writer
Identity of original sample
Reader discards duplicates based on these
© 2009 Real-Time Innovations, Inc. 18
DDS-RTPS Supports Redundant Data Routing
ReadMyData'
Transform and Forward
Router 1
Router 2
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Data
Seq. # 42
Writer 5AC2
Writer 5AC2
Seq. # 42
Seq. # 42
If your impl. doesn’t support, fall back to ownership.
WriteMyData
© 2009 Real-Time Innovations, Inc. 19
DDS Scalability: Two Aspects
1. Application data– How does performance fall off as # participants, writers,
readers increase?– Any single writer/reader pair can saturate gig-E:
achieving high aggregate throughput is not main issue– Interesting issue: Fan-out – number of readers per writer
2. Discovery– How many DDS applications can discover one another?– Interesting issue: How many applications need to discover
one another?
Hard limits, if any, very dependent on DDS implementation, machine configuration, network, etc.
© 2009 Real-Time Innovations, Inc. 20
1. DDS Scalability: Application Data
DDS-RTPS reliable multicast scales at least: – …to hundreds of readers per writer– …with very little degradation in throughput.– Larger testing facilities welcomed.
< 15% decrease1~900 readers
RTI Data Distribution Service 4.3Red Hat EL 5.0, 32-bit2.4 GHz processorsGigabit EthernetUDP/IPv4Reliable ordered delivery
© 2009 Real-Time Innovations, Inc. 21
2. DDS Scalability: P2P Discovery Scenarios
Single domain, symmetric discovery– Everyone discovers everyone else– Easiest to configure; most challenging wrt scalability– RTI test results: 1,800 participants; 3.2M endpoint matches
Single domain, asymmetric discovery– Each participant only knows of certain others in its domain– Good for partitioning domains in which not everyone talks
Multiple domains– Greatest separation for subsystems that rarely exchange data
• DDS-RTPS maps to different IP ports
1
…
n
© 2009 Real-Time Innovations, Inc. 22
Summary: Large-Scale P2P Scalability
Experimental results: DDS-RTPS allows participant to talk to ≥ 1-2K others in single symmetric domain
Implication: Compose arbitrarily large systems out of subsystems this size or smaller
Large (sub)systems often have requirements for:– Increased governance, e.g. over security boundaries– Data transformation– Protocol mediation– These typically require data brokering anyway
© 2009 Real-Time Innovations, Inc. 23
DDS Scalability: Stronger Measures
Not big enough?
1. Brokered application data– Covered this
2. Brokered discovery
…
…
m
…
n
n < m
m
n
© 2009 Real-Time Innovations, Inc. 24
DDS Scalability: Brokered Discovery
Can be built with DDS-RTPS-standard building blocks
Application Topic
Built-In Topic
RTPS Metadata
Data
RTPS Metadata
Data
Orig. Metadata
RTPS Metadata
Data
RTPS InfoSource submessage provides data forwarding capability:“Treat the following as if it came from that guy instead of from me.”
© 2009 Real-Time Innovations, Inc. 25
Future Direction: Enhanced Built-in Security
System of systems often have non-uniform trust– Multiple communities of interest w/ different
entitlements/authorization– Infrastructure components may (not) be trusted
(e.g. brokers, daemons, persistence, recording/playback, etc.)– There is no “system high”
Desirable new specifications:– Fine-grained access control: topic, instance, type member
levels– Sample-level tagging and labeling– Fine-grained signing and/or encryption: topic, member level
© 2009 Real-Time Innovations, Inc. 26
Future Direction: Standardized WAN Support
Site-to-site comms need DDS across WAN(s)– Including untrusted networks– Including firewall, NAT traversal– Challenge: UDP may not be routable, especially if multicast
DDS-RTPS protocol designed for layeringatop diverse lower-level protocols– Proven: UDP, TCP, serial, switched fabric, Infiniband, VME,
shared memory…– Blessed for interop by OMG: UDP
Additional OMG-recognized protocol layeringscan improve site-to-site interop– TCP for WAN routing– (D)TLS for transport-level security (a clarification, not new)– Could be standardized quickly
© 2009 Real-Time Innovations, Inc. 27
Recap
Large systems usually composed of smaller systems– Developed independently– …With different network environments– …And different data interest/entitlements
Not an accident: hierarchical design best practice at every scale– Subsystems guarded by DDS router/gateway
• Restrict access to internal topics/flows• Mediate internal/external protocols• Transform/cleanse internal/external data types• Impose governance: enforce policy, attach monitoring
– Inter-router integration space is itself a “subsystem” for further hierarchical composition
© 2009 Real-Time Innovations, Inc. 28
Recap
DDS uniquely qualified to meet these needs– Strong SLAs for governance– Multiple topologies for fault-tolerant subsystem isolation– High-performance, scalable, deterministic interoperability– What other standard technology offers this combination?
(Not JMS. Not CORBA. Not AMQP. Not web services. …)
Promising future directions:– DDS-RTPS/TCP protocol layering for WAN traversal– Enhanced security model
• Fine-grained access control• Sample-level tagging, signing, and encrypting
© 2009 Real-Time Innovations, Inc. 29
Thank You
top related