Slide 1

Yuan XueVanderbilt University

WISE 2010Assessing Security of Cyber-Physical SystemsOutlineIntroductionWhat is CPS/Example systemsSecurity Issues of CPSTools/Experiment Environment for CPS Security AssessmentRun-time: Integration of multiple tools/Environment Simulation, emulation, real testbed Modeling-time, rapid configuration/deploymentModel integrationStep I: Command and Control Wind Tunnel Heterogeneous simulation integrationStep II, Integration of DeterLab and C2WTSimulation and emulation integrationStep III Future DirectionsCyber-Physical Systems CPS has extraordinary significance for the future of the U.S. industry and military superiority. A 2007 report of the Presidents Council of Advisors on Science and Technology highlights CPS as the number one priority for federal investments in networking and information technology. Application Domains Health-CareAutomotive SystemsBuilding and Process ControlsDefense and Aviation SystemsCritical Infrastructure

Cyber-physical systems (CPS) are tight integrations of communications, computational and physical processes

Example: Electric Power GridCurrent picture:Equipment protection devices trip locally, reactivelyCascading failure: August (US/Canada) and October (Europe), 2003Better future?Real-time cooperative control of protection devicesOr -- self-healing -- (re-)aggregate islands of stable bulk power (protection, market motives)Ubiquitous green technologiesIssue: standard operational control concerns exhibit wide-area characteristics (bulk power stability and quality, flow control, fault isolation)Technology vectors: FACTS, PMUsContext: market (timing?) behavior, power routing transactions, regulation

IT LayerImages thanks to William H. Sanders, Bruce Krogh, and Marija IlicCPS BriefingNSF, May 10, 2007Raj Rajkumar, Carnegie Mellon University4Example: Health Care and MedicineNational Health Information Network, Electronic Patient Record initiativeMedical records at any point of serviceHospital, OR, ICU, , EMT?Home care: monitoring and control Pulse oximeters (oxygen saturation), blood glucose monitors, infusion pumps (insulin), accelerometers (falling, immobility), wearable networks (gait analysis), Operating Room of the Future (Goldman)Closed loop monitoring and control; multiple treatment stations, plug and play devices; robotic microsurgery (remotely guided?) System coordination challengeProgress in bioinformatics: gene, protein expression; systems biology; disease dynamics, control mechanisms

Images thanks to Dr. Julian Goldman, Dr. Fred PearceCPS BriefingNSF, May 10, 2007Raj Rajkumar, Carnegie Mellon University5CPS CharacteristicsCyber capability in physical componentSize and power of computational elementsNetworked at multiple and extreme scalesHigh degrees of automation, control loops must close at all scalesEnhance and leverage nature physical feedback at all levelssensing technologyactuation technologyHuman-System Interaction, human in the control loop

CPS BriefingNSF, May 10, 2007Raj Rajkumar, Carnegie Mellon University6Security Issues of CPSTrustworthiness of software and hardware for cyberphysical systems is an essential concern since such systems are routinely employed in critical settings. Existing systems are built without sufficiently formalized and analyzed properties and guarantees. many existing systems have builtin vulnerabilities which, once identified and exploited by attackers, can lead to catastrophic consequences. Most current systems cannot perform any selfdiagnostics to test whether they have been compromised. Even if attacks/intrusions are detected, existing systems cannot automatically contain, or heal themselves from consequences of, successful attacks.

7Exclusively rely on external intrusion detection systems, which are often unable to detect semanticsbased attacks

The delay incurred by the recovery task from deliberately launched adversarial attacks could be very significant, and thus catastrophic consequences occur.

7Core CPS Programmatic ThemesThere is a pressing need to design and evaluate both cyber- and physical systems (CPS) together and holisticallyScientific foundations for building verifiably correct and safe cyber-physical systemsScalable infrastructure and components with which cyber-physical systems can be deployedTools and Experimental TestbedEducation that encompasses both the cyber and the physical domainsCPS BriefingNSF, May 10, 2007Raj Rajkumar, Carnegie Mellon University8Tools for Design & Implementation of CPSLyapunov functions, eigenspace analysis, etc.Analytical ToolsMATLAB, MatrixX, VisSim, etc.,Software ToolsControl Design:Continuous Statedifferential equations, transfer functions, etc.ModelsBoolean algebra, formal logics, recursion, etc.Statemate, NS-2, OMNeT++, etc.Control Implementation:Discrete State/Eventsautomata, Petri nets, statecharts, etc.9Need for Security Assessment Tool and Experiment EnvironmentEvaluation of CPS security requires a sophisticated modeling and simulation, experiment infrastructure that allows for the concurrent modeling, simulation and evaluation of the CPS system architecture (advanced system-of-systems modeling)running environment (scenario modeling and generation)attack scenario (threat modeling and generation).This requires the integration at two levelsRun-time: Integration of multiple tools/Environment Simulation, emulation, real testbed so that they can interact in a coordinated way. Modeling-time: Model integrationrapid configuration/deployment

10Our ApproachStep I: Command and Control Wind Tunnel Heterogeneous simulation integrationStep II, Integration of DeterLab and C2WTSimulation and emulation integrationStep III Future Directionsrapidsynthesis of heterogeneous simulation environments using a model-basedapproach, where an overarching C2 modeling environment is developed basedon GME (Generic Modeling Environment) to integrated different platformspecific simulation models.11C2 Wind TunnelIntegration of multiple simulation toolsMatlab/Simulink, OMNeT++, DEVSJAVA, Delta3D, CPN, etc.Follow HLA standardCoordinate execution of distributed simulations via RTIRun-Time Infrastructure (RTI)C2 Wind Tunnel Integration FrameworkPassive FederatesData loggersMonitorsAnalysisPrognosticsProjectionsLive components-UAVs-Command & Control-Live deployment feedbackSimulation ToolsSimulinkOmnetDEVSJAVAOGRECPN ToolsJava/C/C++etc.Issues encountered in developing this environment relate to the integration of various simulation platforms, each with its own semantics, and integration of the models that execute on the platforms and the interactions between models. Integration frameworks, such as the High-Level Architecture (HLA) ([1], [11], and [25]) provide APIs that have helped to reduce the complexity of developing simulations that span multiple different platforms, but many challenges remain in developing and deploying these simulations. Pervasive use of models and integration at the domain-specific model level can decrease the effort required to integrate multiple platforms and increase the adaptability of the resulting environment, though this approach is still not without its own set of challenges.

The HLA provides a standard for a Run-Time Infrastructure (RTI) that supports the coordinated execution of distributed simulations. However, integration of the different platform-specific simulation models remains a challenging problem. Our project introduces a new model-integrated approach for the simulation integration problem. The primary contribution of this effort is the development of an overarching modeling environment, based on GME, and a suite of model transformations to synchronize between multiple platform-specific models.

12C2 Wind TunnelModel-integrated approachDevelop an overarching modeling environment based on GMEIntegrate different platform-specific simulation models

The "Command and ControlWind Tunnel (C2W)" tool suite addresses this urgent need and enables rapidsynthesis of heterogeneous simulation environments using a model-basedapproach, where an overarching C2 modeling environment is developed basedon GME (Generic Modeling Environment) to integrated different platformspecific simulation models.

Each simulation platform, when incorporated into the overall simulation environment, requires integration on three levels: on the API level, on the level of interactions, and on the level of model semantics. API level integration provides basic services such as message passing, and shared object management, while interaction level integration addresses the issues of time synchronization and coordination. Semantic integration is more subtle and depends upon the goals and the context of the overall simulation environment. The explicit use of models throughout the simulation environment gives rise to the use of model-based integration techniques. The C2 Wind Tunnel environment uses the metaprogrammable Generic Modeling Environment (GME) [2] engine for a tool chain to integrate the operational semantics of multiple simulation platforms, to manage the configuration and deployment of scenarios in the simulation environment, and to generate the necessary interface code for each integrated simulation platform. GME provides a platform for the graphical development of domain-specific modeling languages (DSML). These languages can capture both the structural and operational semantics of a domain-specific model. The C2 Wind Tunnel project has developed DSMLs that are based on the semantics of each integrated simulation platform and an overarching DSML to model the interactions between simulation models. We have written custom translators (model interpreters) to dynamically import and export platform-native models into the GME DSMLs and to generate HLA configuration files and glue code. These tools allow our environment to be rapidly reconfigured or extended, furthering the goal of rapid evaluation of emerging command and control concepts.

The composition and execution of the entire simulation is configured using the central modeling environment. A large set of well-known simulation tools is supported for building heterogeneous simulations.

We provide reusable run-time components to execute various domain-specific models. These components are configured from the modeling environment with a set of XML files. In some cases, both glue code and configuration files are generated that configure the simulation tool. Fig. 1 shows the general architecture of the model-based simulation integration approach. As the infrastructure was built for testing C2 concepts and architectures, we call the toolsuite the C2 Wind Tunnel (C2W). The virtual component models in the C2 Wind Tunnel are integrated using the GME environment. Model transformations, implemented via custom interpreters, generate the necessary configuration information and glue code. These pieces are incorporated into each of the different simulator platforms. At run-time, the platforms interact using the HLA infrastructure.

13Introducing Network Emulation Into C2WTMotivation to Introduce Network EmulationDesign Consideration and ChallengesOur Approach and SolutionCommunication architectureTime synchronizationIn collaboration with Timothy Busch (AFRL/RISB)

From Simulation to EmulationNetwork components and policies are essential aspects of CPSThe impact of network on CPS system need to be accurately characterizedThink about the network attacksLimit of network simulatorProtocol implementation details are missingPoor scalability

Network simulation is insufficient in providing the level of accuracy required by the evaluation of CPS.Network components and policies (such as security configurations) areessential aspects of C2 systems. Their impact on the performance and thebehavior of C2 systems need to be accurately characterized when evaluatingC2 systems. However, the current network simulators, which lack theimplementation details of network protocols and algorithms, areinsufficient in providing the level of accuracy required by the evaluationof C2 systems. This talk will present our work on introducing the networkemulation into the C2 Wind Tunnel environment as a solution to thisproblem. Network emulation environments, such as Emulab, allow the use ofreal network devices, components and systems, thus providing greaterrealism in the network experiment. Once integrated into the C2 wind tunnelsystem, the emulation environment also provides a platform where prototypesof C2 components can also be deployed and interact with other simulatedcomponents. 15From Simulation to EmulationBenefit of network emulationGreater realism and accuracy with truthful protocol implementation and real network traffic deliveryProviding a computing platform where prototypes of software components can be deployedNetwork emulation platformEmulab DETERNetLarge number of tools available for emulate network attacksNetwork components and policies (such as security configurations) areessential aspects of C2 systems. Their impact on the performance and thebehavior of C2 systems need to be accurately characterized when evaluatingC2 systems. However, the current network simulators, which lack theimplementation details of network protocols and algorithms, areinsufficient in providing the level of accuracy required by the evaluationof C2 systems. This talk will present our work on introducing the networkemulation into the C2 Wind Tunnel environment as a solution to thisproblem. Network emulation environments, such as Emulab, allow the use ofreal network devices, components and systems, thus providing greaterrealism in the network experiment. Once integrated into the C2 wind tunnelsystem, the emulation environment also provides a platform where prototypesof C2 components can also be deployed and interact with other simulatedcomponents. 16Design ConsiderationCommunication between simulated objects and real network objects

Time synchronization between simulated objects and real network objects

(1) Being a simulation environment, the C2W systemruns under the discrete event-driven time model and uses the RTI timemanagement; on the other hand, the network emulation platform runs incontinuous real time. The time synchronization issue between simulation andemulation environment is a non-trivial issue. (2) There is potentiallylarge volume of data communicated between the simulation and emulationenvironment. Controlling the communication overhead while still ensuringthe accuracy of the experiment is also a challenging issue. This talkpresents our approaches to these challenging issues and shows our modelsand algorithms developed for such system integration. 17Challenge in Data CommunicationKey Issue There is potentially large volume of data communicated between the simulation and the emulation environmentTradeoff between realism and performanceHow to control the communication overheadApproachesIdentify the communication platform (e.g., RTI, pub/sub service, socket, etc.)Control the application-level messagesDesign efficient transport-level protocols (e.g., reliable multicast). (2) There is potentiallylarge volume of data communicated between the simulation and emulationenvironment. Controlling the communication overhead while still ensuringthe accuracy of the experiment is also a challenging issue. This talkpresents our approaches to these challenging issues and shows our modelsand algorithms developed for such system integration.

18Challenge in Data CommunicationObservation -- Different types of dataCommand/Signal notification (E.g., Start to send, stop to send, change sending rate)Application Data/Payload (E.g., Images, videos)Our approachIdentify the appropriate communication platform for different types of data Define the appropriate granularity of communication data depending on the application semanticsCharacterize the communication semantics in the modeling phaseModel integration

. (2) There is potentiallylarge volume of data communicated between the simulation and emulationenvironment. Controlling the communication overhead while still ensuringthe accuracy of the experiment is also a challenging issue. This talkpresents our approaches to these challenging issues and shows our modelsand algorithms developed for such system integration.

19Challenge in Time SynchronizationKey IssueSimulated objects run in simulation time which is coordinated by RTI time management Network objects run in the user space of real operating systems and follow system time (usually real time)How to reconcile these two time modelsDesign considerationIdentify the appropriate time models for the integrated systemDesign the time synchronization algorithms

Difference From Existing WorksSimilarityCombining real network elements with simulated ones, each modeling different portions of a networked distributed systemFundamental DifferenceIn the existing work, the network is simulated, the application is real.In our work, the network is real, the application is simulated.Both require time synchronization. In our case, the network communication (e.g., packets in fly) can not be controlled. Need new design for simulation-emulation communication and time synchronization

ArchitectureRun-Time Infrastructure (RTI)Model Integration Layer Experiment Specification NetworkModelsController ModelsOrganization ModelsEnvironment ModelsFusionModelsEmulationFederate SimulinkFederate CPNFederate Delta3DFederate DEVSFederate Simulation PlatformSimulation-Emulation TunnelModelRun-timeNetworkApplicationsEmulation PlatformEmulabData communication Layer (TCP/IP)22Pros and ConsProsLimit the traffic load of RTIThe communication between simulated objects and real objects does not go through RTIFew code changes to simulatorsCons:The node that hosts the Emulation Gateway Federate may be comes a bottleneckAll traffic goes through Emulation Gateway FederateWe may use multiple instances of Emulation Gateway Federate and perform parallel simulation to solve this bottleneck issue

Pro: Easier extension to emulab where the network object is dynamically allocated to different hosts when the job is loaded.

Use Multiple OMNeT++ to solve the bottleneck issue23Meta-Model and ModelsNetwork Topology ModelNetwork Application Process Deployment and Communication ModelNetwork Interaction Model

Meta-Model for Network Topology

UAV1UAV2Access PointControl Station11M bps wireless link11M bps wireless link10M bps

Topology ModelBandwidth: 10MbpsDelay: 10msLoss: 0.02Capacity: 11MbpsPropagation Model: Free spaceMAC: IEEE 802.11Bandwidth: 2MbpsLoss: 0.2Delay: 20msBandwidth: 11MbpsLoss: 0.01Delay: 20msThe user first needs to define the network topology in the network model in GME. This describes the underlying network behavior.

Figure shows an example topology. In the case of mobile network, the wireless link will be formed dynamically. This model will be translated into topology/scenario script in network simulation tools (ns2, OMNeT++, or Emulab)26Deployment MetaModel

Deployment Model ExampleUAV1SendImageRecvCommandControlStationRecvImageSendCommandUAV2SendImageRecvCommandUDPUDPTCPTCPTCP Parameters: Non-periodic Time_to_send; Data = null; Periodic Interval, start_time;Fixed or variable

Connected mode UDP: Data = null;Packet size; s

Unconnected mode: required parameter: dest and port

28Deployment Model Example

Network Interaction MetaModel

ConnectionlessNetwork Interaction

30Network Interaction Model ExampleConnection-oriented UDPMessage driven no fixed packet size, interval, starting timeCommand: StringImage: URL/real data

SendCommandToNetworkNodeName: TBD (ControalStation)ProcName: SendCommandTimestamp:TBDParameter: Command (String)

RecvCommandFromNetworkNodeName: TBD (UAV1)ProcName: RecvCommandTimestamp: TBDPeerNodeName: TBD (ControlStation)PeerProcPort: TBDParameter: Command (String)

SendImageToNetworkNodeName: TBD (UAV1)ProcName: SendImageTimestamp: TBDParameter: ImageURL (String):

RecvImageFromNetworkNodeName: TBD (ControlStation)ProcName: RecvImageTimestamp: TBDPeerNodeName: TBD (UAV 1)PeerProcPort:TBDParameter: PacketDelay(double)

32Network Interaction Model More ExamplesCase II -- Connection-oriented UDPParameterized UDPParameter: frame size, frame intervalCase III Connectionless UDP

SendImageToNetworkNodeName: (to be filled by UAVFed)ProcName: SendImageTimestamp: TBDParameter: FrameInterval (double)Parameter: FrameSize (int)

RecvImageFromNetworkNodeName: TBD (ControlStation)ProcName: RecvImageTimestamp: TBDPeerProcPort: TBDPeerNodeName: TBD (UAV 1)Parameter: PacketDelay(double)

How SendImageToNetwork is processed:

EmuGateway federate: Upon receiving the network-related interactions (SendImageToNetwork), the Interaction Handler will parse the interaction into a tasks (ProcName (SendImage); Timestamp (when the image data should be sent); parameter (FrameInteral); parameter value(10sec) and pass it along with the NodeName to the Tap Server. The Tap Server will translate the NodeName to the Host IP address and send the task to the corresponding Emulation Host according to the Interaction Delivery Protocol.

The Emulation Host for UAV1: The Tap Client receives the task from Tap Server based on Interaction Delivery Protocol, converts the timestamp in the task from simulation time to real time (details in Sec ), preprocess the data if any (e.g., retrieve the data). The task will then wait in the buffer until its execution time indicated by its timestamp.

When the task executes, it will pass the parameter (ImageURL) and its parameter (http://10.0.0.1/imagefile) value to the ProcName (SendImage)SendImage process will construct UDP packets and send them to RecvImage.

Emulation Host for ControlStationWhen the RecvImage receives the UDP packet, it calculates the packet delay and generates a new task with ProcName (RecvImage); Timestamp (time instance when the packet is received); ParameterName (PacketDelay); ParameterValue (0.03ms). The task is passed to Tap Client, which converts its timestamp to simulation time and send it to Tap Server immediately.

At the Simulation Host:The Tap Server gets the task from the Emulation Host and convert the host IP address it to NodeName (ControlStation). The Tap Server constructs a local task using the task information and the NameName and put it to LocalTask buffer. When the timestamp of this local task is reached, the interaction handler will take the task , generate interaction (RecvImageFromNetwork) and publish it to RTI.

Task Class: ProcName (SendImage); Timestamp; ParameterName (ImageURL); ParameterValueLocalTask Class: HostName (ControlStation ) ProcName (RecvImage); Timestamp; ParameterName (PacketDelay); ParameterValue

34

SendImageToNetworkNodeName: TBD (UAV1)ProcName: SendImageTimestamp: TBDPeerProcPort: 7890PeerNodeName: TBD (ControlStation)Parameter: FrameInterval (double)Parameter: FrameSize (int)

RecvImageFromNetworkNodeName: TBD (ControlStation)ProcName: RecvImageTimestamp: TBDPeerProcPort: TBDPeerNodeName: TBD (UAV 1)Parameter: PacketDelay(double)

How SendImageToNetwork is processed:

EmuGateway federate: Upon receiving the network-related interactions (SendImageToNetwork), the Interaction Handler will parse the interaction into a tasks (ProcName (SendImage); Timestamp (when the image data should be sent); parameter (FrameInteral); parameter value(10sec)Treating PeerPorcPort and PeerNodeName as two parameters. and pass it along with the NodeName to the Tap Server. The Tap Server will translate the NodeName to the Host IP address and send the task to the corresponding Emulation Host according to the Interaction Delivery Protocol.

The Emulation Host for UAV1: The Tap Client receives the task from Tap Server based on Interaction Delivery Protocol, converts the timestamp in the task from simulation time to real time (details in Sec ), preprocess the data if any (e.g., retrieve the data). The task will then wait in the buffer until its execution time indicated by its timestamp.

When the task executes, it will pass the parameter (ImageURL) and its parameter (http://10.0.0.1/imagefile) value to the ProcName (SendImage)SendImage process will construct UDP packets and send them to RecvImage.

Emulation Host for ControlStationWhen the RecvImage receives the UDP packet, it calculates the packet delay and generates a new task with ProcName (RecvImage); Timestamp (time instance when the packet is received); ParameterName (PacketDelay); ParameterValue (0.03ms). The task is passed to Tap Client, which converts its timestamp to simulation time and send it to Tap Server immediately.

At the Simulation Host:The Tap Server gets the task from the Emulation Host and convert the host IP address it to NodeName (ControlStation). The Tap Server constructs a local task using the task information and the NameName and put it to LocalTask buffer. When the timestamp of this local task is reached, the interaction handler will take the task , generate interaction (RecvImageFromNetwork) and publish it to RTI.

Task Class: ProcName (SendImage); Timestamp; ParameterName (PeerPorcPort); ParameterValue(7890)LocalTask Class: HostName (ControlStation ) ProcName (RecvImage); Timestamp; ParameterName (PacketDelay); ParameterValue

35Time Synchronization OverviewTime SynchronizationSimulated objects run in simulation time which is coordinated by RTI time management Network objects run in the user space of real operating systems and follow system time (usually real time)How to reconcile these two time modelsRoadmapReview basic conceptsIdentify the appropriate time models for the integrated system Real timeAs fast as possibleDesign the time synchronization algorithms

Lets first review the basicsSlides are adapted from Dr. Fujimotos lecture notesTime Models in SimulationContinuous time simulationState changes occur continuously across timeTypically, behavior described by differential equationsDiscrete time simulationState changes only occur at discrete time instantsTime stepped: time advances by fixed time incrementsEvent stepped: time advances occur with irregular incrementscomputersimulationdiscretemodelscontinuousmodelseventdriventime-steppedstate variablessimulation timeevent driven executionstate variablessimulation timetime stepped executionc2wReal systemModes of ExecutionAs-fast-as-possible execution (unpaced): no fixed relationship necessarily exists between advances in simulation time and advances in wallclock timeReal-time execution (paced): each advance in simulation time is paced to occur in synchrony with an equivalent advance in wallclock timeScaled real-time execution (paced): each advance in simulation time is paced to occur in synchrony with S * an equivalent advance in wallclock timeSimulation Time = W2S(W) = T0 + S * (W - W0)W = wallclock time; S = scale factorW0 (T0) = wallclock (simulation) time at start of simulationSimulation Applicationmodels system behavior compute event and its time stamp event can modify state variables or schedule new eventsSimulation Executiveprocesses events in time stamp order manage event list manage advances in simulation time

calls toscheduleeventscalls to eventhandlersDiscrete Event Simulation System independent of the system modelDiscrete event simulation: - computer model for a system where changes in the state of the system occur at discrete points in simulation time.

Fundamental concepts: system state (state variables) state transitions (events)

dependent on the system modelParallel/Distributed Discrete Event SimulationEncapsulate each simulator in a logical process (LP)LP is capable of concurrent executionLogical processes can schedule events for other logical processesInteractions via message passingNo shared state variables

logicalprocessORDSFOJFKarrival10:00

time stamped event(message)Example: model a network of airportsTime SynchronizationSynchronization Problemensure each LP processes events in time stamp orderObservationAdherence to the local causality constraint is sufficient to ensure that the parallel simulation will produce exactly the same results as a sequential execution where all events across all LPs are processed in time stamp order.Simulation ApplicationSimulation ExecutiveEach can only process events locally

How to ensure that the events are processed in time stamp order globally? -> Many algorithmsSynchronization ImplementationSimulation ApplicationSimulation ExecutiveImplement the time synchronization algorithm.

Support the event scheduling/process advancing Two models Event oriented views Process oriented viewsEvent vs. Process Oriented Views1InTheAir := InTheAir + 1;2WaitUntil (RunwayFree); /* circle */3RunwayFree := FALSE;/* land */4AdvanceTime(R);5RunwayFree := TRUE;/* simulate aircraft on the ground */6InTheAir := InTheAir - 1;7OnTheGround := OnTheGround + 1;8AdvanceTime(G);/* simulate aircraft departure */9OnTheGround := OnTheGround 1;State variables

Integer: InTheAir;Integer: OnTheGround;Boolean: RunwayFree;Event oriented viewEntities modeled by event handlersState variables

Integer: InTheAir;Integer: OnTheGround;Boolean: RunwayFree;Process oriented viewEntities modeled by processes

In C2 Wind TunnelEvent oriented viewProcess oriented viewSimulation ApplicationSimulation ExecutiveHLAOMNeT++Simulation Application: C2W federatesSimulation Executive: Portico RTIOMNeT++ becomes a federateOMNeT++ scheduler communicates with RTI

Time Management in C2 Wind Tunnel

Simulation Application: C2W federatesSimulation Executive: Portico RTITwo Modes: real time as fast as possibleIn Federation Manager:

sleep_time = time_diffif (sleep_time >0)sleep(sleep_time);

next_time = time.getTime() + 0.1;timeAdvanceRequest (next_time); HLA Time management

Time AdvanceRequest(time-stepped mechanism)

NextEvent Request(event-driven federate)

AdvanceGrantGet back to our problemTime synchronization issue in the distributed simulation should be handled by simulation executive.C2W system follows the HLA standard (process-oriented view), where RTI handles the time synchronization. The simulation application calls the time management primitives. We do not deal with the time sync issue directly in C2W. Do we need to worry about it once the emulated network brings in? Why we talk about the time synchronization in distributed simulation?

In C2W, we do not care about deadlocks etc..Who handles the time sync issue (how much we need to worry about the issue?)47Get back to our problemQuestions to answerWhat mode Real time mode -- each synchronizes to real-time As fast as possible mode see next questionWho is handles the time synchronizationIn emulated systems, its system clock (not just the network object) needs to be synchronized. RTI does the job, each emulated system becomes a federate. Why we talk about the time synchronization in distributed simulation?

In C2W, we do not care about deadlocks etc..Who handles the time sync issue (how much we need to worry about the issue?)48Time SynchronizationReal Time ModeThe simulation (ts) and operation system (to) time are synchronized with real time (tr) : ts=to=trCurrently available in C2 Wind TunnelSynchronization is done separately by simulation/real system no need for coordinationIssue (see the example below)The propagation delay between the simulation/emulation environment introduces errors into the measurementSuch error will accumulate0physical time (tr)OS time (to)Simulation time (ts)t1UDP_sentt2= t1+d1UDP_sentt3= t2+tUDP_recvt4= t3+d2UDP_recvd2td149Synchronize To Real Time-- ChallengeSince simulation will receive events from emulation, simulation time should lag behind or equal to emulation time so that the events from emulation will not arrive at simulation in its past timeSince emulation will receive events from simulation, simulation time should lead or equal to emulation, so that the events from simulation will not arrive at emulation in its past timeWithout delay, simulation and emulation should be synchronized to the same time With delay in both directions, this is a non-trial issue.

TODO: The notation UDP_sent needs to be changed to the real function call.50Synchronize To Real TimeBasic IdeaSynchronize only OS time with real time (to=tr)Separate simulation time from real timeThe simulation environment should have at least a lag of (L >= d2) from real time to accommodate the communication delay emulation to simulation environment, if any incoming traffic is expected. (ts = tr- L)Simulation clock advances at the same pace as real physical clockAll the outgoing traffic event with time stamp t will be actually scheduled/tunneled to emulation environment at simulation time t - d1- L to compensate the delay from simulation to emulation and the lag between simulation and emulation so that it could arrive at emulation at real time t. For incoming traffic with time stamp (t+t), it will arrive at the simulation at simulation time ts = t-L+t+d2. Since L d2, the event can be scheduled at ts = t+t0Real time (tr)OS time (to)Simulation time (ts)ts =t-d1-LUDP_sentto =tUDP_sentto = t+tUDP_recvts= t+tUDP_recvd2td1L0ts = t-L+t+d2 TODO: The notation UDP_sent needs to be changed to the real function call.51Synchronize To Real TimeLimitation and ApplicationThe packet does not arrive at the emulated network on its timestamp time (t), only the measured delay information (t) from the emulation environment is correctly . Can not be used if the packet interacts with other existing traffic this is a serious constraint.If the simulation is slower than real time, then there is no easy fix. So it seems that synchronizing to real time has limited usageNot really. Depending on the value of L (d), the system may tolerate some inaccuracy. Using a model-based approach, the simulator could adjust its time management strategy based on the communication contextTime SynchronizationAs Fast As Possible ModeThe simulation (ts) and operation system (to) time are synchronized using a virtual clock(tv) : ts=to=tvCurrently in C2 Wind Tunnel simulation AFAP mode runs in virtual timeChallenge-- Reconcile two time models Real time, which flows naturally (not forced by a progression of events)Virtual time is adjusted by the progression of discrete events in the simulation systemWhy we need this mode? When real-time mode does not workSimulation can be slower53Synchronize To Virtual Time System VirtualizationNeed virtualization of the real systems/networks to control over their run-time behaviorThe execution of a virtual system/network needs to be stalled until the virtual clock proceeds As the system execution is interrupted due to the synchronization process, the internal clocks need to be manipulated to provide the virtual system/network a consistent and continuous time. Use Xen HypervisorThin layer between system hardware and the operating systemFacilitate the parallel execution Control the running behavior of OS on top of itSynchronize To Virtual Time Using RTISynchronization component is implemented in the privileged Xen Control domain as a federateUse a conservative approachDefine small slices to be target barrier of executionRequest RTI to advance time towards the barrier Once granted, release the scheduler to process the jobs with time stamplarger than the barrier

XenHardwareXen Control Domain

RTIVirtualhostschedulerVirtualhostNetworkObjectNetworkObjectSync Compon.Size of the slice determines theaccuracy of the synchronization

If the arrival pattern of the events/jobsIs known, we can pick the time slicebased on the statistical requirementof the experiment accuracy55Handling Packet in TransmissionKey Issue: Packet can not be stopped in the middle of the transmission or accelerated to meet the synchronized time A closer look of the problem what does it mean by in the middlePacket queued at a router. In Emulab, the routers are emulated by host. Queued packets are synchronized to virtual clock not a problemPacket transmitted along the link .The link delay of a packet is determined by its size and the bandwidth of the link this is a true problemIf arrive ahead of time, the real time propagation delay needs to be converted to virtual time and the packet reception will be scheduled later need a time keeping mechanismIf arrive behind the time, then there will be an issue. We need to adjust the bandwidth of the link (in Emulab ) to avoid this problemImplementation DetailsTopology ModelNetwork Interaction Model

Modeling EnvironmentModelInterpreter Run-Time EnvironmentTCL scriptConfiguration/Control EnvironmentHost AssignmentDeterlab Emulation EnvironmentC2WT Simulation EnvironmentTap ClientEmuGatewayFederateRTISimulinkFederateTap ClientTap ClientTap ServerFederatesInvolving network communicationNetwork File System

Network Application Code

Deployment Model

UAV federateControlStationfederate

EmuGateway federateTap ClientTap Client

Tap ServerHostMapNodeName: HostIPSendImageRecvImageUDPEmulation Host for UAV1Interaction Delivery ProtocolEmulation Host for ControlStationRecvCommandSendCommandTCPTaskbufferTaskbufferLocalTaskbufferInteraction HandlerEmulation EnvSimulation EnvTime converterRTITime converterHostMapHostMapHow SendImageToNetwork is processed:

EmuGateway federate: Upon receiving the network-related interactions (e.g.,SendImageToNetwork in our example), the Interaction Handler will parse the interaction into a task (ProcName (SendImage); Timestamp (when the image data should be sent); parameter (ImageURL); parameter value(http://) and pass it along with the NodeName to the Tap Server. The Tap Server will translate the NodeName to the Host IP address and send the task to the corresponding Emulation Host according to the Interaction Delivery Protocol.

The Emulation Host for UAV1: The Tap Client receives the task from Tap Server based on Interaction Delivery Protocol, converts the timestamp in the task from simulation time to real time (details in Sec ), preprocess the data if any (e.g., retrieve the data). The task will then wait in the buffer until its execution time indicated by its timestamp.

When the task executes, it will pass the parameter (ImageURL) and its parameter (http://10.0.0.1/imagefile) value to the ProcName (SendImage)SendImage process will construct UDP packets and send them to RecvImage.

Emulation Host for ControlStationWhen the RecvImage receives the UDP packet, it calculates the packet delay and generates a new task with ProcName (RecvImage); Timestamp (time instance when the packet is received); ParameterName (PacketDelay); ParameterValue (0.03ms). The task is passed to Tap Client, which converts its timestamp to simulation time and send it to Tap Server immediately.

At the Simulation Host:The Tap Server gets the task from the Emulation Host and convert the host IP address it to NodeName (ControlStation). The Tap Server constructs a local task using the task information and the NodeName and put it to LocalTask buffer. When the timestamp of this local task is reached, the interaction handler will take the task , generate interaction (RecvImageFromNetwork) and publish it to RTI.

Task Class: ProcName (SendImage); Timestamp; ParameterList (ParameterName (ImageURL); ParameterValue)LocalTask Class: NodeName (ControlStation ) ProcName (RecvImage); Timestamp; ParameterList( ParameterName (PacketDelay); ParameterValue)59Interaction Delivery ProtocolInitializationTap client has the socket address of server (IP address + port)Tap client registers itself to serverWhat is NodeName it emulates (UAV1)Tap server builds the HostMap tableNodeName to HostIPTap server provides the information to Tap cients which require the HostMap, tap client will put it in /etc/hosts.

Data communicationmessage formatProcNameTimeStampParemeterNameParemeterLengthParemeterValuerepeat.ENDSendImage10sImageURL1810.0.0.2/imagefile

ENDTask: ProcName (SendImage); Timestamp; parameter (ImageURL); parameter value60Integration With DeterLabSimulation FedsDETERLabExperiment Control PlaneExperiment PlaneReal time traffic generationData Loading & Collection PlaneNetwork LogData LoggerAttack traffic generationSimulation-Emulation Gateway61Our Experiment Setup100Mbps SwitchUAV Sim*SimulinkPhysics SimulationDelta3DCAOC Sim.CPN ToolsPythiaCaesarsDEVS/JavaOperator ConsoleDelta3DSimulink (UAV Control Algo.)Sensor SimulatorDelta3D

* Possibly one machine per UAVC2WindTunnelSimulated Applications

EmulabNetwork ObjectNetwork ObjectNetwork ObjectRealNetwork62Future WorkStep II: Integration with DeterLabEnable system virtualization, migrate the virtual clock into Xen HypervisorComponent allocationTime keeping at the emulated routersStep III: Integration with Experiment TestbedEvaluation of security policy on C2 systems

SummarySecurity of CPS is an essential concernBuilding a tool and environment for assessing the security impacts on CPS is a critical stepThree-step effort at VanderbiltSimulation integrationEmulation integrationReal network integration