soi architecture group...•idioms and lemmas developed and documented for effective use of agree...

1

SoI Architecture Group03 Aug 2017

2

Agenda

• Group Description• Group Objectives• Group Accomplishments• Lessons Learned• Recommended Future Directions

Architecture Group

3

Group Description

• Primary goal: Develop a formal architecture for UxAS in an assume/guarantee framework

• Team Members• Sean Regisford – AFRL • Aaron Fifarek – LinQuest (LQ)/AFRL• Brian Hulbert – LinQuest (LQ)/AFRL• Jen Davis – Rockwell Collins (RC)• Chris Elliott – Lockheed Martin (LM)• Ratnesh Kumar – Iowa State University (ISU)• Cat McGhan – University of Cincinnati (UC)• Paul Meng – GE• Hao Ren – Honeywell (H)• Ben Rodes – Dependable Computing (DC)• Han Yu – GE

Architecture Group

4

Group Objectives

• Develop a formal architecture for the current state of UxAS in an assume/guarantee framework

• Identify strengths, weaknesses, and problems for the current architecture• Document decisions made along the way with supporting rationale• Begin to implement a revised formal architecture to mitigate challenges

A

B

CAssumption: Input < 20Guarantee: Output < 2*Input

Assumption: Input < 20Guarantee: Output < Input + 15

Assumption: noneGuarantee: Output = Input1 + Input2

Assumption: Input < 10Guarantee: Output < 50

Architecture Analysis

Architecture Group

5

Group Accomplishments

• UxAS formalization in AADL/AGREE :• Extracted data types, input and output ports, and connections from code (AFRL/LQ, RC)• Created formal contracts for nine services/tasks based on the OpenUxAS Wiki (DC, RC)• Stated and analyzed properties of these component-level contracts including timing aspects (DC, RC, LQ)• Idioms and lemmas developed and documented for effective use of AGREE (DC, RC, LQ/AFRL)

Architecture Group

6



• UxAS formalization in ASSERT:• Created an ontology of the system (GE)• Examined contractual obligations of a representative component (GE)

Architecture Group

7




• Enhanced tool (RELIQE) to compose component contracts in AGREE into a system contract (H, ISU)

Architecture Group

8





• Created an assurance argument for state machine correctness (DC)

Architecture Group

9





• Created an assurance argument for state machine correctness (DC)• Abstracted a task service finite state machine in Stateflow for simulation and formal

analysis (LM)

Architecture Group

10






analysis (LM)• Demonstrated LMCP (UxAS messaging format) <--> ROS communications and extracted

static message types into ROS .msg format (UC)

Architecture Group

11






analysis (LM)• Demonstrated LMCP (UxAS messaging format) <--> ROS communications and extracted

static message types into ROS .msg format (UC)

We improved UxAS through formalization

Architecture Group

12

Developing the UxAS Formal Representation

13

AADL Tools: Architecture-Driven Assurance

TrustedBuild

Architecture Translation

seL4eChronosVxWorks

A

B

CAssumption: Input < 20Guarantee: Output < 2*Input

Assumption: Input < 20Guarantee: Output < Input + 15

Assumption: noneGuarantee: Output = Input1 + Input2

Assumption: Input < 10Guarantee: Output < 50

Architecture AnalysisArchitecture Models

OSATE

Resolute

Assurance Case

AGREEBehavioral Analysis

SIM

Simulation

Kind/JKindModel CheckerSMT Solvers

Rockwell Collins, Dependable Computing, AFRL/LinQuest Architecture Group

14

Formalization in AADL/AGREEFormalize Services

• Wrote a script to extract data types, input/output ports, and connections from code into AADL• Abstracted the common message bus by direct

port connections• The structural representation of the Waterways

mission is nearly complete

• Created formal contracts (captured behaviors) for nine services and tasks based on the OpenUxAS Wiki and conversations with the lead developer

Waterways system level AADL connections diagram


15Architecture Group

Formalization in AADL/AGREEUxAS Formal Model Status

Add legible headers for all sections

Services

Data Types

Logging and Data Services

Communication Services

Rockwell Collins, Dependable Computing, AFRL/LinQuest

16

Communication Services

Logging and Data Services

Architecture Group

Formalization in AADL/AGREEUxAS Formal Model Status

Add legible headers for all sections

Services

Tasks

Data Types

Scenarios


17

Formalization in AADL/AGREEWhat Can We Prove?

RealizabilityChecks that a component implementation is possible and that there are no conflicts amongst the guarantees in the component’s contract

Assume/guarantee reasoningChecks higher-level guarantees and lemmas using the guarantees of the subcomponents, and checks that the assumptions of the subcomponents hold

• Check that all input/output events are possible• Capture response times in components and analyze system-

level response times• Analyze behavior of state machines that represent

component behavior

Some analysis results for the Route Aggregator Service (Aggregator Role):


Observers

Architecture Group

18

Formalization in AADL/AGREEWhat Did We Learn?

• Idioms and lemmas developed and documented for effective use of AGREE• “Eventual” behavior on state transitions• Tracking and maintaining IDs• Dummy parent components for proving lemmas• Preventing inadvertent solutions space constraints• Abstraction of complex data structures• Skolemization idea for arrays• Fully specifying output event conditions• Use of AGREE connections• Real-time patterns


19

System-Level Contracts

• System-level guarantees are in progress

Responsiveness/timing example

Task Manager Service:

Waterways (system-level) – not yet proven:

Architecture GroupRockwell Collins, Dependable Computing, AFRL/LinQuest

20


• System-level guarantees are in progress• Challenges


21



• Many system-level requirements for UxAS are informal


22




• Others require significant infrastructure (supporting contract guarantees) in order to prove them


23




• Others require significant infrastructure (supporting contract guarantees) in order to prove them

• Perhaps we can compose the component contracts we have to discover new system-level properties…


24

RELIQE (REduced Logic Inference using Quantifier Elimination): Framework for Compositional Reasoning

Architecture Group 15

Strongest System Property: the system property that implies any other system properties established upon thecomponent properties and their connectivity.

Quantifier Elimination: Derive quantifier-free formula equivalent to a quantified formula, eg, ∃𝑥𝑥: 𝑦𝑦 ≥ 𝑥𝑥2 ≡ (𝑦𝑦 ≥ 0)

𝑃𝑃𝑠𝑠𝑦𝑦𝑠𝑠= ∃ 𝑥𝑥1 …∃𝑥𝑥𝑚𝑚 �𝑖𝑖=1

𝑁𝑁

𝐴𝐴𝑖𝑖 ⇒ 𝐺𝐺𝑖𝑖 ∧ �𝑝𝑝,𝑞𝑞 ∈𝐶𝐶

𝑥𝑥𝑝𝑝 = 𝑥𝑥𝑞𝑞

(Conjunct component contracts and connectivity constraints; eliminate variables internal to system.)

Component variables: 𝑥𝑥1, … , 𝑥𝑥𝑚𝑚, 𝑥𝑥𝑚𝑚+1, … , 𝑥𝑥𝑛𝑛System variables: 𝑥𝑥𝑚𝑚+1, … , 𝑥𝑥𝑛𝑛Component contracts: 𝐴𝐴𝑖𝑖 ⇒ 𝐺𝐺𝑖𝑖 , 𝑖𝑖 = 1, … ,𝑁𝑁Connectivity set: 𝐶𝐶 = {(𝑝𝑝, 𝑞𝑞)|𝑥𝑥𝑝𝑝 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐 𝑥𝑥𝑞𝑞 }

Theorem: The strongest system property, established upon the component contracts and connectivity is given by:

Notation: 𝑆𝑆𝑦𝑦𝑐𝑐 ≔ 𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝 𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝1, … ,𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝𝑁𝑁 ;

∀ 𝑥𝑥𝑚𝑚+1 …∀𝑥𝑥𝑛𝑛 𝑃𝑃𝑠𝑠𝑦𝑦𝑠𝑠⇒ 𝑃𝑃𝑝𝑝𝑝𝑝𝑠𝑠𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ≡ 𝑐𝑐𝑡𝑡𝑡𝑡𝑐𝑐

Use Strongest System Property to prove postulated system property:

25

RELIQE (REduced Logic Inference using Quantifier Elimination): Framework for Compositional Reasoning

Architecture Group 15

Strongest System Property: the system property that implies any other system properties established upon thecomponent properties and their connectivity.

Quantifier Elimination: Derive quantifier-free formula equivalent to a quantified formula, eg, ∃𝑥𝑥: 𝑦𝑦 ≥ 𝑥𝑥2 ≡ (𝑦𝑦 ≥ 0)

𝑃𝑃𝑠𝑠𝑦𝑦𝑠𝑠= ∃ 𝑥𝑥1 …∃𝑥𝑥𝑚𝑚 �𝑖𝑖=1

𝑁𝑁

𝐴𝐴𝑖𝑖 ⇒ 𝐺𝐺𝑖𝑖 ∧ �𝑝𝑝,𝑞𝑞 ∈𝐶𝐶

𝑥𝑥𝑝𝑝 = 𝑥𝑥𝑞𝑞

(Conjunct component contracts and connectivity constraints; eliminate variables internal to system.)

Component variables: 𝑥𝑥1, … , 𝑥𝑥𝑚𝑚, 𝑥𝑥𝑚𝑚+1, … , 𝑥𝑥𝑛𝑛System variables: 𝑥𝑥𝑚𝑚+1, … , 𝑥𝑥𝑛𝑛Component contracts: 𝐴𝐴𝑖𝑖 ⇒ 𝐺𝐺𝑖𝑖 , 𝑖𝑖 = 1, … ,𝑁𝑁Connectivity set: 𝐶𝐶 = {(𝑝𝑝, 𝑞𝑞)|𝑥𝑥𝑝𝑝 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑐𝑐𝑐𝑐 𝑥𝑥𝑞𝑞 }

Theorem: The strongest system property, established upon the component contracts and connectivity is given by:

Notation: 𝑆𝑆𝑦𝑦𝑐𝑐 ≔ 𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝 𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝1, … ,𝐶𝐶𝑐𝑐𝐶𝐶𝑝𝑝𝑁𝑁 ;

∀ 𝑥𝑥𝑚𝑚+1 …∀𝑥𝑥𝑛𝑛 𝑃𝑃𝑠𝑠𝑦𝑦𝑠𝑠⇒ 𝑃𝑃𝑝𝑝𝑝𝑝𝑠𝑠𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ≡ 𝑐𝑐𝑡𝑡𝑡𝑡𝑐𝑐

Use Strongest System Property to prove postulated system property:

AGREE contracts can be composed into a system-level contract

26

Tool RELIQE: Implementation and Illustration

Architecture Group

RELIQE uses AADL for system architecture, AGREE annex for component contracts, REDLOG for Quantifier Elimination

EclipseAGREEOSATE2

Extended-AGREE

Redlogresult.aadl Redlog model

Result parser

RELIQE

input output

Integer domain

Reals domain

Motivation from UxAS context for extension to Time-dependent Property Composition: • Properties of Route-Aggregator Service vs Route-Planner-Visibility Service involve time-dependent properties• Extension to Time-dependent properties completed under SoI, more details presented under Hybrid Systems

group (our compositional reasoning framework applies not only to Cyber components, but also to Physical components)

27

Formalization in ASSERTProposed Approach

Software Codebase(C, C++, Java, C#, Python, Ada, …)

Reverse Engineering

Software Design Information

Architecture Information Extraction

Automated Ontology

Generation

Architecture & Ontology Mapping

TemplateOntology Model

ASSERT™ SADL Ontology files

(.sadl files)

*SADL: Semantic Application Design Language

• Extract architecture information through reverse engineering.

• Transform extracted architecture information into SADL ontology.

• Import SADL ontology into ASSERT™ to enable requirement capture, analysis, and test generation in ASSERT™

GE Architecture Group

28

ASSERT™ Requirement Analysis Checks

Architecture Group

• Contingency: Contingency analysis fails for requirement r if it is provably false (unrealizable) or true (redundant) under all value assignments to variables of r.

• Conflict: requirements r1 and r2 that set a common controlled variable v, if for some input value of the monitored variables r1 assigns v to a different value than r2

• Completeness: A set of requirements with the same controlled variable is complete if there is a requirement for all values of the monitored variables in the set.

• Subjectivity: controlled variable v with respect to requirements R fails if there is some output value for v (in the type domain of v) that is not assigned (set) by any requirement in R.

• Independence: Independence Analysis for requirement r with respect to requirements R fails if the conjunction of 'when' parts of R implies the 'when' part of r and both R and r set the controlled variable of r to the same value.

29

Modeling UxAS System Architecture in SADL Ontology

Architecture GroupGE

30

Modeling Message Type Definition in SADL Ontology


31

Modeling Dataflow in SADL Ontology

Context: TaskAssignmentSummary__Subscription of PlanBuilderService is

TaskAssignmentSummary_out of AssignmentTreeBranchBoundBaseandTaskImplementationRequest__Subscription of TaskServiceBase is

TaskImplementationRequest_out of PlanBuilderServiceand…

Illustrative Example in SADL Ontology

PlanBuilderService

AssignmentTreeBranchBoundBase

TaskServiceBase

TaskImplementationRequest_out

TaskImplementationRequest_Subscription

TaskAssignmentSummary_out

TaskAssignmentSummary_Subscription


32

Illustrative Example: Sample AADL Contracts Captured in SADL


33

Illustrative Example: Contracts Analysis

Conflicts!A counter example: num_route_requests_being_serviced = 1561

routePlanRequest_In of RouteAggregatorService isreceived

previous state of RouteAggregatorService is IDLE

state of RouteAggregatorService could be either PENDING or IDLE!


34

Formalization in ASSERTIllustrative Example: Modified Contracts Pass Analysis

• What we learned: errors introduced in the manual contracts/requirements capturing process can be found earlier through automated analysis.

GE Architecture Group

35

From the lead developer: Impact of Formalization on UxAS

• Recognition of key assumptions• IDs are unique (not guaranteed by the implementation today)

• Illumination of design flaws• Missing ID checks• All services need to use the Route Planner for routes to ensure there is no conflict with

an airspace constraint

• Restructuring to avoid redundant or problematic logic• Split Route Aggregator Service into two services for two functional roles• Removal of some dead code in the Route Planner

We improved UxAS through formalization

Architecture Group

36

State Machine Reasoning

37

State Machine for the Route Aggregator Service

Architecture Group

IDLE

PENDING


38

State Machine Reasoning in AGREE

• Example properties that we can prove:• States are reachable

• Transitions are viable

• Even so, the (human) modeler can make mistakes• We use an assurance case to capture the rest of the argument


39

State Machine Correctness Argument

• Argument pattern for individual AADL component state machine correctness:• Structure repeated for each component• Vision: Situated within a larger model

correctness argument

• Argument situates proposed AGREE lemmas:• Purpose• Rationale• Limitations and benefits

Dependable Computing

40

Stateflow Representation

• Created a Stateflow representation of the Task Service Base state machine• Benefits

• Evidence in the assurance argument: “gold standard state machine”• Enable simulation• Enable analysis from Simulink

• QVtrace• Simulink-to-PVS• Simulink Design Verifier

• Visualize counterexamples• Simulink/Stateflow abstractions cross-checked with AADL/AGREE model

• Found erroneous initial conditions, unused from/goto flags, and incorrect settings on if-else cases in Stateflow model

Lockheed Martin, Dependable Computing, Rockwell Collins Architecture Group

41

TaskServiceBaseentry: TaskServiceOn = 1

Processing Time for RouteAggregatorService - AGGREGATOR ROLE (Consider Random Fail on Receipt)

FinalRoutes : Upon reception of a TaskImplementationRequest, a Task is informed of the option that was selected by theassignment service. At this point, a Task must create the final set of waypoints that include both enroute and on-task

waypoints from the specified vehicle location. The Task is required to create the enroute waypoints since a routerefinement is possible, taking advantage of the concrete prior position of the selected vehicle. The Task remains in theFinalRoutes state until the route request is fulfilled by the RouteAggregatorService

.

FinalRoutes

Variable Delay a function ofAREA Search ParametersMATRIX OPTIMIZATION here Discussion w/ DerekROUTE COLLECTOR

OptionSelected : When the final waypoints are returned from the RouteAggregatorService, the Task publishes a completeTaskImplementationResponse message. A Task will remain in this state until an EntityState message includes this Task IDin its AssociatedTaskList. If during this state, a subsequent UniqueAutomationRequest is made, the Task returns to theSensorRequest state and immediately attempts to fulfill the requirements of the new UniqueAutomationRequest. This

behavior implies that a Task can only be part of a single AutomationRequest and subsequent requests always overrideprevious requests.

OptionSelected

Simple Delay (Consider Closed Loop onTask Execution)

Active: If the Task is in the OptionSelected state and an EntityState message is received which includes the Task ID inthe AssociatedTaskList, then the Task switches to the Active state and is allowed to publish new waypoints and sensorcommands at will. A Task remains in the Active state until a subsequent EntityState message does not list the Task ID inits AssociatedTaskList. At which point, a transition to Completed is made. Note that a Task can reliquish control indirectlyby sending the vehicle to a waypoint not tagged with its own ID. Likewise, it can maintain control indefinitely by ensuringthat the vehicle only ever go to a waypoint that includes its ID. If a UniqueAutomationRequest message that includes thisTask ID is received in the Active state, it transitions to the Completed state.

Active

TaskImplementationRequest (2ms) (Consider Random Fail on Receipt)

OptionsPublished : When routes are returned to the Task, it will utilize all route and sensor information to identify andpublish the applicable TaskOptions. The determination of TaskOptions is key to overall mission performance and vehiclebehavior. It is from this list of options that the assignment will select in order to perform this particular Task. After

publication of the options, a Task waits in the OptionsPublished state until the TaskImplementationRequest message is

received, whereupon it switches to FinalRoutes.

OptionsPublished

Simple Delay (Consider Random Fail onTaskInit)

Init: This is the state that all Tasks start in and remain until all internal initialization is complete. For example, a Task may need to load complex terrain or weather data upon creation and will require some (possibly significant) start-up time. When a Task has completed its internal initialization, it must report transition from this state via the TaskInitialized message.

guarantee "all tasks remain in init until all internal initialization is complete, transitioning to idle after. When aÂ TaskÂ has completed its internal initialization, it must report transition from this state via theÂ TaskInitializedÂ message."

Init

Processing Time for RouteAggregatorService - COLLECTOR ROLE (Consider Random Fail on Receipt)

OptionRoutes : After the SensorManagerService has replied with the appropriate sensor calculations, the Task can request

waypoints from the RouteAggregatorService that carry out the on-Task goals. For example, an AreaSearchTask can request routesfrom key surveillance positions that ensure sensor coverage of the entire area. The Task remains in the OptionRoutes state untilthe RouteAggregatorService replies.

guarantee "[Not originally specified: if in the OPTION_ROUTES state and an expected RouteResponse_in is received send anevent(TaskPlanOptions_out)]"

guarantee "TheÂ TaskÂ remains in theÂ OptionRoutesÂ state until theÂ RouteAggregatorServiceÂ replies. When routes are returned [IMPLIED: from the route aggregator service the service will be in theOPTIONS_PUBLISHED state] [ASSUMPTION: the replies are assumed to be RoutePlanResponse_in]"

OptionRoutes

Processing Time for SensorManagerService (Consider Random Fail on Receipt)

SensorRequest : When a Task is notified of its inclusion (by noting the presence of its ID in the Tasks list of an

UniqueAutomationRequest message), it can request calculations that pertain to the sensors onboard the vehicles that arealso included in the UniqueAutomationRequest message. While waiting for a response from the SensorManagerService, aTask is in the SensorRequest state and will remain so until the response from the SensorManagerService is received.

guarantee "[Not originally specified: in the SENSOR_REQUEST state, if the expected SensorFootprintResponse_in isreceived send a RouteRequest_out]"

guarantee "[IMPLIED: in the SensorRequest state] After theÂ SensorManagerServiceÂ has replied with the appropriatesensor calculations [IMPLIED: the service will then be in OptionRoutes state]" :

SensorRequest

UniqueAutomationRequest (2ms) (Consider Random Fail on Receipt)

Idle: This represents the state of a Task after initialization, but before any requests have been made that include the Task.

UniqueAutomationRequest messages trigger a transition from this state into the SensorRequest state.

guarantee "[Not originally specified: in the idle state, once an automation request is recieved that is present in the set of expected task IDs, send a SensorFootprintRequests_out]"

guarantee "UniqueAutomationRequestÂ messages trigger a transition from the idle state into theÂ SensorRequestÂ state. When aÂ TaskÂ is notified of its inclusion (by noting the presence of its ID in theÂ TasksÂ list of anÂ UniqueAutomationRequestÂ message), it can request calculations that pertain to the sensors onboard the vehicles Â Â

Idle

[RouteRequest_out]

[TaskImplementationResponse_out]

[TaskPlanOptions_out]

[TaskComplete_out]

[SensorFootprintRequests_out][init_complete && TaskInitialized_out]

[SensorFootprintRequests_out]1

[RouteRequest_out]

[EntityState_in && ...entity_state_task_id_present]

2

42

Analysis with QVtrace

• Matlab Simulink Construction for Simulation QVtrace

• QVtrace is a model checker for Simulink that uses SMT Solvers such as Z3

• Helped validate expected behavior of the abstraction

Lockheed Martin, Dependable Computing, Rockwell Collins

Enables path from contracts to code generation

Architecture Group

43

Lessons Learned

• Many problems are identified through the process of formalization, before analysis

• Data types, ports, and connections (or an ontology) can be automatically extracted from code (with some moderate up front work)

• Reusable (template) properties help find common mistakes• Not every error can be detected with a template property--a review of the

specification itself is still worthwhile• Assurance arguments are useful to capture the “rest of the story”• SpeAR, ASSERT, AADL/AGREE, and QVtrace are all very capable analysis tools.

There are synergies between them.• Challenges: arrays, inheritance, polymorphism

Architecture Group

44

Recommended Future Directions

• Write contracts for the remaining thirty-one components

• State/compose and analyze system-level properties (in progress)

• Generate code from AADL

• Generate test cases from AGREE and run on UxAS implementation

• Continue documenting decisions and the assurance case for the architecture

Architecture Group

45

Backup

46

Formalization in ASSERTSADL Ontology Generation

Modeling Message Type Definition

Architecture Group

Context: TaskAssignmentSummary__Subscription of PlanBuilderService isTaskAssignmentSummary_out of AssignmentTreeBranchBoundBaseandTaskImplementationRequest__Subscription of TaskServiceBase isTaskImplementationRequest_out of PlanBuilderServiceand…

PlanBuilderService

AssignmentTreeBranchBoundBase

TaskServiceBase

TaskImplementationRequest_out

TaskImplementationRequest_Subscription

TaskAssignmentSummary_out

TaskAssignmentSummary_Subscription

Modeling Dataflow in SADL Ontology

GE

47

Argument-Derived Observations

• Best practices/standards for AADL/AGREE use are needed• Situate proofs within modeling fault analysis/mitigation• Supports arguments that the set of lemmas are complete and correct

• Additional tool support to reduce human error• E.g., lemmas and repetitive infrastructure are developed manually• Minimize repetitive concerns in the argument about human error

• Enhanced integration between AADL and AGREE • Minimize repetitive arguments about AADL and AGREE consistency/agreement

• A simpler higher-level protocol state machine spec for validation• Provide a structure to validate against• Simplify validation activity and simplify validation arguments

Dependable Computing Architecture Group

soi architecture group...•idioms and lemmas developed and documented for effective use of agree...

Documents