a model-driven approach to trace checking of pattern-based temporal properties

.lusoftware verification & validationVVS

A Model-Driven Approach to Trace Checking of

Pattern-based Temporal PropertiesWei Dou, Domenico Bianculli, Lionel Briand

SnT Centre - University of Luxembourg

joint work with

3

Project Context

Centre des Technologies de l'Information de l'EtatCTIE

5

Model-driven Engineering at CTIE

6

Creation Date: 16/10/2010 Last Modification: 29/11/2011 (LMA) Version: 0.6 email: [email protected]: LAN

Based on some data, a decision will be made as for what to do next. Each branch is annotated with the condition under which the branch is taken. All conditions are exclusive, so that only one branch is taken. If all possible cases are not covered by explicit conditions, a default branch must be present. When merging, each incoming flow directly activates the outgoing flow without synchronization.

With a splitting inclusive gateway, each outgoing branch should show the condition under which the branch is taken. When the gate is activated, the conditions of all branches are evaluated and one or more branch(es) will be taken.When joining, inclusive gateways aresynchronizing : therefore, the branch coming out of the gateway is only activated when all active incoming branches have completed.

The Event-Based Gateway represents a branching point in the Process where the alternative paths that follow the Gateway are based on events that occur (message reception, timeout, signal, etc).When the first event is triggered, the path that follows that event will be used and all the remaining paths will no longer be valid.

The error end event indicates that a named error is generated and that all currently active threads in the particular subprocess are terminated.The error must be caught by using a catch error intermediate event on the boundary of the nearest enclosing parent activity.

COMMON PATTERNS PROCESS MODELLING METHOD

RULES and GUIDELINES

Data based exclusive decision

Data based inclusive decision

Event based decision

Error handling

1. All flow objects other than start events, boundary events, and compensating activities must have an incoming sequence flow.

2. All flow objects other than end events and compensating activities must have an outgoing sequence flow.

3. Message flows can only be used between pools.Sequence flows can only be used within a pool.

4. Only message events, pools and send/receive/user/service tasks can be used with messages flows.

5. A start event cannot have incoming sequence flow or outgoing message flow, but can have incoming message flow.

6. An end event cannot have incoming message flow or outgoing sequence flow, but can have outgoing message flow.

7. Each white box pool must have at least one start event and one end event. Multiple start events should be used sparingly.

8. A gateway is either « diverging» (one input, multiple outputs) or « converging » (multiple inputs, one output).

9. An activity must have exactly one input sequence flow and one output sequence flow.

10. Activity names start with a verb and contain a noun.

11. All tasks should have an activity type.

12. Only one outgoing path from exclusive, inclusive and complex gateways may be left unlabeled.

Level%1

1a.%Process%MapA"Process"Map"provides"a"structured"overview"of"all"processes. Level%1

1b.%Function%TreeA"Function"Tree"provides"a"structured"inventory"of"all"activities,"as"well"as"a"view"of"each"function’s"hierarchy.

3a.%BPMN%Function%Allocation%DiagramA"BPMN"FAD"can"be"used"to"describe"various"elements"of"the"process’"context"such"as"risks,costs,"assisting"roles,"KPIs,"etc.

2.%Value%Added%ChainOne"or"two"levels"of"Value"Added"Chain"can"be"used"to"link"processes"together"or"to"break"down"the"most"complex"processes.

Level%3 3b.%BPMN%Collaboration%diagramsProcesses"are"described"using"layers"of"BPMN"diagrams."BPMN"Collaboration"diagrams"are"used"to"describe"interactions"between"participants"and"for"highIlevel"and"intermediary"processes.

Level%4

4.%BPMN%Process%diagramsDetails"of"activities"performed"by"a"single"participant"are"described"at"lowIlevel"by"BPMN"process"diagrams."For"this"type"of"diagram,"all"elements"of"the"Collaboration"diagrams"can"be"used,"except"pools"and"message"flows"(as"only"one"participant"is"involved).

Level%2

BPMN%Conventions

Level%3

Credit: Centre des Technologies de l'Information de l'Etat

eGovernment Services

Business Processesdeveloped as

7

The Problem

8

Compliance of a business process with respect to its requirements

Compliance of a Business Process with respect to its Requirements

• Model checking

• Run-time monitoring and verification

• Offline trace checking

9

Compliance of a Business Process with Respect to its Requirements

Offline trace checking as a first step, and possible development,

towards run-time checking

10

Goal of the project

To develop a practical and scalable solution for offline trace checking of the requirements of an

eGovernment business process

11

Requirements (1)

• To be viable in the long term, any procedure shall rely on standard MDE technology (tools implementing OMG specifications)

• Why?

• our industrial partner has adopted MDE in practice

• requires any software solution added to the development process to adhere to OMG specifications and rely on the corresponding tools, for ease of integration with existing tools and solutions

• It can be generalized to other contexts in which solutions have to be engineered by using standard MDE technologies

12

Requirements (2)

• Any procedure shall:

• scale linearly with respect to the length of the trace

• complete within practical time limits

• e.g., a trace with millions of events should be checked within seconds

• to enable real-time log analysis, to promptly detect violations

13

Our contributions

• A scalable and practical solution, based on standard MDE technologies, for offline trace checking of temporal properties

• Tool: TemPsy-Check

http://weidou.github.io/TemPSy-Check/

14

http://weidou.github.io/TemPsy-Check/

Main Ingredients of Offline Trace Checking

15

2.2. Definition of TemPsy

hTemPsyBlocki ::= hTemPsyExpressioni+hTemPsyExpressioni ::= [‘temporal’ hIdi ‘:’]

hScopei hPatternihScopei ::= ‘globally’

| ‘before’ hBoundary1i| ‘after’ hBoundary1i| ‘between’ hBoundary2i

‘and’ hBoundary2i| ‘after’ hBoundary2i

‘until’ hBoundary2ihPatterni ::= ‘always’ hEventi

| ‘eventually’ hRepeatableEventExpi| ‘never’ [‘exactly’ hInti] hEventi| hEventChainExpi ‘preceding’

[hTimeDistanceExpi] hEventChainExpi| hEventChainExpi ‘responding’

[hTimeDistanceExpi] hEventChainExpihBoundary1i ::= [hInti] hEventi [hTimeDistanceExpi]hBoundary2i ::= [hInti] hEventi [‘at least’ hInti ‘tu’]

hEventChainExpi ::= hEventi(‘,’ [‘#’ hTimeDistanceExpi] hEventi)*

hTimeDistanceExpi ::= hComparingOpi hInti ‘tu’

hRepeatableEventExpi ::= [hComparingOpi hInti] hEventihComparingOpi ::= ‘at least’ | ‘at most’ | ‘exactly’

hEventi ::= hIdihIdi ::= hIdStartChari hIdChari*

| hIdi (hIdConnectori hIdi)*hIdStartChari ::= [A-Z] | ‘_’ | [a-z]

hIdChari ::= hIdStartChari | [0-9]

hIdConnectori ::= ‘.’ | ‘::’

hInti ::= [1-9] ([0-9])*

Figure 2.2: Syntax of TemPsy

boundaries (expressed with hBoundary1i or hBoundary2i) that denote a specific occurrence of anevent as a boundary, possibly with a time distance; notice that hBoundary2i represents a syntacticrestriction of hBoundary1i. Chains of events, used in precedence and response patterns, are definedas hEventChainExpi, which denotes a comma-separated list of events, possibly with a time distance(hTimeDistanceExpi) between each pair of events (denoted with the ‘#‘ symbol). Time distances areexpressed with an integer value, followed by the ‘tu’ keyword, which represents a generic time unit(i.e., any denomination of time).

11

specification language

offline trace checking procedure

Specification Language: TemPsy

Requirements Analysis of eGovernment Business Processes

• Analysis of the requirement documents of eGovernment business processes

• The majority of the requirements could be expressed as temporal properties

17

Specification of Temporal Properties

18

globally at most 2 P before R at most 2 P after Q at most 2 P between Q and R at most 2 P after Q until R at most 2 P

Temporal Logic

Domain-specific Language (DSL)

Requirements Specification Language

UsabilityExpressiveness

Efficiency of verification

19

Domain-specific Language for the Specification of Temporal Properties

20

TemPsy Temporal Properties made easy

21

Property Specification Patterns(Dwyer et al, 1999)

PatternScope

22

Property Specification Patterns

{GloballyBeforeAfterBetween-andAfter-until

Scope

23

Property Specification Patterns

PatternUniversalityExistenceAbsence

ResponsePrecedence{

24

Excerpt from TemPsy Grammar

25

2.2. Definition of TemPsy

hTemPsyBlocki ::= hTemPsyExpressioni+hTemPsyExpressioni ::= [‘temporal’ hIdi ‘:’]

hScopei hPatternihScopei ::= ‘globally’

| ‘before’ hBoundary1i| ‘after’ hBoundary1i| ‘between’ hBoundary2i

‘and’ hBoundary2i| ‘after’ hBoundary2i

‘until’ hBoundary2ihPatterni ::= ‘always’ hEventi

| ‘eventually’ hRepeatableEventExpi| ‘never’ [‘exactly’ hInti] hEventi| hEventChainExpi ‘preceding’

[hTimeDistanceExpi] hEventChainExpi| hEventChainExpi ‘responding’

[hTimeDistanceExpi] hEventChainExpihBoundary1i ::= [hInti] hEventi [hTimeDistanceExpi]hBoundary2i ::= [hInti] hEventi [‘at least’ hInti ‘tu’]

hEventChainExpi ::= hEventi(‘,’ [‘#’ hTimeDistanceExpi] hEventi)*

hTimeDistanceExpi ::= hComparingOpi hInti ‘tu’

hRepeatableEventExpi ::= [hComparingOpi hInti] hEventihComparingOpi ::= ‘at least’ | ‘at most’ | ‘exactly’

hEventi ::= hIdihIdi ::= hIdStartChari hIdChari*

| hIdi (hIdConnectori hIdi)*hIdStartChari ::= [A-Z] | ‘_’ | [a-z]

hIdChari ::= hIdStartChari | [0-9]

hIdConnectori ::= ‘.’ | ‘::’

hInti ::= [1-9] ([0-9])*

Figure 2.2: Syntax of TemPsy

boundaries (expressed with hBoundary1i or hBoundary2i) that denote a specific occurrence of anevent as a boundary, possibly with a time distance; notice that hBoundary2i represents a syntacticrestriction of hBoundary1i. Chains of events, used in precedence and response patterns, are definedas hEventChainExpi, which denotes a comma-separated list of events, possibly with a time distance(hTimeDistanceExpi) between each pair of events (denoted with the ‘#‘ symbol). Time distances areexpressed with an integer value, followed by the ‘tu’ keyword, which represents a generic time unit(i.e., any denomination of time).

11

Extensions toDwyer’s patterns system

26

Specific Occurrence of an Event in a Scope Boundary

“A password reset email will be sent to the user after the third incorrect login attempt.”

27

Constraint on the Time Distance from a Scope Boundary

“The card requester has to show up for the collection of her physical information within five days from the first notification.”

28

Constraint on the Time Distance between Two Events in Precedence and Response Patterns

29

“If an ID card is reported as lost, then a new card will be issued within five days.”

Constraint on the Number of Occurrences of the Events in Existence and Absence Patterns

30

“The notification of renewing an ID card should be sent at least twice before it expires.”

TemPsy Scopes

{GloballyBeforeAfterBetween-andAfter-until

Scope

31

Between-and Scope

between [n1-th] E1 [distance constraint] and [n2-th] E2 [distance constraint]

32

specific occurrence of an event in a scope boundary

Constraint on the time distance from a scope boundary

between 1 P and 2 Q at least 2 tu

33

Q P PQ Q Q P

2 time units

TemPsy Patterns

PatternUniversalityExistenceAbsence

ResponsePrecedence{

34

block1 preceding [distance constraint]

block2

35

Constraint on the time distance between two blocks

Example - Request

36

“Once a card request is approved, the applicant is notified within three days; this notification has to occur before the

production of the card is started.”

(Before + Response)

37

temporal R1:

before ICM.issueCard

ICM.notifyApproval responding at most 3*24*3600 tu ICM.approveRequest


production of the card is started”

Example - Request

38

temporal R1:

before ICM.issueCard

ICM.notifyApproval responding at most 3*24*3600 tu ICM.approveRequest


production of the card is started”

Example - Request

Offline Trace Checking of Temporal Properties

Offline Trace Checking of TemPsy Properties

40

CheckerTrace

TemPsy Properties

Approach Based on Off-the-shelf Trace Checker for Temporal Logic

41

Trace

TemPsy Properties

Off-the-shelf trace checker for

temporal logic

MonPoly, SOLOIST-ZOT

Approach Based on Off-the-shelf Trace Checker for Temporal Logic

42

Trace

TemPsy Properties

Temporal Logic MonPoly,

SOLOIST-ZOT

Off-the-shelf trace checker for

temporal logic

from the project requirements: “…solution based on OMG standards and stable MDE technology”

Our Approach for Offline Trace Checking of TemPsy Properties

Reducing offline trace checking of TemPsy properties to the evaluation of OCL constraints (semantically-equivalent to TemPsy properties) on an instance of a trace model

43

Trace Meta-Model

44

Trace TraceElementevent: String timestamp: Integer*properties:

Elist<TemPsyExpression>

traceElements

TemPsy Meta-Model

45

Chapter 3. Model-driven Offline Trace Checking of Temporal Properties

traceElements*

Trace

properties : EList<TemPsyExpression>

applyScopeGlobally(Scope):Elist<EList<TraceElement>>applyScopeBefore(Scope):Elist<EList<TraceElement>>applyScopeAfter(Scope):Elist<EList<TraceElement>>applyScopeBetweenAnd(Scope):Elist<EList<TraceElement>>applyScopeAfterUntil(Scope):Elist<EList<TraceElement>>checkPatternUniversality(EList<TraceElement>, Pattern):BooleancheckPatternExistence(EList<TraceElement>, Pattern):BooleancheckPatternAbsence(EList<TraceElement>, Pattern):BooleancheckPatternPrecedence(EList<TraceElement>, Pattern):BooleancheckPatternResponse(EList<TraceElement>, Pattern):Boolean

TraceElement

event : EStringtimestamp : EInt

Figure 3.1: Conceptual model for execution traces

*temPsyExpressions

1scope 1pattern

boundary1 boundaryBegin

1

1

boundaryEnd

0..1

timeDistance

1 event

timeDistance

0..1block21..*1..*block1

0..1

timeDistance

1

event

1event

TemPsyBlock

TemPsyExpression

Scope

type:ScopeType

⌧enumeration�ScopeType

GLOBALBEFOREAFTERBETWEENANDAFTERUNTIL

Global

UniScope

BiScopeBoundary

ordinal:EInt[0..1]

Pattern

type:PatternType

Universality

OccurrencePattern

comparisonOperator:ComparisonOperator[0..1]times:EInt[0..1]

OrderPattern

EventChainElement EventRepresentation

name:EStrison

TimeDistance

comparisonOperator:ComparisonOperatorquantity:EInt

⌧enumeration�PatternType

UNIVERSALITYEXISTENCEABSENCEPRECEDENCERESPONSE

⌧enumeration�ComparisonOperator

ATLEASTATMOSTEXACTLY

Figure 3.2: TemPsy model

type string, which represents the actual event recorded in the trace, and an attribute timestamp oftype integer, which indicates the time at which the event occurred. Class Trace contains also an at-tribute properties, which is a collection of TemPsyExpressions1, representing the propertiesto be checked on the trace.

We have defined some side-effect-free operations in OCL for the Trace class; these operationsconsist of two types of functions. The first type, of the form applyScope*S*, are named afterthe different types of scope (e.g., applyScopeBefore, applyScopeBetweenAnd) and returnsegment(s) of a trace (i.e., sub-traces) as determined by the parameters of the scope provided ininput. The second type, of the form checkPattern*P*, are named after the different types ofpattern (e.g., checkPatternExistence, checkPatternPrecedence) and check whether

1Class TemPsyExpression belongs to the meta-model of the language (not shown here for space reasons) andrepresents objects corresponding to the non-terminal hTemPsyExpressioni of the grammar shown in Figure 2.2.

28

Excerpt of TemPsy Meta-Model

ScopeTemPsyExpressionname: String

Pattern

1

1

46

Model-driven Offline Trace Checking of TemPsy Properties

47

Eclipse OCL

Trace

TemPsy Properties

Trace Instance

TemPsyExpression

Instances

OCL Constraints on Trace Class

TemPsy-Check

Simplified Template of OCL Constraints

context Trace

inv: let subtraces=applyScope*S*(scope) in

subtraces->forAll(subtrace |

checkPattern*P*(subtrace, pattern))

48


context Trace




49


context Trace




50

applyScope*S*

{ Globally

BeforeAfter

Between-andAfter-until

*S*

51

500 LO

C

between A at least 5 s and B at least 5 s

5s

C A A AB B B B

5s 5s5s

52

IterationapplyOriginalBetweenAnd(scope)

53

Find the next occurrence of event A

If event A has occurred, find the next occurrence of event B

Select the trace segment between the occurrences of event A and B

Append the trace segment to the output

Apply the time distance constraints on the boundaries, if either of them is not empty

checkPattern*P*

*P*

Universality

Existence

Absence

Response

Precedence{54

4,000

LOC

block1 preceding at most T tu

block2

55

IterationcheckPrecedenceAtMost(subtrace, pattern)

56

Find the latest occurrence of block1

If the first event of block2 occurs:

If t(firstOfBlock2) - t(lastOfBlock1) > T, report a violation if size(block2) = 1 or postpone until the entire block2 is found; otherwise ignore and continue the iteration

Or else if the next event of block2 occurs, report a violation if the entire block2 is found

TemPsy-Checkhttp://weidou.github.io/

TemPSy-Check/

57



Evaluation

Evaluation

• Real properties checked on synthesized traces

• Focus on scalability

• Comparison with MonPoly from ETHZ

59

Why Synthesized Traces?

1. Real traces are often inadequate to cover a large range of trace lengths and a variety of properties

2. We want to be able to control the diversity of occurrences of patterns in the traces

3. Real traces are not suitable for scalability evaluation, since they may contain faults

4. Real traces could not be shared for forming a public benchmark

60

Why MonPoly?

• Winner of the first International Competition on Software for Runtime Verification (CSRV 2014)

• Caveat: MonPoly supports MFOTL, which is more expressive than TemPsy

61

Properties (globally scope)

62

•P1: globally always A

•P2: globally never B

•P3: globally eventually at least 2 A

•P4: globally eventually at most 3 A

•P5: globally A responding at most 1000 tu B

•P6: globally A responding exactly 1000 tu B

•P7: globally A preceding at most 6000 tu B

•P8: globally A preceding at least 100 tu B

•P9: globally A preceding exactly 100 tu B

•P10: globally A, B preceding at least 1000 tu C, D

•P11: globally A responding at least 1000 tu B, C

•P12: globally A responding B

Research Questions (globally scope)

RQ-G1) What is the relation between the execution time of TemPsy-Check and the length of a trace?

RQ-G2) What are the types of pattern most taxing on the execution time?

RQ-G3) How does TemPsy-Check compare with MonPoly in terms of execution time?

63

Experiment Results (globally scope)

64

Aver

age e

xecu

tion

time

1s

2s

3s

Trace length100k 200k 300k 400k 500k 600k 700k 800k 900k 1.000k

P1 P2 P3 P4 P5 P6 P7 P8P9 P10 P11 P12

Answers to Research Questions RQ-G1 and RQ-G2

RQ-G1 TemPsy-Check scales linearly with respect to the length of a trace

RQ-G2 The precedence and response patterns that contain distance constraints are most taxing, e.g., P5, P6, P7, P9, P11

65

Aver

age e

xecu

tion

time

1s

2s

3s

Trace length200k 400k 600k 800k 1.000k

P1 P2 P3 P4 P5P6 P7 P8 P9 P10P11 P12

Comparison with MonPoly for Batch Checking of P3-P12

66

Aver

age e

xecu

tion

time

60s

120s

180s

240s

300s

360s

Trace length1M 2M 3M 4M 5M 6M 7M 8M 9M 10M

TemPsy-Check MonPoly

Comparison with MonPoly for Batch Checking of P3-P12

67

Aver

age e

xecu

tion

time

90s

180s

270s

360s

Trace length1M 2M 3M 4M 5M 6M 7M 8M 9M10M

TemPsy-CheckMonPoly

•TemPsy-Check performs similarly to or better than MonPoly in terms of batch execution time

Summing up

Future Work

• Run-time verification of temporal properties

• More informative output when violations are detected in a trace

70

.lusoftware verification & validationVVS

A Model-Driven Approach to Trace Checking of

Pattern-based Temporal [email protected]