pattern-based debugging of declarative models

Pattern-Based Debugging of

Declarative Models Vajih Montaghami and Derek Rayside {vmontagh,drayside}@uwaterloo.ca

Electrical and Computer Engineering University of Waterloo Waterloo, Ontario, Canada

Fri 2 Oct 2015

Pattern-Based Debugging of Declarative Models {vmontagh,drayside}@uwaterloo.ca

Say what you mean

Adapted from: http://cloudcomputingcell.com/computer-programmer-images/

sig Node{next:

Node}one sig Head

extends Node{}

pred StructuralConstr

aints{

all n: Node |

head

2


Say what you mean


sig Node{next:

Node}one sig Head

extends Node{}


aints{

all n: Node |

head

φintended

2


Say what you mean


sig Node{next:

Node}one sig Head

extends Node{}


aints{

all n: Node |

head

φexpressed φintended

2


Say what you mean


sig Node{next:

Node}one sig Head

extends Node{}


aints{

all n: Node |

head

⇎φexpressed φintended

2


Engineer intends to model a linear linked list


sig Node{next:

Node}one sig Head

extends Node{}

pred StructuralConstra

ints{

all n: Node | one

n.next}

head

3


Engineer intends to model a linear linked list


sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head head

3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head head

✗Underconstraint

But, what engineer expresses is not what he intended

3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head

∅

head

✗Underconstraint

But, what engineer expresses is not what he intended

3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head

∅

head

✗Underconstraint

Engineer Strengthens the constraints, but gets nothing!

✗Full-Overconstraint

3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head

∅

head

head

✗Underconstraint

Engineer Strengthens the constraints, but gets nothing!


3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head

∅

head

head

✗Underconstraint

Engineer Weakens the constraints, but excludes some examples


✗Partial-Overconstraint

3



sig Node{next:

Node}one sig Head

extends Node{}


ints{

all n: Node | one

n.next}

head

∅

head

head

✗Underconstraint

Engineer Weakens the constraints, and gets intended examples


✗Partial-Overconstraint

✓head

head

head

3

Pattern-Based Debugging of Declarative Models {vmontagh,drayside}@uwaterloo.ca4

Can we invent a tool to guide this strengthening

and weakening of the model?


Idea: Discriminating Examples

5

φintended

φexpressed



5

φintended

φexpressed example



5

φintended

φexpressed example discriminating example


Mean what you say

No bug φintended ⇔ φexpressed

Underconstraint φintended ⇒ φexpressed

Partial-overconstraint φintended ⇐ φexpressed

Fully-overconstraint ( ︎∄i: I| i ⊨ φexpressed) ∧ (∃j: I| j ⊨ φintended)

Loosely Intersecting Intention

∃i: I | (i ⊨ φintended ⋀ φexpressed) ⋀ (i ⊭ φintended ⇒ φexpressed) ⋀ (i ⊭ φexpressed

⇒ φintended)

6


Mean what you say

No bug φintended ⇔ φexpressed

Underconstraint φintended ⇒ φexpressed

Partial-overconstraint φintended ⇐ φexpressed

Fully-overconstraint ( ︎∄i: I| i ⊨ φexpressed) ∧ (∃j: I| j ⊨ φintended)

Loosely Intersecting Intention

∃i: I | (i ⊨ φintended ⋀ φexpressed) ⋀ (i ⊭ φintended ⇒ φexpressed) ⋀ (i ⊭ φexpressed

⇒ φintended)

6

φintendedφexpressed


Pattern-based debugging

Existing tools Pattern-based

Underconstraint Random examples

Minimal examples

Discriminating examples

Help the engineer decide if certain properties need to be strengthened or weakened

Overconstraint UnSAT-Core Subset of UnSAT-Core

Accepting or Rejecting discriminating examples localizes area to fix

Explanation None By Pattern Semantics

7

Better Understanding

Increase Localization

Best-effort Fix suggestion


Context❖ Model: A text written in a logical

language

❖ Bug: a divergence between the engineers intended logical model and the expressed logic model

8

sig Node{next: Node} one sig Head extends Node{}

pred StructuralConstraints{ all n: Node | one n.next}

pred Noloop{ all n: Node | n !in n.^next}

pred showLists{ StructuralConstraints Noloop}

run showLists

❖ Logical language: Alloy ๏ First-order relational logic with closure ๏ Alloy Analyzer: Instance finder equipped

with Inconsistent core finder in finite domain ๏ Bugs:

- Unintended instances - Inconsistent constraints


Generating Discriminating Examples

9

Engineer: expressed her intention φintended



9


φexpressed

Debugger: Infer Semantic

⇒ p[R] ⋀ …

P[R]: Pattern P applied to relation R



9


φexpressed


⇒ p[R] ⋀ …


Debugger: Take a guess that the model might be underconstrained

w.r.t. φintended

p’ [R]Weaken p to p’

p[R]⇒p’ [R] Strengthening is in reverse



9


φexpressed


⇒ p[R] ⋀ …



w.r.t. φintended



Debugger: Mutate the semanticp’ [R] ⋀ ¬p[R] ⋀ …



9


φexpressed


⇒ p[R] ⋀ …



w.r.t. φintended



Debugger: Mutate the semantic

Debugger: Find discriminating example

p’ [R] ⋀ ¬p[R] ⋀ …



9


φexpressed


⇒ p[R] ⋀ …



w.r.t. φintended



Debugger: Mutate the semantic

Debugger: Find discriminating example

p’ [R] ⋀ ¬p[R] ⋀ …

Engineer: accepts or rejects the discriminating

example


PatternsBinary relations

❖ Relational properties

❖ 21 are commonly known

❖ used in structural modelling

❖ e.g.: acyclic, antisymmetric, irreflexive, …

Ternary relations

❖ Inclusion and Ordering

❖ we define 160

❖ used in behavioural modelling

❖ e.g.: ExpandHeadOfRight, …(ece.uwaterloo.ca/~vmontagh/alloy/debugger/)

10


Pattern-based Debugger’s components

11


Pattern-based Debugger’s components:

UI: DiscriminatingExample

12

❖ Back to the Linked list example

❖ Debugger finds: φexpressed ⇒ Total_Function[next],…

❖ Debugger generates a discriminating example, w.r.t.:

E ⊨ ¬Total_Function[next] ⋀ Partial_Function[next]

Engineer accepts E φexpressed is overconstrained w.r.t. φintended

Fix by weakening

Engineer rejects E Generate another example

❌✓



Knowledge-base

Consistency Graph ❖ Undirected graph ❖ Nodes: Patterns ❖ Edges: Two nodes are consistent

over the same relation ❖ Nodes: 160 Edges: 6764

Implication Lattice ❖ Directed graph ❖ Nodes: Patterns ❖ Edges: The source node implies the

sink node over the same relation ❖ Sources:12 Sinks:6 Longest Path:6

Acyclic IrreflexiveAcyclic Partial-function

13



Reasoner

Generate discriminating examples

❖ Infer Semantic

❖ Semantic Mutation

❖ Generation tactics (see paper)

❖ antecedent satisfaction,

❖ disjunction vacuity detection

❖ conjunction vacuity detection

14

Localize the issue

❖ UnSAT-Core

❖ Correspondence tactics (see paper)

❖ quantifier unrolling

❖ function and predicate inlining


A case study ❖ Dijkstra’s Dining Philosophers algorithm

❖ The bogus model shipped with Alloy for years

❖ Was fixed in 2012 by team from KIT Germany

๏ Fix is subtle and was not explained in their publications

❖ In four interactions, the pattern-based debugger assists the engineer to

Understand, Localize, and Fix the defects

Source: wikipedia.org

15


Summary and Future workNow

❖ Pattern-based debugging

๏ Discriminating examples

๏ Library of ternary patterns

๏ Generation tactics

๏ Correspondence tactics

❖ Dining Philosophers case study

16

Next

❖ Automating search procedure

❖ Improve pattern library

❖ More case studies

pattern-based debugging of declarative models

Software