unicorn: a unified approach for localizing non-deadlock concurrency bugs

UNICORN: A UNIFIED APPROACH FOR LOCALIZING NON-DEADLOCK

CONCURRENCY BUGS

Sangmin Park, Richard Vuduc, and Mary Jean Harrold

College of ComputingGeorgia Institute of Technology

2

Concurrency Bugs Technique Studies Conclusion

Interleaving Space

Motivation

Correct Interleaving

Buggy Interleaving involving multiple

resources

Buggy Interleaving involving single

resource

3


Existing Techniques

Detect concurrency bugs — single variable Data race detectors [FastTrack, Eraser]

Order violation detectors [Falcon]

Atomicity violation detectors [AVIO, Falcon, CTrigger]

Detect atomicity violations --- multiple variables Atomicity violation detectors [ColorSafe, AtomTracker]

Detect concurrency bugs --- single and multiple General bug detectors w/ statistical analysis

[CCI, Bugaboo, Recon, DefUse]

4


Existing Techniques




Detect atomicity violations --- multiple variables Atomicity violation detectors [ColorSafe, AtomTracker]



LimitationDon't detect concurrency bugs involving multiple variables

5


Existing Techniques




Detect atomicity violations — multiple variables Atomicity violation detectors [ColorSafe, AtomTracker]




6


Existing Techniques








LimitationDon't detect order violations

7






Detect concurrency bugs — single and multiple General bug detectors w/ statistical analysis



Existing Techniques


8






Detect concurrency bugs — single and multiple General bug detectors w/ statistical analysis



Existing Techniques


LimitationDon't provide bug type information

9


Overview

Concurrency Bugs Order violation Single-variable atomicity violation Multi-variable atomicity violation

UNICORN Technique Detects three types of concurrency bugs Provides bug type information

Empirical Studies Conclusion

10


Bug: Order Violation

void main(…) { … pthread_create(consumer);

…

…

mutex = NULL;

…

}

void * consumer (…) {

…

Lock(mutex);

…}

Thread 1 Thread 2

W1R2

* This example is from PBZip2.

11


Bug: Single-Variable Atomicity

void new_file(…) {

…

saved_type = log_type;

log_type = CLOSED;

…

// update file

…

log_type = saved_type;

…}

int mysql_insert(…) {

// actual insert

…

if (log_type != CLOSED)

{

LOG.write(…);

}

…

}

Thread 1 Thread 2

W3

W1 R2

* This example is from MySQL-791.

12


Bug: Multi-Variable Atomicity

int mysql_delete(…) {

…

Lock(X);

Table.remove ();

Unlock(X);

…

Lock(Y);

LOG.write();

Unlock(Y);

…}


…

Lock(X);

Table.insert();

Unlock(X);

…

Lock(Y);

LOG.write();

Unlock(Y);

…}

Thread 1 Thread 2

W4

W1 W2

W3


13


Bug: Multi-Variable Atomicity

int mysql_delete(…) {

…

Lock(X);

Table.remove ();

Unlock(X);

…

Lock(Y);

LOG.write();

Unlock(Y);

…}


…

Lock(X);

Table.insert();

Unlock(X);

…

Lock(Y);

LOG.write();

Unlock(Y);

…}

Thread 1 Thread 2

W4

W1 W2

W3


NoteThese three types of concurrency bugs are the most important types of non-deadlock concurrency bugs [Lu 08].

14


Problematic Access Patterns#Var Type Memory Access Patterns

Single

Order

R1,S(x) W2,S(x)

W1,S(x) R2,S(x)

W1,S(x) W2,S(x)

Single-VariableAtomicity

R1,S(x) W2,S(x) R1,S(x)

W1,S(x) W2,S(x) R1,S(x)

W1,S(x) R2,S(x) W1,S(x)

W1,S(x) R2,S(x) W1,S(x)

W1,S(x) W2,S(x) W1,S(x)

MultiMulti-Variable

Atomicity

W1,S(x) W2,S(x) W2,S(y) W1,S(y)

W1,S(x) W2,S(y) W2,S(x) W1,S(y)

W1,S(x) W2,S(y) W1,S(y) W2,S(x)

W1,S(x) R2,S(x) R2,S(y) W1,S(y)

W1,S(x) R2,S(y) R2,S(x) W1,S(y)

R1,S(x) W2,S(x) W2,S(y) R1,S(y)

R1,S(x) W2,S(y) W2,S(x) R1,S(y)

R1,S(x) W2,S(y) R1,S(y) W2,S(x)

W1,S(x) R2,S(x) W1,S(y) R2,S(y)

* The patterns were identified by previous work [Lu 06, Vaziri 06, Hammer 08].

15


UNICORN: Overview

P

T

ranked-patternsUNICORN

[ Fault-localization technique ]

16


UNICORN: Overview

P

TCollect Pairs

From Executions

pairs,outcomes

Step 1:Combine PairsInto Patterns

patterns,outcomes

Step 2:RankPatterns

ranked-patterns

Step 3:

[ UNICORN: A UNIfied approach for localizing non-deadlock COncuRreNcy Bugs ]

Intuition: Patterns consist of (one or two) pairs

Find both single-variable and multi-variable patterns

17


#Var Type Memory Access Patterns

Single

Order

R1,S(x) W2,S(x)

W1,S(x) R2,S(x)

W1,S(x) W2,S(x)


R1,S(x) W2,S(x) R1,S(x)

W1,S(x) W2,S(x) R1,S(x)

W1,S(x) R2,S(x) W1,S(x)

W1,S(x) R2,S(x) W1,S(x)

W1,S(x) W2,S(x) W1,S(x)

MultiMulti-

VariableAtomicity

W1,S(x) W2,S(x) W2,S(y) W1,S(y)

W1,S(x) W2,S(y) W2,S(x) W1,S(y)

W1,S(x) W2,S(y) W1,S(y) W2,S(x)

W1,S(x) R2,S(x) R2,S(y) W1,S(y)

W1,S(x) R2,S(y) R2,S(x) W1,S(y)

R1,S(x) W2,S(x) W2,S(y) R1,S(y)

R1,S(x) W2,S(y) W2,S(x) R1,S(y)

R1,S(x) W2,S(y) R1,S(y) W2,S(x)

W1,S(x) R2,S(x) W1,S(y) R2,S(y)

Problematic Access Patterns#Var Type Memory Access Patterns Memory Access Pairs

Single

Order

R1,S(x) W2,S(x) R1,S(x) W2,S(x)

W1,S(x) R2,S(x) W1,S(x) R2,S(x)

W1,S(x) W2,S(x) W1,S(x) W2,S(x)


R1,S(x) W2,S(x) R1,S(x) R1,S(x) W2,S(x) W2,S(x) R1,S(x)

W1,S(x) W2,S(x) R1,S(x) W1,S(x) W2,S(x) W2,S(x) R1,S(x)

W1,S(x) R2,S(x) W1,S(x) W1,S(x) R2,S(x) R2,S(x) W1,S(x)

W1,S(x) R2,S(x) W1,S(x) W1,S(x) R2,S(x) R2,S(x) W1,S(x)

W1,S(x) W2,S(x) W1,S(x) W1,S(x) W2,S(x) W2,S(x) W1,S(x)

MultiMulti-

VariableAtomicity

W1,S(x) W2,S(x) W2,S(y) W1,S(y) W1,S(x) W2,S(x) W2,S(y) W1,S(y)

W1,S(x) W2,S(y) W2,S(x) W1,S(y) W1,S(x) W2,S(x) W2,S(y) W1,S(y)

W1,S(x) W2,S(y) W1,S(y) W2,S(x) W1,S(x) W2,S(x) W2,S(y) W1,S(y)

W1,S(x) R2,S(x) R2,S(y) W1,S(y) W1,S(x) R2,S(x) R2,S(y) W1,S(y)

W1,S(x) R2,S(y) R2,S(x) W1,S(y) W1,S(x) R2,S(x) R2,S(y) W1,S(y)

R1,S(x) W2,S(x) W2,S(y) R1,S(y) R1,S(x) W2,S(x) W2,S(y) R1,S(y)

R1,S(x) W2,S(y) W2,S(x) R1,S(y) R1,S(x) W2,S(x) W2,S(y) R1,S(y)

R1,S(x) W2,S(y) R1,S(y) W2,S(x) R1,S(x) W2,S(x) W2,S(y) R1,S(y)

W1,S(x) R2,S(x) W1,S(y) R2,S(y) W1,S(x) R2,S(x) R2,S(y) W1,S(y)

18


Step 1: Collect Pairs in Executions

Collect pairs using sliding window [Falcon, ICSE 2010]

Maintain a window per shared variable (pair window) Collect WR, RW, WW access-pairs within window

P

TCollect Pairs

From Executions

pairs,outcomes


patterns,outcomes

Step 2:RankPatterns

Step 3:ranked-patterns

19



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2

Tbl:

LOG:

[Run 1] Failing

Accesses

Windows

Pairs

R0 R1 W0R0 R1 W0 W3 W4W1 W2

R0-W0, R0-W1

W1-W2

W3-W4

R0 R1 W0R0 R1 W0 W3 W4W1 W2

, R1-W0, R1-W1R0-W0, R0-W1

…

…

pairs for the code in the slide

20



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2

ID Pair

R1

R2

R3

R4

R5

R6

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

F P P F P P

ID Pair

R1

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

F

21


Step 2: Combine Pairs to Patterns

1. Set pairs to patterns for order violation

2. Combine pairs to patterns using sliding window Sort pairs in time order Combine two-pair-based patterns within pattern window

P

TCollect Pairs

From Executions

pairs,outcomes


patterns,outcomes

Step 2:RankPatterns


22



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2[Run 1] Failing

Pairs Patterns

R0-W0R0-W1R0-W0R0-W1R1-W0R1-W1R1-W2R1-W0R1-W1R1-W2W1-W2W3-W4

R0-W0R0-W1R0-W0R0-W1R1-W0R1-W1R1-W2R1-W0R1-W1R1-W2W1-W2W3-W4

W1-W2-W3-W4

23



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2 ID Pattern

R1

R2

R3

R4

R5

R6

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

F P P F P P

ID Pattern

R1

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

F

24


Step 3: Rank Patterns

Rank patterns Compute suspiciousness of patterns

Jaccard formula [Abreu07]

suspiciousness(P) =failed (P)

totalfailed + passed(P)

P

TCollect Pairs

From Executions

pairs,outcomes


patterns,outcomes

Step 2:RankPatterns


25



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2

ID Pattern

R1

R2

R3

R4

R5

R6 Susp

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

F P P F P P

ID Pattern

R1

R2

R3

R4

R5

R6 Susp

1W1 (Tbl)-W2 (Tbl) 0.5

2W3 (LOG)-W4 (LOG) 0.5

3W2 (Tbl)-W1 (Tbl) 0.0

4W4 (LOG)-W3 (LOG) 0.0

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

1.0

F P P F P P



26



int delete(…) {

…

Lock(X);

W1: Tbl();

Unlock(X);

…

Lock(Y);

W4: LOG();

Unlock(Y);

…}

int insert(…) {

…

Lock(X);

W2: Tbl();

Unlock(X);

…

Lock(Y);

W3: LOG();

Unlock(Y);

…}

Thread 1 Thread 2

ID Pattern

R1

R2

R3

R4

R5

R6 Susp

1W1 (Tbl)-W2 (Tbl)

2W3 (LOG)-W4 (LOG)

3W2 (Tbl)-W1 (Tbl)

4W4 (LOG)-W3 (LOG)

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

P F P F P P

ID Pattern

R1

R2

R3

R4

R5

R6 Susp

1W1 (Tbl)-W2 (Tbl) 0.5

2W3 (LOG)-W4 (LOG) 0.5

3W2 (Tbl)-W1 (Tbl) 0.0

4W4 (LOG)-W3 (LOG) 0.0

5

W1 (Tbl)-W2 (Tbl)-

W3 (LOG)-W4 (LOG)

1.0

F P P F P P



SummaryUNICORN • Detects memory access pairs• Combines pairs into patterns• Ranks patterns w.r.t. suspiciousness

27


Empirical Studies

Studies1. Evaluate effectiveness of the technique

2. Measure distance of the pattern window

3. Evaluate efficiency of the technique (see paper)

Empirical Setup Implemented in Java (Soot) and C++ (LLVM) Evaluated on a set of subjects

28


Subjects

Language Program LOC % Failed Bug Type

C++Extracted

TimerThread 68 14.4 Order

LoadScript 110 49.8 Single-variable atomicity

MysqlLog 89 2.8 Single-variable atomicity

JsString 95 1.4 Multi-variable atomicity

MysqlDelete 103 3.8 Multi-variable atomicity

MysqlSlave 94 0.4 Multi-variable atomicity

C++Complete

PBZip2 2k 2.8 Order

Mysql-791 372k 64.0 Single-variable atomicity

Aget 1.2k 49.0 Multi-variable atomicity

Mysql-169 334k 63.0 Multi-variable atomicity

JavaStringBuffer 1.4k 22.3 Multi-variable atomicity

Vector 9.5k 7.6 Multi-variable atomicity

29


Study 1: Effectiveness

Goal To determine how well UNICORN ranks the

pattern of the actual bug

Method Set pair-window size to 5 Set pattern-window size to 100 Ran each subject 100 times and computed

suspiciousness and rankings

30


Study 1: Effectiveness

ProgramNum of Pairs

Num of Patterns

Susp(Pair 1)

Susp(Pair 2)

Susp(Pattern)

Rank(Bug)

Num of Top rank

TimerThread 6 7 1.0 - 1.0 1 3

LoadScript 3 4 0.86 1.0 1.0 1 2

MysqlLog 4 5 0.02 1.0 1.0 1 2

JsString 5 8 0.04 1.0 1.0 1 3

MysqlDelete 7 1 0.04 0.27 1.0 1 1

MysqlSlave 6 11 0.01 0.66 1.0 1 1

PBZip2 59 333 0.42 - 0.42 1 6

Mysql-791 1082 10936 1.0 0.64 1.0 1 4

Aget 37 94 0.51 0.50 0.70 1 1

Mysql-169 894 9051 0.63 0.63 1.0 1 7

StringBuffer 8 18 0.58 1.0 1.0 1 3

Vector 11 25 1.0 1.0 1.0 1 4

1

2

3

31


Study 2: Pattern-window size

Goal To investigate the practicality of the pattern-

window size

Method Computed the distance between the two pairs of

the actual concurrency bugs (using data from Study 1)

Computed the median and maximum distances

32


Study 2: Pattern-window size

ProgramNum of

PairsNum of

PatternsMedian

distanceMaximum distance

TimerThread 6 7 N/A N/A

LoadScript 3 4 2 2

MysqlLog 4 5 2 2

JsString 5 8 3 3

MysqlDelete 7 1 2 2

MysqlSlave 6 11 2 2

PBZip2 59 333 N/A N/A

Mysql-791 1082 10936 18 27

Aget 37 94 2 3

Mysql-169 894 9051 41 41

StringBuffer 8 18 4 5

Vector 11 25 2 2

33


Future Work

Provide information to understand bugs Problem: Several patterns ranked top together Approach: In-depth analysis on the ranking

Provide information on test-suite Problem: No study on effectiveness vs. test suite Approach: Investigation on quality of test suite

Improve overhead Problem: Significant overhead on monitoring Approach: Use techniques such as sampling

34


Contributions

UNICORN is the first unified technique that detects memory-access pairs, combines pairs to

get patterns, and ranks patterns with respect to suspiciousness

handles both single- and multi-variable violations Empirical studies show that UNICORN

is effective in identifying patterns associated with real bugs

is efficient it needs small-/fixed-sized windows to find patterns its runtime overhead comparable to other techniquesQuestions

unicorn: a unified approach for localizing non-deadlock concurrency bugs

Technology

types of concurrency

order violation thread

order violations

int mysql

type w1 log

w3 log

atomtracker limitation

actual insert