tmpa-2017: dl-check: dynamic potential deadlock detection tool for java programs

Dl-Check:dynamicpotentialdeadlockdetectiontoolforJavaprograms

NikitaKoval1,2 DmitryTsitelov2 RomanElizarov2

1ITMOUniversity,ComputerTechnologyDepartment,St.Petersburg,[email protected]

2Devexperts,LLC,St.Petersburg,Russia,

[email protected]

TMPAConference,March2017

Tableofcontents

1. Introduction2. Algorithm3. Implementation4. Evaluation5. Conclusion

Slide2/65. Copyright©2017.DevexpertsLLC.Allrightsreserved.

INTRODUCTION

Multithreadedprogrammingproblems

• Datarace• Starvation• Livelock• Deadlock- Communicationdeadlock- Resourcedeadlock


Resourcedeadlockvoid transfer(Account from,

Account to,int amount) {

from.lock()to.lock()// Do transferunlock(from, to)

}transfer(a, b, …):1 a.lock()

4 b.lock()

transfer(b, a, …):

2 b.lock()3 a.lock()

DEADLOCK!


Analysisapproaches

• Staticanalysis- Canguaranteedeadlock-freedom- Manyfalsepositives

• Modelchecking- Canguaranteedeadlock-freedom- Requiresalotofcomputationalresources

• Dynamicanalysis- Dependsontheexecution- Producesfewfalsepositives- Applicableforlargeprograms


Dynamicanalysis

• Collectingexecutiontraces- Analyzescollectedtracesoff-line- e.g.JCarder,MulticoreSDK

• Detectingimmediatelywhenapotentialdeadlockhappens- Hasallcontextinformation,suchasstacktraces- e.g.VisualThreads


ALGORITHM

“Acquiredbefore”relation

• Lock𝒖isacquiredbefore lock𝒗(𝒖 → 𝒗) ifatsomepointlock𝒗isacquiredunderlock𝒖inthesamethread

Thread 1 Thread 21 a.lock()2 b.lock()

3 c.lock()4 d.lock()

𝑎 → 𝑏𝑎 → 𝑐𝑎 → 𝑑𝑏 → 𝑐𝑏 → 𝑑𝑐 → 𝑑


Lockhierarchy

• Lockhierarchy isapartialorderonlocks,where𝑢 → 𝑣 ⇒ 𝑜𝑟𝑑 𝑢 < 𝑜𝑟𝑑 𝑣

𝑜𝑟𝑑 𝑎 < 𝑜𝑟𝑑 𝑏𝑜𝑟𝑑 𝑎 < 𝑜𝑟𝑑 𝑐𝑜𝑟𝑑 𝑎 < 𝑜𝑟𝑑 𝑑𝑜𝑟𝑑 𝑏 < 𝑜𝑟𝑑 𝑐𝑜𝑟𝑑 𝑏 < 𝑜𝑟𝑑 𝑑𝑜𝑟𝑑 𝑐 < 𝑜𝑟𝑑(𝑑)


3 c.lock()4 d.lock()


Potentialdeadlock

• Lockhierarchyisaprimarymethodtoavoiddeadlocks• Potentialdeadlockisalockhierarchyviolation


3 b.lock()4 a.lock()

𝑎 → 𝑏𝑏 → 𝑎

𝑜𝑟𝑑 𝑎 < 𝑜𝑟𝑑 𝑏𝑜𝑟𝑑 𝑏 < 𝑜𝑟𝑑(𝑎)


Lock-ordergraph(1)

• Everyvertexisassociatedwithalock• Edge 𝒖, 𝒗 meansthat𝒖 → 𝒗• Cyclesrefertopotentialdeadlocks

Thread 1 Thread 21 a.lock()2 b.lock()3 unlock(a, b)

4 b.lock()5 a.lock()6 unlock(a, b)


Lock-ordergraph(2)





Lock-ordergraph(3)





Lock-ordergraph(4)





Minimizationprinciple:problem

• Twocyclesmayrefertothesamepotentialdeadlock

Thread 1 Thread 21 a.lock()2 c.lock()3 b.lock()4 unlock(a,b,c)

5 b.lock()6 a.lock()7 unlock(a,b)


Minimizationprinciple:problem(1)





Minimizationprinciple:rule

• Theshortercycleismoreuseful• Onlyoneshortestcycleistobeproduced


Algorithmlayout

• Capturelockacquireandreleaseoperations• Afterlockacquisition:- Addlocktothemultisetoflocksheldbythecurrentthread- Addnewedgestothelock-ordergraph- Reportnewcycles

• Beforelockrelease:- Removelockfromthemultisetofheldlocks


Lockacquisitionandreleasecapturing

Thread 1 Thread 2

4 a.lock()5 c.lock()

7 b.lock()8 a.unlock()9 c.unlock()10 b.unlock()

1 b.lock()2 a.lock()3 a.unlock()

6 b.unlock()

Thread#1

Thread#2


Lockacquisitionandreleasecapturing (1)

Thread#1

Thread#2

b

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

ba

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

b

a

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

b

ac

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

ac

b

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

acb

Thread 1 Thread 2




6 b.unlock()



Thread#2

Thread#1

cb

Thread 1 Thread 2




6 b.unlock()



Thread#1

Thread#2

b

Thread 1 Thread 2




6 b.unlock()


Multisetofheldlocks

• Commonlockusagepatternssynchronized(o) {// Do something

}

synchronized void f() {// Do something

}

l.lock()try {// Do something

} finally {l.unlock()

}

• LocksareacquiredandreleasedinLIFO(stack)order


Multisetofheldlocks

• Stackwithpossibilityofremovingfromthemiddle• Commonlyworksin𝑂 1• Worksin𝑂(𝐵) atworst- 𝐵 – numberoflocksacquiredbythecurrentthread


Topologicalorder

• Incrementaltopologicalordermaintenance- MNRalgorithmsuggestedbyMarchetti-Spaccamela etal.

• Theacyclicpartofthelock-ordergraphistobekept• Adding(𝑢, 𝑣) leadstotopologicalorderviolation⇒ theshortestpath𝑣 ↝ 𝑢relatestothecycle


Newnodesprocessing

• Typicalpatterninthelock-ordergraph

x

a

bc

d

e

f g

h

• Onelong-livedlockatthecenterandmanyhundredsofshort-livedlocksaroundit


Newnodesprocessing

• Lock-freebufferfornewnodes• Initializetopologicalordervalue- ifbufferisfull- whenoutgoingedgeshouldbeadded

Newlocknodesbuffer

Lock-ordergraph


Complexity• Onlockacquire- Ifacquisitiondoesnotproducenewcycles• Typicalcase• 𝑂(𝐵 + |𝑉| + |𝐸|)

- Newcycledetected• 𝑂 𝐵 ⋅ 𝑉 + 𝐸

• Onlockrelease- Removesthelockfromthelockmultiset• 𝑂 1 typically• 𝑂 𝐵 atworst


Limitations:cycleminimizationvoid transfer(Account from,



}1 transfer(a, c, …)2 transfer(c, b, …)3 transfer(b, a, …)4 transfer(a, b, …)


Limitations:cycleminimization(1)void transfer(Account from,





Limitation:cycleminimization(3)

• Edge(𝑣, 𝑢) hasformedacycle• Minimizecycle 𝑢,… , 𝑣 ⇔minimizepath𝑢 ↝ 𝑣- CanbeachievedviaBFS

• Usuallycyclelengthis2or3• Currentimplementationskipsthispart


Limitations:independentcycles

• Logicallyindependentcycles:- 𝑎, 𝑐, 𝑏- 𝑎, 𝑏

• Independentcycleisignoredduetominimizationprinciple• OneusefulcycleisbetterJ


Limitations:singlethreadedandguardedcycles

• Singlethreadedcycle

Lock a, b

1 a.lock()2 b.lock()3 unlock(a, b)




• GuardedcycleThread 1 Thread 2

1 g.lock()2 a.lock()3 b.lock()4 unlock(a, b)5 g.unlock()

6 g.lock()7 b.lock()8 a.lock()9 unlock(a, b)10 g.unlock()



• Theyarestilllockhierarchyviolation• Suchpotentialdeadlockmaybecomepossibleaftercoderefactoring


IMPLEMENTATION

Dl-Check

• Implementedasjavaagent- java -javaagent:dlcheck.jar -jar your_app.jar

- CanbeusedwithJunit: -Ddlcheck.fail=true

• Insertsanalysiscodeintoclassesatrun-time


Dl-Checkoutputexample==========================!!! Potential deadlock !!!==========================### Cycle in lock graph: ###Lock ReentrantLock@75915908 was acquired at:

com/intellij/ide/util/treeView/AbstractTreeUi.acquireLock(AbstractTreeUi.java:2464)com/intellij/ide/util/treeView/AbstractTreeUi.attemptLock(AbstractTreeUi.java:2460)

Lock ProjectViewTreeUpdater@46e87eab was acquired at:com/intellij/ide/util/treeView/AbstractTreeUpdater.reset(AbstractTreeUpdater.java:415)com/intellij/ide/util/treeView/AbstractTreeUpdater.getUpdateCount(AbstractTreeUpdater.java:344)com/intellij/ide/util/treeView/AbstractTreeUpdater.performUpdate(AbstractTreeUpdater.java:242)com/intellij/ide/util/treeView/AbstractTreeUpdater.isEnqueuedToUpdate(AbstractTreeUpdater.java:380)com/intellij/ide/util/treeView/AbstractTreeUpdater.cancelAllRequests(AbstractTreeUpdater.java:326)

### Current lock stack: ###Lock Object@5cbe2bca was acquired at:

com/intellij/openapi/application/impl/LaterInvocator$FlushQueue.a(LaterInvocator.java:320)Lock ReentrantLock@75915908 was acquired at:

com/intellij/ide/util/treeView/AbstractTreeUi.acquireLock(AbstractTreeUi.java:2464)com/intellij/ide/util/treeView/AbstractTreeUi.attemptLock(AbstractTreeUi.java:2460)

### Current stacktrace: ###com.intellij.ide.util.treeView.AbstractTreeUpdater.cancelAllRequests(AbstractTreeUpdater.java:326)com.intellij.ide.util.treeView.AbstractTreeUi.releaseNow(AbstractTreeUi.java:457)com.intellij.ide.util.treeView.AbstractTreeUi.access$1400(AbstractTreeUi.java:66)com.intellij.ide.util.treeView.AbstractTreeUi$12.perform(AbstractTreeUi.java:438)...


Byte-codeinstrumentation

• Captureslockacquireandreleaseoperations:- AfterMONITORENTER andbeforeMONITOREXIT instructions- Atthebeginningandtheendofsynchronizedmethod- java.util.concurrent.locks.Lock• Afterlockmethod• AftersuccessfultryLockmethodinvocation• Beforeunlockmethod


Scalability

• MNRalgorithmisNOT concurrent• Read-writelock- Readlockforread-onlyoperationsandcurrentnodemodification- Writelockfortopologicalordermodification- SWMR(single-writermulti-reader)datastructures


Memorymanagement

• JavahasaGarbageCollector• OldlockinstancesandnodesshouldbecollectedbyGC• WeakReference isusedforinternaldatastructures


EVALUATION

Benchmarks

• SpecJVM2008:ApacheDerby(4and40threads)• DaCapo:ApacheLucene• DaCapo:Bankingapplication• Fine-Grained(10threads,100and10’000locks)- Liketransfer(from, to, amount) fromexamples

• …


Memoryusageimpact

11,07 1,06

1

1,61

1,34

0,61

1,22

1 1

0,74

1,12

1,35

1

0

0,2

0,4

0,6

0,8

1

1,2

1,4

1,6

1,8

2

Derby1 Derby2 Lucene Bank FG1 FG2

INCR

EASEFA

CTOR

Dl-Check Jcarder MulticoreSDK


Performanceimpact

1,53 1,76

1,03

2,6

4,29

18,05

9,7

128,2

1,25

2,6

7,3

120,71

40,33

3,25

1

2

4

8

16

32

64

128

Derby1 Derby2 Lucene Bank FG1 FG2

SLOWDO

WNFA

CTOR

Dl-Check Jcarder MulticoreSDK


CONCLUSION

Summary

• Algorithmtodetectpotentialdeadlocksatrun-time• Dl-Check toolhasbeenimplemented• https://github.com/Devexperts/dlcheck• Futurework:- Lockgroupingfeature- Contractstodescribelockacquisitionrules


Thankyouforyourattention!


Dl-Check:dynamicpotentialdeadlockdetectiontoolforJavaprograms

NikitaKoval1,2 DmitryTsitelov2 RomanElizarov2

1ITMOUniversity,ComputerTechnologyDepartment,St.Petersburg,[email protected]

2Devexperts,LLC,St.Petersburg,Russia,

[email protected]

TMPAConference,March2017

Topologicalorder:extra• Canbemaintainedinasingleiterationincasenocyclesareformed• Alreadyacquiredlocksareorderedalready• Edgefromtherightestnodeshouldbeused


...1 a.lock()2 b.lock()3 c.lock()

...Topologicalorder

Topologicalorder:example(0)void transfer(Account from,



}

1 transfer(d, b)2 transfer(d, c)3 transfer(c, d)4 transfer(e, a)5 transfer(b, a)





}


Affectedregion
