chess: a systematic testing tool for concurrent software csci6900 george
TRANSCRIPT
![Page 1: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/1.jpg)
CHESS: A Systematic Testing Tool for Concurrent Software
CSCI6900
George
![Page 2: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/2.jpg)
Outline Introduction
Using CHESS
Capturing nondeterminism
Exploring nondeterminism
Evaluation
Related work
![Page 3: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/3.jpg)
Introduction Concurrent programming is difficult
Testing concurrent program is difficult Famous concurrency bug: “Heisenbugs”
Most strategy used in finding concurrency bugs Stress testing
![Page 4: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/4.jpg)
Introduction What is “Concurrency testing”
Concurrency scenario testing
Model checking techniques
Three key challengers by applying model checking
Perturbation problem Rich and complex concurrency APIs in system State-space explosion.
![Page 5: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/5.jpg)
Introduction What is “CHESS”
Concurrency scenario testing
Model checking techniques
Scale to large concurrent program
Reproduce an erroneous execution
Been integrated into the test framework inside Microsoft
![Page 6: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/6.jpg)
Introduction How does CHESS solve the previous key
problems?
Testing methodology of CHESS reduce the perturbation
Only one perturbation in CHESS
Wrappers to provides enough hooks
Variety of techniques to address the state-explosion
![Page 7: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/7.jpg)
Contributions of this paper The first system for integrating model
checking into concurrent systems and test frameworks with minimal perturbation;
Techniques for systematic exploration of systems code for fine-grained concurrency with shared memory and multithreading;
![Page 8: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/8.jpg)
Contributions of this paper Validation of the CHESS tool and its
accompany testing and wrapper methodology on three different platforms;
A substantial number of previously unknown bugs, even in well-tested systems;
The ability to consistently reproduce crashing
bugs with unknown cause.
![Page 9: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/9.jpg)
Using CHESS Traditional testing methodology
Disadvantages of this testing methodology
![Page 10: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/10.jpg)
Using CHESS CHESS architecture
![Page 11: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/11.jpg)
Using CHESS Testing methodology in CHESS
![Page 12: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/12.jpg)
Using CHESS To guarantee the previous advantages, CHESS
has to control the scheduling of the tasks
Two ways to control it
Modify the scheduler
Severely limited the deployment
![Page 13: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/13.jpg)
Using CHESS Testing methodology in CHESS
![Page 14: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/14.jpg)
Using CHESS Two assumption of about the testing program
Quiescence Definition of “Quiescence State”
How does CHESS handle “Quiescence State”
![Page 15: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/15.jpg)
Capturing nondeterminism Tasks in CHESS
Threads Timers Asynchronous I/Os
Wrappers in CHESS Goal of wrappers
![Page 16: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/16.jpg)
Capturing nondeterminism Design of Wrappers in CHESS
![Page 17: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/17.jpg)
Capturing nondeterminism Synchronization wrappers
![Page 18: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/18.jpg)
Capturing nondeterminism Synchronization wrappers
![Page 19: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/19.jpg)
Capturing nondeterminism Hooking the wrappers
use various mechanisms to dynamically intercept calls to the real API functions and forward them to the wrappers.
Programs in this paper Win32 .NET Singularity
![Page 20: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/20.jpg)
Exploring nondeterminism How CHESS systematically drives the test
Basic scheduler operation Allows only one thread to execute at a time
Repeatedly executes the same test driving each iteration of the test through a different schedule
Three phases in each iteration
![Page 21: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/21.jpg)
Exploring nondeterminism Replay phase
Replays a sequence of scheduling choices from a trace file
Record phase Schedules a thread till the thread yields the
processor
Search phase Uses the enabled information at each schedule
point to determine the scheduler for the next iteration
![Page 22: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/22.jpg)
Exploring nondeterminism Imperfect replay
Not rely on perfect replay capability
Common sources of nondeterminism Lazy-initialization
Interference from environment
Nondeterministic calls
![Page 23: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/23.jpg)
Exploring nondeterminism Ensuring fair schedules
Can not enumerate all fair schedules
State-explosion Definition of State-explosion
Inserting preemptions
Capturing states
![Page 24: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/24.jpg)
Evaluation Systems on which CHESS has been run on
![Page 25: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/25.jpg)
Evaluation Test scenarios and findings
![Page 26: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/26.jpg)
Evaluation Validation CHESS against stress-testing
Common objection of the CHESS
Failure and bug in CHESS
Succeeded in reproducing every stress-test failure
![Page 27: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/27.jpg)
Evaluation Description of two bugs
PLINQ bug
![Page 28: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/28.jpg)
Evaluation Description of two bugs
Singularity bug
![Page 29: CHESS: A Systematic Testing Tool for Concurrent Software CSCI6900 George](https://reader035.vdocuments.us/reader035/viewer/2022062313/56649cdb5503460f949a66a3/html5/thumbnails/29.jpg)
Related work Repeatable deterministic testing
Systematic generation of thread schedules
Applying state exploration directly to executing concurrent programs
Replay a concurrent execution