pe x xxx white box test generation for .net nikolai tillmann, peli de halleux microsoft research
DESCRIPTION
Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research. Unit Testing Today. A unit test is a small program with assertions. void AddTest () { HashSet set = new HashSet (); set.Add (7); set.Add (3); Assert.IsTrue ( set.Count == 2); } - PowerPoint PPT PresentationTRANSCRIPT
PexxxxWhite Box Test Generation
for .NETNikolai Tillmann, Peli de Halleux
Microsoft Research
2
Unit Testing TodayA unit test is a small program with assertions.
void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3);
Assert.IsTrue(set.Count == 2); }
Many developers write such unit tests by hand.
Vision: Parameterized Unit Testing void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y);
Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); }
Parameterized Unit Tests separate two concerns:1) The specification of externally visible behavior. (assertions)2) The selection of internally relevant test inputs. (coverage)
Demo: Parameterized Unit Test Write
[PexMethod] with parameters,
invoke Analysis
Demo: Pex Exploration Results
Table: dataand
results
Demo: Generated Unit Tests
Generated Test Inputs are stored as C# Unit
Tests
Pex: OverviewPurpose: Test input generator
Starting from parameterized unit testsGenerated tests emitted as traditional unit testsGoal: Test suite that covers all reachable statements
Technology: Dynamic symbolic executionWhole-program, white-box code analysisAt the level of the .NET instructions (bytecode)Symbolic execution based on monitoring and re-execution
Path conditions obtained by monitoring dataflow, branchesIterative selection of potential execution pathsConstraint solver determines test inputs for paths
Dynamic Symbolic Execution By ExampleCode to generate inputs for: Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==123456890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Input
null{}
{0}
{123…}
a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolveChoose next path
Done: There is no path left.
Negated condition
How to test this code?(Actual code from .NET base class libraries.)
Going from simple exampleInto the real world
10
Real World (II)
There is no detailed specification.Most basic test oracle:
Code should not crash, i.e. throwing unexpected exceptions
11
Possible test case, written by Hand
Real World (III)
Test input, generated by Pex
12
Pex – Demo
Test Input Generation byDynamic Symbolic Execution
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Initially, choose Arbitrary
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Initially, choose Arbitrarya[0] = 0;a[1] = 0;a[2] = 0;a[3] = 0;…
Test Input Generation byDynamic Symbolic Execution
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Initially, choose Arbitrary
Test Input Generation byDynamic Symbolic Execution
Path Condition:… ⋀ magicNum != 0x95673948
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Initially, choose Arbitrary
… ⋀ magicNum != 0x95673948… ⋀ magicNum == 0x95673948
Test Input Generation byDynamic Symbolic Execution
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
a[0] = 206;a[1] = 202;a[2] = 239;a[3] = 190;
Initially, choose Arbitrary
Test Input Generation byDynamic Symbolic Execution
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Initially, choose Arbitrary
Test Input Generation byDynamic Symbolic Execution
TestInputs
Constraint System Execution
Path
KnownPaths
Run Test and Monitor
RecordPath Condition
Choose an Uncovered Path
Solve
Result: small test suite, high code coverage
Initially, choose Arbitrary
Finds only real bugsNo false warnings
Test Input Generation byDynamic Symbolic Execution
Advanced FeaturesCommand-line tool
Explores your code on your build serverGenerates HTML reports
Parameterized Mock ObjectsStubs and mock objects can use parameters too
Parameterized Object FactoriesFactories that take parameters help Pex
20
Assumptions and Assertions void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); }
void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); }
Assumptions and assertions are explored just like all other branches; Pex will try to break them!Executions which cause assumption violations are ignored, not reported as errors or test cases
21
LimitationsAssumption: Environment is deterministic
"Environment" includes all code that is not monitored, e.g. native code, uninstrumented code, code that executes on another machine on the network…Pex simply ignores non-deterministic behavior
Assumption: Program is single-threadedLimitations of constraint solver
Little support for floating point arithmeticapproximation with rationals (linear arithmetic only)
No support for System.Decimal arithmeticBounds on time and memory consumptionhttp://research.microsoft.com/projects/Z3/
22
Summary
23
Pex automates test input generation for .NET programsPex enables Parameterized Unit TestingUsed in Microsoft to test core .NET components
Pex is publicly available for academic use.
Watch our website for future announcements.http://research.microsoft.com/Pex