pe x xxx white box test generation for .net nikolai tillmann, peli de halleux microsoft research

23
Pexxxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Upload: hart

Post on 23-Feb-2016

29 views

Category:

Documents


0 download

DESCRIPTION

Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research. Unit Testing Today. A unit test is a small program with assertions. void AddTest () { HashSet set = new HashSet (); set.Add (7); set.Add (3); Assert.IsTrue ( set.Count == 2); } - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

PexxxxWhite Box Test Generation

for .NETNikolai Tillmann, Peli de Halleux

Microsoft Research

Page 2: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

2

Unit Testing TodayA unit test is a small program with assertions.

void AddTest() { HashSet set = new HashSet(); set.Add(7); set.Add(3);

Assert.IsTrue(set.Count == 2); }

Many developers write such unit tests by hand.

Page 3: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Vision: Parameterized Unit Testing void AddSpec(int x, int y) { HashSet set = new HashSet(); set.Add(x); set.Add(y);

Assert.AreEqual(x == y, set.Count == 1); Assert.AreEqual(x != y, set.Count == 2); }

Parameterized Unit Tests separate two concerns:1) The specification of externally visible behavior. (assertions)2) The selection of internally relevant test inputs. (coverage)

Page 4: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Demo: Parameterized Unit Test Write

[PexMethod] with parameters,

invoke Analysis

Page 5: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Demo: Pex Exploration Results

Table: dataand

results

Page 6: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Demo: Generated Unit Tests

Generated Test Inputs are stored as C# Unit

Tests

Page 7: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Pex: OverviewPurpose: Test input generator

Starting from parameterized unit testsGenerated tests emitted as traditional unit testsGoal: Test suite that covers all reachable statements

Technology: Dynamic symbolic executionWhole-program, white-box code analysisAt the level of the .NET instructions (bytecode)Symbolic execution based on monitoring and re-execution

Path conditions obtained by monitoring dataflow, branchesIterative selection of potential execution pathsConstraint solver determines test inputs for paths

Page 8: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Dynamic Symbolic Execution By ExampleCode to generate inputs for: Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==123456890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Input

null{}

{0}

{123…}

a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolveChoose next path

Done: There is no path left.

Negated condition

Page 9: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

How to test this code?(Actual code from .NET base class libraries.)

Going from simple exampleInto the real world

Page 10: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

10

Real World (II)

Page 11: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

There is no detailed specification.Most basic test oracle:

Code should not crash, i.e. throwing unexpected exceptions

11

Possible test case, written by Hand

Real World (III)

Page 12: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Test input, generated by Pex

12

Pex – Demo

Page 13: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Test Input Generation byDynamic Symbolic Execution

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Initially, choose Arbitrary

Page 14: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Initially, choose Arbitrarya[0] = 0;a[1] = 0;a[2] = 0;a[3] = 0;…

Test Input Generation byDynamic Symbolic Execution

Page 15: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Initially, choose Arbitrary

Test Input Generation byDynamic Symbolic Execution

Path Condition:… ⋀ magicNum != 0x95673948

Page 16: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Initially, choose Arbitrary

… ⋀ magicNum != 0x95673948… ⋀ magicNum == 0x95673948

Test Input Generation byDynamic Symbolic Execution

Page 17: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

a[0] = 206;a[1] = 202;a[2] = 239;a[3] = 190;

Initially, choose Arbitrary

Test Input Generation byDynamic Symbolic Execution

Page 18: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Initially, choose Arbitrary

Test Input Generation byDynamic Symbolic Execution

Page 19: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

TestInputs

Constraint System Execution

Path

KnownPaths

Run Test and Monitor

RecordPath Condition

Choose an Uncovered Path

Solve

Result: small test suite, high code coverage

Initially, choose Arbitrary

Finds only real bugsNo false warnings

Test Input Generation byDynamic Symbolic Execution

Page 20: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Advanced FeaturesCommand-line tool

Explores your code on your build serverGenerates HTML reports

Parameterized Mock ObjectsStubs and mock objects can use parameters too

Parameterized Object FactoriesFactories that take parameters help Pex

20

Page 21: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Assumptions and Assertions void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException(); }

void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException(); }

Assumptions and assertions are explored just like all other branches; Pex will try to break them!Executions which cause assumption violations are ignored, not reported as errors or test cases

21

Page 22: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

LimitationsAssumption: Environment is deterministic

"Environment" includes all code that is not monitored, e.g. native code, uninstrumented code, code that executes on another machine on the network…Pex simply ignores non-deterministic behavior

Assumption: Program is single-threadedLimitations of constraint solver

Little support for floating point arithmeticapproximation with rationals (linear arithmetic only)

No support for System.Decimal arithmeticBounds on time and memory consumptionhttp://research.microsoft.com/projects/Z3/

22

Page 23: Pe x xxx White Box Test Generation for .NET Nikolai Tillmann, Peli de Halleux Microsoft Research

Summary

23

Pex automates test input generation for .NET programsPex enables Parameterized Unit TestingUsed in Microsoft to test core .NET components

Pex is publicly available for academic use.

Watch our website for future announcements.http://research.microsoft.com/Pex