Parameterized Unit Testing:Principles, Techniques, and Applications in Practice

Tutorial

Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research)

Tao Xie (North Carolina State University)

Outline

Unit Testing Parameterized Unit Testing Input Generation with Pex Patterns for Parameterized Unit

Testing Parameterized Models Wrapping up

About the tool…

We will use the tool Pex as part of the exercises To install it, you need

Windows XP or Vista .NET 2.0

Download Pex fromhttp://research.microsoft.com/en-us/projects/pex/downloads.aspx

Two variations : pex.devlabs.msi:

requires Visual Studio 2008 Team System pex.academic.msi:

does not require Visual Studio, but for non-commercial use only

To run the samples, you need VS Team System

Unit TestingIntroduction

Unit Testing

void ReadWrite() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count);}

A unit test is a small program with assertions Test a single (small) unit of code

Unit Testing Good/Bad

Design and specification By Example Documentation Short feedback loop Code coverage and regression testing Extensive tool support for test execution and

management

Happy path only Hidden integration tests

Touch the database, web services, requires multiple machines, etc…

New Code with Old Tests Redundant tests

Unit Testing: Measuring Quality

Coverage: Are all parts of the program exercised? statements basic blocks explicit/implicit branches …

Assertions: Does the program do the right thing? test oracle

Experience: Just high coverage of large number of assertions

is no good quality indicator. Only both together are!

Exercise: “TrimEnd”Unit TestingGoal: Implement and test TrimEnd:

Open Visual Studio Create CodeUnderTestProject, and implement TrimEnd

File -> New -> Project -> Visual C# -> Class Library Add new StringExtensions class Implement TrimEnd

Create TestProject, and test TrimEnd File -> New -> Project -> C# -> Test -> Test Project Delete ManualTest.mht, UnitTest1.cs, … Add new StringExtensionsTest class Implement TrimEndTest

Execute test Test -> Windows -> Test View, execute test, setup and inspect Code

Coverage

// trims occurences of the ‘suffix’ from ‘target’string TrimEnd(string target, string suffix);

Exercise: “TrimEnd”Unit Testingpublic class StringExtensions { public static string TrimEnd(string target, string suffix) { if (target.EndsWith(suffix)) target = target.Substring(0, target.Length-suffix.Length); return target; }}[TestClass]public class StringExtensionTest { [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = StringExtensions.TrimEnd(target, suffix); Assert.IsTrue(result == "Hello "); }}

Parameterized Unit TestingIntroduction

The Recipe of Unit Testing

var list = new List(); list.Add(item); var count = list.Count;

Assert.AreEqual(1, count);}

Three essential ingredients: Data Method Sequence Assertionsvoid Add() { int item = 3;

The (problem with) Data

Which value matters? Bad choices cause redundant, or worse,

incomplete test suites. Hard-coded values get stale when

product changes. Why pick a value if it doesn’t matter?

list.Add(3);

Parameterized Unit Testing

void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Parameterized Unit Test = Unit Test with Parameters

Separation of concerns Data is generated by a tool Developer can focus on functional

specification

Parameterized Unit Tests areAlgebraic Specifications

A Parameterized Unit Test can be read as a universally quantified, conditional axiom.void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);} string name, string data: name ≠ null data ≠ null ⋀ ⇒ equals( ReadResource(name,WriteResource(name,data)), data)

Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly

supported by various test frameworks .NET: Supported by .NET test frameworks

http://www.mbunit.com/ http://www.nunit.org/ …

Java: Supported by JUnit 4.X http://www.junit.org/

Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex

http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne

http://www.agitar.com/

Exercise: “TrimEnd”Manual Parameterized Unit Testing

Goal: Create a parameterized unit test for TrimEnd..

Refactor: Extract Method[TestMethod]public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = ParameterizedTest(target, suffix); Assert.IsTrue(result == "Hello ");}

static string ParameterizedTest(string target, string suffix) { var result = StringExtensions.TrimEnd(target, suffix); return result;}

17

Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible

Observations: Reachability not decidable, but

approximations often good enough Encoding of functional correctness checks

as assertions that reach an error statement on failure

Data Generation Challenge

Assumptions and Assertions

void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException();}

Assumptions and assertions induce branches

Executions which cause assumption violations are ignored, not reported as errors or test cases

Data Generation

Human Expensive, incomplete, …

Brute Force Pairwise, predefined data, etc…

Semi - Random: QuickCheck Cheap, Fast “It passed a thousand tests” feeling

Dynamic Symbolic Execution: Pex, Cute,EXE Automated white-box Not random – Constraint Solving

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

Exercise: “CoverMe”Dynamic Symbolic Execution with Pex Open CoverMe project in Visual Studio. Right-click on CoverMe method, and select

“Run Pex”. Inspect results in table.

public static void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

How to test this code?(Actual code from .NET base class libraries)

Motivation: Unit Testing HellResourceReader

Motivation: Unit Testing HellResourceReader

Exercise: “ResourceReaderTest”ResourceReader[PexClass, TestClass]

[PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(typeof(BadImageFormatException))][PexAllowedException(typeof(IOException))][PexAllowedException(typeof(NotSupportedException))]public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}

When does a test case fail? If the test does not throw an exception, it

succeeds. If the test throws an exception,

(assumption violations are filtered out), assertion violations are failures, for all other exception, it depends on further

annotations.

Annotations Short form of common try-catch-assert test code [PexAllowedException(typeof(T))] [PexExpectedException(typeof(T))]

Dynamic Symbolic ExecutionPex Architecture

A brief overview

Code Instrumentation

Z3

Pex

ExtendedReflection

Instrumentation framework “Extended Reflection” Instruments .NET code at runtime Precise data flow and control flow analysis

Code Under Test

Dynamic Symbolic Execution

Z3

Pex

ExtendedReflection

Pex listens to monitoring callbacks Symbolic execution along concrete

execution

Code Under Test

Constraint Solving

Z3

Pex

ExtendedReflection

Constraint solving with SMT solver Z3 Computes concrete test inputs

Code Under Test

Test Generation

Z3

Pex

ExtendedReflection

Execute with test inputs from constraint solver Emit test as source code if it increases

coverage

Code Under Test

Components

Z3

Pex

ExtendedReflection

ExtendedReflection: Code Instrumentation Z3: SMT constraint solver Pex: Dynamic Symbolic Execution

Code Under Test

ldtoken Point::GetXcall __Monitor::EnterMethodbrfalse L0ldarg.0call __Monitor::Argument<Point>

L0: .try { .try { call __Monitor::LDARG_0 ldarg.0 call __Monitor::LDNULL ldnull call __Monitor::CEQ ceq call __Monitor::BRTRUE brtrue L1 call __Monitor::BranchFallthrough call __Monitor::LDARG_0 ldarg.0 …

ldtoken Point::X call __Monitor::LDFLD_REFERENCE ldfld Point::X call__Monitor::AtDereferenceFallthrough br L2

L1: call __Monitor::AtBranchTarget call __Monitor::LDC_I4_M1 ldc.i4.m1

L2: call __Monitor::RET stloc.0 leave L4 } catch NullReferenceException {

‘ call__Monitor::AtNullReferenceException rethrow }

L4: leave L5} finally { call __Monitor::LeaveMethod endfinally }

L5: ldloc.0ret

class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } }

Prologue

Epilogue

Calls will performsymbolic computation

Calls to build path condition

Calls to build path condition

Record concrete values to have all information

when this method is calledwith no proper context(The real C# compiler

output is actually more complicated.)

pp

null

p == null

Code Instrumentationby example

Pex’s representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, …Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references

to values" Elements of arrays, memory accessed through unsafe

pointers by mutable “mapping of integers to values"

Efficiency by Many reduction rules, including reduction of ground terms to

constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical operations Patricia Trees to represent AC1 operators

(including parallel array updates)

Symbolic State Representation

SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to

theories SAT solver for Boolean structure Decision procedures for relevant theories:

uninterpreted functions with equalities,linear integer arithmetic, bitvector arithmetic, arrays, tuples

Model generation for satisfiable formulas Models used as test inputs

Limitations No decision procedure for floating point arithmetic

http://research.microsoft.com/z3

Constraint Solving: Z3

Pex inVisual StudioParameterized Unit Testing

Test Project

Test Generation WorkFlow

Code-Under-Test

Project

Parameterized Unit Tests

Pex Generated Tests

Right-click on the code-under-test project In the Solution Explorer

Select “Pex” Select “Create Parameterized Unit

Test Stubs”

Creating a (Parameterized)Test Project

(Parameterized) Test Project

Generated Test ProjectReferencesMicrosoft.Pex.Framework(and more)

ReferencesVisualStudio UnitTestFramework

Contains source files with generated Parameterized Unit Test stubs

Right-click on Parameterized Unit Test For now, it just calls the implementation,

but the you should edit it, e.g. to add assertions

Select “Run Pex Explorations”

Running Pex from the Editor

Exploration Result View

Exploration

Status

Current Parameterize

d Unit Test

Issue barImportant messages here !!!

Input/Output tableRow = generated testColumn = parameterized test input or output

Input/Output table

Test outcomefiltering

PassingTest

Fix available

New Test

FailingTest

Input/Output table

ExceptionStack traceReview test

Allow exception

Pex Explorer

Pex Explorer makes it easy to digest the results of many Parameterized Unit Tests, and many generated test inputs

Pex from theCommand linepex.exe

Pex.exe test.dll

HTML Reports

Rich information, used by Pex developers Full event log history Parameter

table Code

coverage etc…

HTML rendered from XML report file:It is easy to programmatically extract information!

Annotations: Code Under Test

Tell Pex which Type you are testing Code coverage Exception filtering Search prioritization

[PexClass(typeof(Foo))]public class FooTest { …

Code Coverage Reports

3 Domains UserCodeUnderTest

UserOrTestCode

SystemCode

public void CodeUnderTest() { … }

[PexClass] public class FooTest { [PexMethod] public void ParamTest(…) { …

public class String { public bool Contains(string value) { …

Command line: Filters

Namespace, type, method filters pex.exe test.dll /tf:Foo /mf:Bar

Partial match, case insensitive Full match: add “!”: /tf:Foo! Prefix: add “*”: /tf:Foo*

Many other filters e.g. to select a “suite”: /sf=checkin For more information:pex.exe help filtering

ExerciseData generation

Goal: Explore input generation techniques with TrimEnd

Write or discuss a random input generator for TrimEnd

Adapt your parameterized unit tests for Pex Start over with “Create Parameterized Unit Test Stubs”,

or…▪ Add reference to Microsoft.Pex.Framework▪ Add using Microsoft.Pex.Framework;▪ Add [PexClass(typeof(StringExtensions))] on

StringExtensionsTest▪ Add [PexMethod] on your parameterized tests

Right-click in test and ‘Run Pex Explorations’ Compare test suites

LimitationsIt’s called Parameterized Unit Testing

The yellow event bar notifies about important events, including certain limitations

Event Bar

Click on issue kind for more information

Events View

You should act on these events: Refactor your code, or tell Pex to ignore it in the future, let Pex analyze (“instrument”) more code,

if possible.

Click on a particular event for more information

Understand the issue, and take an action

Instrumenting more code If Pex reports that some code was

uninstrumented, you may tell Pex to instrument and analyze it(if possible)

Instrumenting more code

Code instrumentation on Demand Instrumentation has high performance

overhead Some parts of the code better ignored

Use PexInstrument… attributes

Pex will often suggest and insert those attributes for you

[assembly: PexInstrumentAssembly(“Foo”)]

Pex understand managed .NET code only Pex does not understand native code.

Problem if branching over values obtained from the environment Pex may not automatically detect all such

cases.

Testability

if (!File.Exists(f)) throw ...

File System?

Hidden Complexity

Pex analyzes every executed .NET instruction

Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different

representationsvoid Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }

Don’t do this.

There are decision procedures for individual path conditions, but… Number of potential paths grows

exponentially with number of branches Reachable code not known initially Without guidance, same loop might be

unfolded forever

Explosion of Search Space

Explosion of Search Space

To deal with the path explosion problem, Pex uses search heuristics.

Heuristics try to create “diversity”, i.e. they prefer paths which are so far underrepresented e.g. by their coverage vectors

Exploration Boundaries

In the presence of loops and recursion,Pex could search forever, so Pex has many ways to bound the search

Named arguments of [PexMethod]

Pex will often suggest and update those bounds for you.

[PexMethod(MaxConstraintSolverTime = 4)]

Exploration Boundaries

Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions

▪ Number of conditions that depend on test inputs

MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit

Division and modulo are expensive to reason about

No decision procedures for floating-point arithmetic Pex currently uses heuristic search

techniques for floating-point constraints

Constraint Solving

Constraint Solvingvs. Search Strategies

TestInputs

Constraint System

Execution Path

KnownPaths

Initially, choose ArbitraryPath-constraint

solving is just hard.

Reachability is

undecidable! (Loops)

(With external

functions and dynamic loading,

summariesare difficult.)

Multi-threaded code

Pex only explores single-threaded code

Related approach to explore thread-schedules (but not input parameters): CHESShttp://research.microsoft.com/en-us/projects/chess/

Lack of Test Oracle

Write assertions and Pex will try to break them

Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.

Assertions leveraged in product and test code

Pex can leverage Code Contracts (discussed later)

Object Creation

Pex only uses public methods to configure non-public object fields

Heuristics built-in to deal with common types

User can help if neededvoid (Foo foo) { if (foo.Value == 123) throw …[PexFactoryMethod]Foo Create(Bar bar) { return new Foo(bar);}

Exercise Limitations

Goal: Understand limitations of DSE. PexGoal/PexExpectedGoals Apply Pex to

if (File.Exists(fileName)) PexGoal.Reached();

if (DateTime.Parse(s).Year == 2009) PexGoal.Reached();

class Foo { private Foo() {} }…if (foo != null) PexGoal.Reached();

Patternshow to create objectsTips and tricks

Creating complex objectsProblem:• Constraint solver determines

test inputs = initial state of test• Most classes hide their state (private fields)• State is initialized by constructor, and can be

mutated only by calling methods• What sequence of method calls reaches a

given target state?• There may be no such sequence• In general, undecidable

Two approaches:• (Guided) exploration of constructor/mutator

methods• Testing with class invariants

Example: ArrayList

Specification:

[PexMethod]public void ArrayListTest(ArrayList al, object o){ PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}

Factory methods Parameterized factories create and configure

objects Explicit: Public static method annotated with

[PexFactoryMethod] Implicit: Pex guesses “best” constructor/setters

Result: Exploration of reachable states Reachable using factory method Reachable within configured bounds Under-approximation of possible states

Object creation:Guided Exploration

We will now describe how you can create objects via the exploration of a class invariant.

Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code

Write special constructor for testing only May be marked as "debug only" Constructor sets fields, assumes invariant

Result: Exploration of feasible states May include states that are not reachable

Object creation:Class invariants

Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit

Inspect generated test cases Notice constructor implicitly used by Pex

Exercise:Factory methods

Exercise:Factory method by invariant Edit code of ArrayList class: Add “ad-hoc” Invariant method:

Add DEBUG-only constructor that allows to configure object freely, and then checks invariant:

private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; }

#if DEBUG public ArrayList(object[] items, int size) { this._items = items; this._size = size; if (!this.Invariant()) throw new Exception(); }#endif

Patternsfor Parameterized Unit TestsTips and tricks

Pattern4A

Assume, Arrange, Act, Assert

[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert}

PatternRoundtrip

For an API f(x), f-1(f(x)) = x for all x

void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped);}

PatternRoundtrip

For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed);}

PatternSanitized Roundtrip

For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x

void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped);}

PatternState Relation

Observe a state change

void ContainedAfterAdd(string value) { var list = new List<string>(); list.Add(value); Assert.IsTrue(list.Contains(value));}

PatternSame Observable Behavior

Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.

public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right));}

PatternCommutative Diagram Given two implementations f and g of the

same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.

string Multiply(string x, string y);int Multiply(int x, int y);

void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2);}

PatternAllowed Exception

Allowed exception -> negative test case

[PexAllowedException(typeof(ArgumentException))]void Test(object item) { var foo = new Foo(item) // validates item// generated test (C#)[ExpectedException(typeof(ArgumentException))]void Test01() { Test(null); // argument check}

PatternReachability

Indicate which portions of a PUT should be reachable.

[PexExpectedGoals]public void InvariantAfterParsing(string input){ ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant();}

PatternRegression Tests

Generated test asserts any observed value Return value, out parameters, PexGoal

When code evolves, breaking changes in observable will be discovered

int AddTest(int a, int b) { return a + b; }

void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result);}

Exercise Patterns

Goal: Identify and apply patterns Apply patterns to parameterized unit tests

of TrimEnd.

ParameterizedModels

Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.

Integration test: exercises large portions of a system.

Observation: Integration tests are often “sold” as unit tests

White-box test generation does not scale well to integration test scenarios.

Possible solution: Introduce abstraction layers, and mock components not under test

Unit Testing vs. Integration Testing

AppendFormat(null, “{0} {1}!”, “Hello”, “World”); “Hello World!”

.Net Implementation:

public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) {

if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf =

(ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …

ExampleTesting with Interfaces

Introduce a mock class which implements the interface.

Write assertions over expected inputs, provide concrete outputs

public class MFormatProvider : IFormatProvider {

public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); }}

Problems: Costly to write detailed behavior by example How many and which mock objects do we need to

write?

Stubs / Mock Objects

Parameterized Mock Objects - 1 Introduce a mock class which implements the

interface. Let an oracle provide the behavior of the mock

methods.

public class MFormatProvider : IFormatProvider {

public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); return o; }}

Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 92

Exercise: “AppendFormatSample”AppendFormat

Show AppendFormat code Show AppendFormat test Run Pex

93

Chosen values can be shaped by assumptions

public class MFormatProvider : IFormatProvider {

public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); PexAssume.IsTrue(o is ICustomFormatter);

return o; }}

(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)

Parameterized Mock Objects - 2

94

Models from Choices

Choices to build parameterized models

class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue<string,string> files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}

Exercise Parameterized Models

Goal: Test a ‘CopyFile’ method with a File System model

Write a CopyFile method using IFileSystem

Implement IFileSystem with System.IO.File and use it to test Copy.

Write a parameterized model for IFileSystem and use it to test Copy.

interface IFileSystem { string ReadFile(string fileName); void WriteFile(string name, string content);}

Code Contracts SupportDesign By Contracts meets Automated Whitebox Testing

Code Contracts

http://research.microsoft.com/en-us/projects/contracts/

Library to state preconditions, postconditions, invariants

Supported by two tools: Static Checker Rewriter: turns Code Contracts into runtime

checks Pex analyses the runtime checks

Contracts act as Test Oracle Pex may find counter examples for

contracts Missing Contracts may be suggested

If present, Pex leverages [ContractInvariantMethod] to create objects via reflection and checking if invariant holds: Install Code Contracts:

Add reference to Microsoft.Contracts library Add empty [ContractInvariantMethod] Run Pex, click on “Add Invariant” Repeat, edit manually

Exercise: (optional)Class invariants with Code Contracts

[ContractInvariantMethod]protected void Invariant() { Contract.Invariant(this._items != (object[])null); Contract.Invariant( (uint)(this._size) <= (uint)(this._items.Length));}

Code Contracts allow specification of interface contracts

Pex‘ stubs framework can leverage contracts to check input and restrict output of mock objects.

Consider the following interfaces annotated with contracts (using Spec# notation):

interface IFormatProvider { object GetFormat(Type formatType) requires formatType != null; ensures result != null && formatType.IsAssignableFrom(result.GetType());

}

Design-by-contractvs. Parameterized Models

Exercise: Testing with Interface Contracts

Create IFormatProvider interface with Code Contractsin code-under-test project

Create (parameterized) test project Show stubs Create parameterized unit test using

IFormatProvider Explore parameterized unit test

Wrapping up

Some Related Toolsfor Data-Generation Random input generators: many Combining random testing and constraint

solving DART, CUTE, EXE, KLEE, CREST (C) jCUTE, jFuzz (Java) SAGE (X86) …

Program model checkers JPF, Kiasan/KUnit (Java), XRT (.NET)

Commercial tools AgitarOne, …

103

Pex on MSDN DevLabsIncubation Project for Visual Studio 2010

Thank you

http://research.microsoft.com/pexhttp://codeplex.com/Pexhttps://sites.google.com/site/asergrp/