Contract Checking and Automated Test Generation With Pex

Mike BarnettRSDEMicrosoft Research

Nikolai TillmannRSDEMicrosoft Research

TL51

Code ContractsFor .NET

Mike BarnettRSDEMicrosoft Research

Nikolai TillmannRSDEMicrosoft Research

AutomatedTest GenerationWith Pex

Two Tools.One Talk.

Code Contracts for .NET

announcingContract

Library in .NET 4.0

Coming soon: Tools for contracts at

string GetDescription(int x);

/// <param name=“x”>/// should be greater than zero/// </param>/// <returns>/// non-null string/// </returns>string GetDescription(int x);

How Do You Know What It Does?

string s = o.GetDescription(i);… s.Length …

Let’s Fix It For Once And All!

string GetDescription(int x) { Contract.Requires( 0 < x ); Contract.Ensures( Contract.Result<string>() != null ); …}

Design-by-Contract meets .NET!

Contracts document design decisions Preconditions, postconditions, invariants Facilitate code evolution

Contracts are executable Checked documentation Test oracles

Contracts help static verification Early error detection Clarify responsibility at API boundaries

Assertions help, but everyone has their own flavor Difficult to tool against Not seen as a contract between 2 parties

Contracts

Introduction

Mike BarnettRSDEResearch in Software Engineering

demo

class Rational {

public Rational(int n, int d) { Contract.Requires( 0 < d ); this.N = n; this.D = d; }}

What Can I Do With Contracts?

Documentation

Runtime Checking

Static Checking

Test Generation

(Pex)

Requires What must be true at method entry

Ensures What must be true at method exit

Invariants What must be true at all method exits

Assertions What must be true at a particular point

Assumptions What should be true at a particular point

What Contracts Can I Write?

Any boolean expression In your favorite programming language! Including method calls (but must

be marked Pure) Quantifiers

Contract.ForAll(0,A.Length,i => A[i] > 0); Contract.Exists(0,A.Length,i => A[i] > 0);

Contract.Result refer to the return value of the method

Contract.OldValue refer to values at method entry

What Can I Put In A Contract?

void ObjectInvariant() { Contract.Invariant( items != null );}

How Do I Write A Contract?

Features• Declarative• Language expression syntax• Type checking / IDE

• Special Encodings• Result and OldValue

public virtual int Add(object value){

Contract.Requires( value != null ); Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) ); if (count == items.Length) EnsureCapacity(count + 1); items[count] = value; return count++;}

Runtime Checking

Mike BarnettRSDEResearch in Software Engineering

demo

.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret} // end of method BaseList::Add

csc/vbc/…

csc/vbc/…

.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0029 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_004b ldloc.0 ret}

How Does It Work?

ReleaseCompile

/d:CONTRACTS_FULL

ccrewrite

• Inheritance• Style checks

Executable Runtime Contract Checking

public virtual int Add(object value){ Contract.Requires( value != null );

Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );

if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++;}

.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ .locals init (int32 'Contract.Old(Count)', int32 'Contract.Result<int>()') ldarg.0 call instance int32 TabDemo.BaseList::get_Count() stloc.3 ldarg.1 ldnull ceq ldc.i4.0 ceq ldstr "value != null" call void __RewriterMethods::RewriterRequires$PST06000009(bool, string) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue IL_004d nop ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) nop nop ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br IL_0072 ldloc.0 stloc.s 'Contract.Result<int>()' br IL_007a ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldloc.3 ldc.i4.1 add ceq ldstr "Count == Contract.Old(Count) + 1" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ldloc.s V_4 ceq ldstr "Contract.Result<int>() == Contract.Old(Count)" call void __RewriterMethods::RewriterEnsures$PST0600000B(bool, string) ldloc.s 'Contract.Result<int>()' ret}

No silver bullet But helps catch errors earliest Best used in a focused manner

Guides development Discovers implicit assumptions Propagates assumptions

Not only explicit contracts Dereferencing null Indexing arrays Arithmetic exceptions

Static Contract Checking

What Do You Ship?

srcsrc

srcsrc

PowerLib.dll

(minimal runtime checks)

PowerLib.Contracts.dll

All contracts, no code

+

ReleaseAssemblies

Contract ReferenceAssemblies

Contract Reference Assemblies

Compile/d:CONTRACTS_FULL

ccRefGen

.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.0 ldfld object[] TabDemo.BaseList::items ldlen conv.i4 ceq ldc.i4.0 ceq stloc.1 ldloc.1 brtrue.s IL_0069 ldarg.0 ldarg.0 ldfld int32 TabDemo.BaseList::count ldc.i4.1 add call instance void TabDemo.BaseList::EnsureCapacity(int32) ldarg.0 ldfld object[] TabDemo.BaseList::items ldarg.0 ldfld int32 TabDemo.BaseList::count ldarg.1 stelem.ref ldarg.0 dup ldfld int32 TabDemo.BaseList::count dup stloc.2 ldc.i4.1 add stfld int32 TabDemo.BaseList::count ldloc.2 stloc.0 br.s IL_008b ldloc.0 ret} // end of method BaseList::Add

Contract ReferenceAssembly

.method public hidebysig newslot virtual instance int32 Add(object 'value') cil managed{ ldarg.1 ldnull ceq ldc.i4.0 ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Requires(bool) ldarg.0 call instance int32 TabDemo.BaseList::get_Count() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ldc.i4.1 add ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool) call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Result<int32>() ldarg.0 call instance int32 TabDemo.BaseList::get_Count() call !!0 [Microsoft.Contracts]Microsoft.Contracts.Contract::Old<int32>(!!0) ceq call void [Microsoft.Contracts]Microsoft.Contracts.Contract::Ensures(bool)} // end of method BaseList::Add

public virtual int Add(object value){ Contract.Requires( value != null );

Contract.Ensures( Count == Contract.OldValue(Count) + 1 ); Contract.Ensures( Contract.Result<int>() == Contract.OldValue(Count) );

if (_size == _items.Length) EnsureCapacity(_size+1); _items[_size] = value; return _size++;}

[ContractClass(typeof(CloneableContract))]public interface ICloneable {

object Clone();

}

Interface Contracts

[ContractClassFor(typeof(ICloneable))]public class CloneableContract : ICloneable {

public object Clone() { Contract.Ensures( Contract.Result<object>() != null );

… }

}All classes implementing the interface inherit the contract

Contract library class enables contracts in all .NET languages No restrictions on what can be expressed

Contracts are being used in the BCL today Contract library is a core component of .NET

4.0 Same contracts used for

Runtime checking Static checking Documentation generation

Code Contracts Summary

Get Pex for Visual Studio 2010 at

Do you care about quality?Shake your code with Pex.

announcing Pex

Testing is tedious Too easy to miss cases Old tests get stale Too much legacy code −

what does it do?

Testing Before Pex

Don’t give up, Pex has arrived.

Tests for Freewith Pex

Nikolai TillmannRSDEMicrosoft Research

demo Pex

It’s painless: Just right-click your code It’s effective: Test cases for free It’s fun:

Inspect table, Save tests, file work items, or debug right away

What The Demo Showed

Code Coverage for Free with Pex Nikolai Tillmann

RSDEMicrosoft Research

demo Pex

Pex automatically generatedcomprehensive test suitewith high code coverage

Pex finds contract violationsThe generated test suite

integrates withVisual Studio Team Test

can be used asBuild Verification Test (BVT)

What The Demo Showed

A traditional Unit Test is amethod without parameters.

There Is More…

[TestMethod]void AddZeroTest(){ var r = new Rational(1,2); var q = r + 0; Assert(r == p);}

Handwritten

A Parameterized Unit Test is simply atest method that takes parameters

Pex generates traditional Unit Tests (in C#) which fix particular values

There Is More…Parameterized unit testing with pex

[PexMethod]void AddZeroTest( Rational r){ var q = r + 0; Assert(r == p);}

[TestMethod]void AddZeroTest01(){ var r = new Rational(1,2); this.AddZeroTest(r);}

Pex

calls

Automatically generatedHandwritten

How to test this code?(Actual code from .NET base class libraries)

A Tester's Nightmare: ResourceReader

A Tester's Nightmare: ResourceReader

Parameterized Unit Tests

[PexAllowedException(typeof(ArgumentException))][TestClass]public partial class ResourceReaderTest { [PexMethod] public void ReadEntries(byte[] data) { if (data == null) return; fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}

Parameterized Unit Testing

Nikolai TillmannRSDEMicrosoft Research

demo Pex

Pex does not guess. Pex does not generate random inputs, enumerate all possible values, or make you write test input generators

Instead, Pex analyzes your .NET code. Pex partitions inputs into equivalence classes One equivalence class per branching behavior Test inputs computed by Z3, the world class

constraint solver for program analysisfrom Microsoft Research

Precise inter-procedural, path-sensitive analysis

As a result, you geta small test suite with high code coverage

Pex Understands The Code

Pex generates small test suites with high code coverage and bug reports for free

Use previously generated test suites asBuild Verification Test (BVT)

Reduce test maintenance costsby parameterized unit testing

Pex has been used in Microsoftto test core .NET components

Download Pex for Visual Studio 2010 CTP on

http://msdn.microsoft.com/DevLabs

Pex Summary

Code Contracts for .NET:preconditions, postconditions, invariants http://research.microsoft.com/Contracts/

Pex: test generation for .NET http://research.microsoft.com/Pex/

Summary

Research in Software Engineering

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

Please use the microphones provided

Q&A

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.