tutorial nikolai tillmann, peli de halleux, wolfram schulte (microsoft research) tao xie (north...
Post on 21-Dec-2015
215 Views
Preview:
TRANSCRIPT
Parameterized Unit Testing:Principles, Techniques, and Applications in Practice
Tutorial
Nikolai Tillmann, Peli de Halleux, Wolfram Schulte (Microsoft Research)
Tao Xie (North Carolina State University)
Outline
Unit Testing Parameterized Unit Testing Input Generation with Pex Patterns for Parameterized Unit
Testing Parameterized Models Wrapping up
About the tool…
We will use the tool Pex as part of the exercises To install it, you need
Windows XP or Vista .NET 2.0
Download Pex fromhttp://research.microsoft.com/en-us/projects/pex/downloads.aspx
Two variations : pex.devlabs.msi:
requires Visual Studio 2008 Team System pex.academic.msi:
does not require Visual Studio, but for non-commercial use only
To run the samples, you need VS Team System
Unit TestingIntroduction
Unit Testing
void ReadWrite() { var list = new List(); list.Add(3); Assert.AreEqual(1, list.Count);}
A unit test is a small program with assertions Test a single (small) unit of code
Unit Testing Good/Bad
Design and specification By Example Documentation Short feedback loop Code coverage and regression testing Extensive tool support for test execution and
management
Happy path only Hidden integration tests
Touch the database, web services, requires multiple machines, etc…
New Code with Old Tests Redundant tests
Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised? statements basic blocks explicit/implicit branches …
Assertions: Does the program do the right thing? test oracle
Experience: Just high coverage of large number of assertions
is no good quality indicator. Only both together are!
Exercise: “TrimEnd”Unit TestingGoal: Implement and test TrimEnd:
Open Visual Studio Create CodeUnderTestProject, and implement TrimEnd
File -> New -> Project -> Visual C# -> Class Library Add new StringExtensions class Implement TrimEnd
Create TestProject, and test TrimEnd File -> New -> Project -> C# -> Test -> Test Project Delete ManualTest.mht, UnitTest1.cs, … Add new StringExtensionsTest class Implement TrimEndTest
Execute test Test -> Windows -> Test View, execute test, setup and inspect Code
Coverage
// trims occurences of the ‘suffix’ from ‘target’string TrimEnd(string target, string suffix);
Exercise: “TrimEnd”Unit Testingpublic class StringExtensions { public static string TrimEnd(string target, string suffix) { if (target.EndsWith(suffix)) target = target.Substring(0, target.Length-suffix.Length); return target; }}[TestClass]public class StringExtensionTest { [TestMethod] public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = StringExtensions.TrimEnd(target, suffix); Assert.IsTrue(result == "Hello "); }}
Parameterized Unit TestingIntroduction
The Recipe of Unit Testing
var list = new List(); list.Add(item); var count = list.Count;
Assert.AreEqual(1, count);}
Three essential ingredients: Data Method Sequence Assertionsvoid Add() { int item = 3;
The (problem with) Data
Which value matters? Bad choices cause redundant, or worse,
incomplete test suites. Hard-coded values get stale when
product changes. Why pick a value if it doesn’t matter?
list.Add(3);
Parameterized Unit Testing
void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Separation of concerns Data is generated by a tool Developer can focus on functional
specification
Parameterized Unit Tests areAlgebraic Specifications
A Parameterized Unit Test can be read as a universally quantified, conditional axiom.void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);} string name, string data: name ≠ null data ≠ null ⋀ ⇒ equals( ReadResource(name,WriteResource(name,data)), data)
Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly
supported by various test frameworks .NET: Supported by .NET test frameworks
http://www.mbunit.com/ http://www.nunit.org/ …
Java: Supported by JUnit 4.X http://www.junit.org/
Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex
http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne
http://www.agitar.com/
Exercise: “TrimEnd”Manual Parameterized Unit Testing
Goal: Create a parameterized unit test for TrimEnd..
Refactor: Extract Method[TestMethod]public void TrimEndTest() { var target = "Hello World"; var suffix = "World"; var result = ParameterizedTest(target, suffix); Assert.IsTrue(result == "Hello ");}
static string ParameterizedTest(string target, string suffix) { var result = StringExtensions.TrimEnd(target, suffix); return result;}
17
Goal: Given a program with a set of input parameters, automatically generate a set of input values that, upon execution, will exercise as many statements as possible
Observations: Reachability not decidable, but
approximations often good enough Encoding of functional correctness checks
as assertions that reach an error statement on failure
Data Generation Challenge
Assumptions and Assertions
void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException();}
Assumptions and assertions induce branches
Executions which cause assumption violations are ignored, not reported as errors or test cases
Data Generation
Human Expensive, incomplete, …
Brute Force Pairwise, predefined data, etc…
Semi - Random: QuickCheck Cheap, Fast “It passed a thousand tests” feeling
Dynamic Symbolic Execution: Pex, Cute,EXE Automated white-box Not random – Constraint Solving
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
Exercise: “CoverMe”Dynamic Symbolic Execution with Pex Open CoverMe project in Visual Studio. Right-click on CoverMe method, and select
“Run Pex”. Inspect results in table.
public static void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
How to test this code?(Actual code from .NET base class libraries)
Motivation: Unit Testing HellResourceReader
Motivation: Unit Testing HellResourceReader
Exercise: “ResourceReaderTest”ResourceReader[PexClass, TestClass]
[PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(typeof(BadImageFormatException))][PexAllowedException(typeof(IOException))][PexAllowedException(typeof(NotSupportedException))]public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}
When does a test case fail? If the test does not throw an exception, it
succeeds. If the test throws an exception,
(assumption violations are filtered out), assertion violations are failures, for all other exception, it depends on further
annotations.
Annotations Short form of common try-catch-assert test code [PexAllowedException(typeof(T))] [PexExpectedException(typeof(T))]
Dynamic Symbolic ExecutionPex Architecture
A brief overview
Code Instrumentation
Z3
Pex
ExtendedReflection
Instrumentation framework “Extended Reflection” Instruments .NET code at runtime Precise data flow and control flow analysis
Code Under Test
Dynamic Symbolic Execution
Z3
Pex
ExtendedReflection
Pex listens to monitoring callbacks Symbolic execution along concrete
execution
Code Under Test
Constraint Solving
Z3
Pex
ExtendedReflection
Constraint solving with SMT solver Z3 Computes concrete test inputs
Code Under Test
Test Generation
Z3
Pex
ExtendedReflection
Execute with test inputs from constraint solver Emit test as source code if it increases
coverage
Code Under Test
Components
Z3
Pex
ExtendedReflection
ExtendedReflection: Code Instrumentation Z3: SMT constraint solver Pex: Dynamic Symbolic Execution
Code Under Test
ldtoken Point::GetXcall __Monitor::EnterMethodbrfalse L0ldarg.0call __Monitor::Argument<Point>
L0: .try { .try { call __Monitor::LDARG_0 ldarg.0 call __Monitor::LDNULL ldnull call __Monitor::CEQ ceq call __Monitor::BRTRUE brtrue L1 call __Monitor::BranchFallthrough call __Monitor::LDARG_0 ldarg.0 …
ldtoken Point::X call __Monitor::LDFLD_REFERENCE ldfld Point::X call__Monitor::AtDereferenceFallthrough br L2
L1: call __Monitor::AtBranchTarget call __Monitor::LDC_I4_M1 ldc.i4.m1
L2: call __Monitor::RET stloc.0 leave L4 } catch NullReferenceException {
‘ call__Monitor::AtNullReferenceException rethrow }
L4: leave L5} finally { call __Monitor::LeaveMethod endfinally }
L5: ldloc.0ret
class Point { int x; static int GetX(Point p) { if (p != null) return p.X; else return -1; } }
Prologue
Epilogue
Calls will performsymbolic computation
Calls to build path condition
Calls to build path condition
Record concrete values to have all information
when this method is calledwith no proper context(The real C# compiler
output is actually more complicated.)
pp
null
p == null
Code Instrumentationby example
Pex’s representation of symbolic values and state is similar to the ones used to build verification conditions in ESC/Java, Spec#, …Terms for Primitive types (integers, floats, …), constants, expressions Struct types by tuples Instance fields of classes by mutable ”mapping of references
to values" Elements of arrays, memory accessed through unsafe
pointers by mutable “mapping of integers to values"
Efficiency by Many reduction rules, including reduction of ground terms to
constants Sharing of syntactically equal sub-terms BDDs over if-then-else terms to represent logical operations Patricia Trees to represent AC1 operators
(including parallel array updates)
Symbolic State Representation
SMT-Solver (“Satisfiability Modulo Theories”) Decides logical first order formulas with respect to
theories SAT solver for Boolean structure Decision procedures for relevant theories:
uninterpreted functions with equalities,linear integer arithmetic, bitvector arithmetic, arrays, tuples
Model generation for satisfiable formulas Models used as test inputs
Limitations No decision procedure for floating point arithmetic
http://research.microsoft.com/z3
Constraint Solving: Z3
Pex inVisual StudioParameterized Unit Testing
Test Project
Test Generation WorkFlow
Code-Under-Test
Project
Parameterized Unit Tests
Pex Generated Tests
Right-click on the code-under-test project In the Solution Explorer
Select “Pex” Select “Create Parameterized Unit
Test Stubs”
Creating a (Parameterized)Test Project
(Parameterized) Test Project
Generated Test ProjectReferencesMicrosoft.Pex.Framework(and more)
ReferencesVisualStudio UnitTestFramework
Contains source files with generated Parameterized Unit Test stubs
Right-click on Parameterized Unit Test For now, it just calls the implementation,
but the you should edit it, e.g. to add assertions
Select “Run Pex Explorations”
Running Pex from the Editor
Exploration Result View
Exploration
Status
Current Parameterize
d Unit Test
Issue barImportant messages here !!!
Input/Output tableRow = generated testColumn = parameterized test input or output
Input/Output table
Test outcomefiltering
PassingTest
Fix available
New Test
FailingTest
Input/Output table
ExceptionStack traceReview test
Allow exception
Pex Explorer
Pex Explorer makes it easy to digest the results of many Parameterized Unit Tests, and many generated test inputs
Pex from theCommand linepex.exe
Pex.exe test.dll
HTML Reports
Rich information, used by Pex developers Full event log history Parameter
table Code
coverage etc…
HTML rendered from XML report file:It is easy to programmatically extract information!
Annotations: Code Under Test
Tell Pex which Type you are testing Code coverage Exception filtering Search prioritization
[PexClass(typeof(Foo))]public class FooTest { …
Code Coverage Reports
3 Domains UserCodeUnderTest
UserOrTestCode
SystemCode
public void CodeUnderTest() { … }
[PexClass] public class FooTest { [PexMethod] public void ParamTest(…) { …
public class String { public bool Contains(string value) { …
Command line: Filters
Namespace, type, method filters pex.exe test.dll /tf:Foo /mf:Bar
Partial match, case insensitive Full match: add “!”: /tf:Foo! Prefix: add “*”: /tf:Foo*
Many other filters e.g. to select a “suite”: /sf=checkin For more information:pex.exe help filtering
ExerciseData generation
Goal: Explore input generation techniques with TrimEnd
Write or discuss a random input generator for TrimEnd
Adapt your parameterized unit tests for Pex Start over with “Create Parameterized Unit Test Stubs”,
or…▪ Add reference to Microsoft.Pex.Framework▪ Add using Microsoft.Pex.Framework;▪ Add [PexClass(typeof(StringExtensions))] on
StringExtensionsTest▪ Add [PexMethod] on your parameterized tests
Right-click in test and ‘Run Pex Explorations’ Compare test suites
LimitationsIt’s called Parameterized Unit Testing
The yellow event bar notifies about important events, including certain limitations
Event Bar
Click on issue kind for more information
Events View
You should act on these events: Refactor your code, or tell Pex to ignore it in the future, let Pex analyze (“instrument”) more code,
if possible.
Click on a particular event for more information
Understand the issue, and take an action
Instrumenting more code If Pex reports that some code was
uninstrumented, you may tell Pex to instrument and analyze it(if possible)
Instrumenting more code
Code instrumentation on Demand Instrumentation has high performance
overhead Some parts of the code better ignored
Use PexInstrument… attributes
Pex will often suggest and insert those attributes for you
[assembly: PexInstrumentAssembly(“Foo”)]
Pex understand managed .NET code only Pex does not understand native code.
Problem if branching over values obtained from the environment Pex may not automatically detect all such
cases.
Testability
if (!File.Exists(f)) throw ...
File System?
Hidden Complexity
Pex analyzes every executed .NET instruction
Some used libraries may be surprisingly expensive to analyze XML parsing repeatedly converting data between different
representationsvoid Sum(string[] A) { var sum = “0”; foreach(var a in A) sum = (int.Parse(a) + int.Parse(sum)).ToString(); if(sum == “123”) throw new Exception(); }
Don’t do this.
There are decision procedures for individual path conditions, but… Number of potential paths grows
exponentially with number of branches Reachable code not known initially Without guidance, same loop might be
unfolded forever
Explosion of Search Space
Explosion of Search Space
To deal with the path explosion problem, Pex uses search heuristics.
Heuristics try to create “diversity”, i.e. they prefer paths which are so far underrepresented e.g. by their coverage vectors
Exploration Boundaries
In the presence of loops and recursion,Pex could search forever, so Pex has many ways to bound the search
Named arguments of [PexMethod]
Pex will often suggest and update those bounds for you.
[PexMethod(MaxConstraintSolverTime = 4)]
Exploration Boundaries
Configurable bounds include: TimeOut MaxBranches MaxCalls MaxConditions
▪ Number of conditions that depend on test inputs
MaxRuns ConstraintSolverTimeOut ConstraintSolverMemoryLimit
Division and modulo are expensive to reason about
No decision procedures for floating-point arithmetic Pex currently uses heuristic search
techniques for floating-point constraints
Constraint Solving
Constraint Solvingvs. Search Strategies
TestInputs
Constraint System
Execution Path
KnownPaths
Initially, choose ArbitraryPath-constraint
solving is just hard.
Reachability is
undecidable! (Loops)
(With external
functions and dynamic loading,
summariesare difficult.)
Multi-threaded code
Pex only explores single-threaded code
Related approach to explore thread-schedules (but not input parameters): CHESShttp://research.microsoft.com/en-us/projects/chess/
Lack of Test Oracle
Write assertions and Pex will try to break them
Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.
Assertions leveraged in product and test code
Pex can leverage Code Contracts (discussed later)
Object Creation
Pex only uses public methods to configure non-public object fields
Heuristics built-in to deal with common types
User can help if neededvoid (Foo foo) { if (foo.Value == 123) throw …[PexFactoryMethod]Foo Create(Bar bar) { return new Foo(bar);}
Exercise Limitations
Goal: Understand limitations of DSE. PexGoal/PexExpectedGoals Apply Pex to
if (File.Exists(fileName)) PexGoal.Reached();
if (DateTime.Parse(s).Year == 2009) PexGoal.Reached();
class Foo { private Foo() {} }…if (foo != null) PexGoal.Reached();
Patternshow to create objectsTips and tricks
Creating complex objectsProblem:• Constraint solver determines
test inputs = initial state of test• Most classes hide their state (private fields)• State is initialized by constructor, and can be
mutated only by calling methods• What sequence of method calls reaches a
given target state?• There may be no such sequence• In general, undecidable
Two approaches:• (Guided) exploration of constructor/mutator
methods• Testing with class invariants
Example: ArrayList
Specification:
[PexMethod]public void ArrayListTest(ArrayList al, object o){ PexAssume.IsTrue(al != null); int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}
Factory methods Parameterized factories create and configure
objects Explicit: Public static method annotated with
[PexFactoryMethod] Implicit: Pex guesses “best” constructor/setters
Result: Exploration of reachable states Reachable using factory method Reachable within configured bounds Under-approximation of possible states
Object creation:Guided Exploration
We will now describe how you can create objects via the exploration of a class invariant.
Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code
Write special constructor for testing only May be marked as "debug only" Constructor sets fields, assumes invariant
Result: Exploration of feasible states May include states that are not reachable
Object creation:Class invariants
Open project “ArrayListSample”. Explore AddTestExplicit Explore AddTestImplicit
Inspect generated test cases Notice constructor implicitly used by Pex
Exercise:Factory methods
Exercise:Factory method by invariant Edit code of ArrayList class: Add “ad-hoc” Invariant method:
Add DEBUG-only constructor that allows to configure object freely, and then checks invariant:
private bool Invariant() { return this._items != null && this._size >= 0 && this._items.Length >= this._size; }
#if DEBUG public ArrayList(object[] items, int size) { this._items = items; this._size = size; if (!this.Invariant()) throw new Exception(); }#endif
Patternsfor Parameterized Unit TestsTips and tricks
Pattern4A
Assume, Arrange, Act, Assert
[PexMethod]void Add(List target, T value) { PexAssume.IsNotNull(target); // assume var count = target.Count; // arrange target.Add(value); // act Assert(target.Count == count + 1)//assert}
PatternRoundtrip
For an API f(x), f-1(f(x)) = x for all x
void PropertyRoundtrip(string value) { var target = new Foo(); target.Name = value; var roundtripped = target.Name; Assert.AreEqual(value, roundtripped);}
PatternRoundtrip
For an API f(x), f-1(f(x)) = x for all x
void ToStringParseRoundtrip(int value) { string s = value.ToString(); int parsed = int.Parse(s); Assert.AreEqual(value, parsed);}
PatternSanitized Roundtrip
For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x
void ParseToString(string x) { var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate); Assert(normalized == roundtripped);}
PatternState Relation
Observe a state change
void ContainedAfterAdd(string value) { var list = new List<string>(); list.Add(value); Assert.IsTrue(list.Contains(value));}
PatternSame Observable Behavior
Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.
public void ConcatsBehaveTheSame( string left, string right) { PexAssert.AreBehaviorsEqual( () => StringFormatter.ConcatV1(left, right), () => StringFormatter.ConcatV2(left, right));}
PatternCommutative Diagram Given two implementations f and g of the
same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.
string Multiply(string x, string y);int Multiply(int x, int y);
void CommutativeDiagram1(int x, int y) { string z1 = Multiply(x, y).ToString(); string z2 = Multiply(x.ToString(), y.ToString()); PexAssert.AreEqual(z1, z2);}
PatternAllowed Exception
Allowed exception -> negative test case
[PexAllowedException(typeof(ArgumentException))]void Test(object item) { var foo = new Foo(item) // validates item// generated test (C#)[ExpectedException(typeof(ArgumentException))]void Test01() { Test(null); // argument check}
PatternReachability
Indicate which portions of a PUT should be reachable.
[PexExpectedGoals]public void InvariantAfterParsing(string input){ ComplexDataStructure x; bool success = ComplexDataStructure.TryParse( input, out x); PexAssume.IsTrue(success); PexGoal.Reached(); x.AssertInvariant();}
PatternRegression Tests
Generated test asserts any observed value Return value, out parameters, PexGoal
When code evolves, breaking changes in observable will be discovered
int AddTest(int a, int b) { return a + b; }
void AddTest01() { var result = AddTest(0, 0); Assert.AreEqual(0, result);}
Exercise Patterns
Goal: Identify and apply patterns Apply patterns to parameterized unit tests
of TrimEnd.
ParameterizedModels
Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.
Integration test: exercises large portions of a system.
Observation: Integration tests are often “sold” as unit tests
White-box test generation does not scale well to integration test scenarios.
Possible solution: Introduce abstraction layers, and mock components not under test
Unit Testing vs. Integration Testing
AppendFormat(null, “{0} {1}!”, “Hello”, “World”); “Hello World!”
.Net Implementation:
public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) {
if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf =
(ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …
ExampleTesting with Interfaces
Introduce a mock class which implements the interface.
Write assertions over expected inputs, provide concrete outputs
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); }}
Problems: Costly to write detailed behavior by example How many and which mock objects do we need to
write?
Stubs / Mock Objects
Parameterized Mock Objects - 1 Introduce a mock class which implements the
interface. Let an oracle provide the behavior of the mock
methods.
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); return o; }}
Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 92
Exercise: “AppendFormatSample”AppendFormat
Show AppendFormat code Show AppendFormat test Run Pex
93
Chosen values can be shaped by assumptions
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); PexAssume.IsTrue(o is ICustomFormatter);
return o; }}
(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)
Parameterized Mock Objects - 2
94
Models from Choices
Choices to build parameterized models
class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue<string,string> files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}
Exercise Parameterized Models
Goal: Test a ‘CopyFile’ method with a File System model
Write a CopyFile method using IFileSystem
Implement IFileSystem with System.IO.File and use it to test Copy.
Write a parameterized model for IFileSystem and use it to test Copy.
interface IFileSystem { string ReadFile(string fileName); void WriteFile(string name, string content);}
Code Contracts SupportDesign By Contracts meets Automated Whitebox Testing
Code Contracts
http://research.microsoft.com/en-us/projects/contracts/
Library to state preconditions, postconditions, invariants
Supported by two tools: Static Checker Rewriter: turns Code Contracts into runtime
checks Pex analyses the runtime checks
Contracts act as Test Oracle Pex may find counter examples for
contracts Missing Contracts may be suggested
If present, Pex leverages [ContractInvariantMethod] to create objects via reflection and checking if invariant holds: Install Code Contracts:
Add reference to Microsoft.Contracts library Add empty [ContractInvariantMethod] Run Pex, click on “Add Invariant” Repeat, edit manually
Exercise: (optional)Class invariants with Code Contracts
[ContractInvariantMethod]protected void Invariant() { Contract.Invariant(this._items != (object[])null); Contract.Invariant( (uint)(this._size) <= (uint)(this._items.Length));}
Code Contracts allow specification of interface contracts
Pex‘ stubs framework can leverage contracts to check input and restrict output of mock objects.
Consider the following interfaces annotated with contracts (using Spec# notation):
interface IFormatProvider { object GetFormat(Type formatType) requires formatType != null; ensures result != null && formatType.IsAssignableFrom(result.GetType());
}
Design-by-contractvs. Parameterized Models
Exercise: Testing with Interface Contracts
Create IFormatProvider interface with Code Contractsin code-under-test project
Create (parameterized) test project Show stubs Create parameterized unit test using
IFormatProvider Explore parameterized unit test
Wrapping up
Some Related Toolsfor Data-Generation Random input generators: many Combining random testing and constraint
solving DART, CUTE, EXE, KLEE, CREST (C) jCUTE, jFuzz (Java) SAGE (X86) …
Program model checkers JPF, Kiasan/KUnit (Java), XRT (.NET)
Commercial tools AgitarOne, …
103
Pex on MSDN DevLabsIncubation Project for Visual Studio 2010
Thank you
http://research.microsoft.com/pexhttp://codeplex.com/Pexhttps://sites.google.com/site/asergrp/
top related