Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger Nikolai Tillmann, Jonathan de Halleux

Tao Xie Microsoft Research

Univ. Illinois

at Urbana-Champaign

(Automated) Test Generation

Human Expensive, incomplete, …

Brute Force Pairwise, predefined data, etc…

Tool Automation!!

MSR Automated Test Generation Tool: Pex & Relatives

Pex (released on May 2008) 30,388 download# (20 months, Feb 08-Oct 09) Active user community: 1,436 forum posts

during ~3 years (Oct 08- Nov 11) Moles (released on Sept 2009)

Shipped with VS 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio

editions” got 1,457 community votes Code Digger (released on Oct 2008 for VS

08/10, on Apr 2013 in VS Gallery for VS 12) 22,466 download# (10 months, Apr 13-Jan 14)

http://research.microsoft.com/pex/

Example Comments for Code Digger in VS Gallery

“Great tool to generate unit tests for parameter boundary tests. I like to see it integrated into Visual Studio and the testing features as far as in ReSharper! :)”

“What an awesome tool.. Help us to explore our logic by providing accurate input parameter for each logic branch.. You should try this as one of your ultimate tool :) It really saves a lot of our time to explore every logic branch in our apps..”

Pex4Fun & Code Hunt

1,565,195 clicked 'Ask Pex!'

http://pex4fun.com/

https://www.codehunt.com/

What are Behind Pex

NOT Random: Cheap, Fast “It passed a thousand tests” feeling

…But Dynamic Symbolic Execution:

e.g., Pex, CUTE,EXE White box Constraint Solving

Dynamic Symbolic Execution

Code to generate inputs for:

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==1234567890

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Observed constraints

a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890

a!=null &&a.Length>0 &&a[0]==1234567890

Data

null

{}

{0}

{123…}a==null

a.Length>0

a[0]==123…T

TF

T

F

F

Execute&MonitorSolve

Choose next path

Done: There is no path left.

Negated condition

MSR Automated Test Generation Tool: Pex & Relatives

Pex (released on May 2008): 30,388 download# (20 months, Feb 08-Oct 09) Active user community: 1,436 forum posts

during ~3 years (Oct 08- Nov 11) Moles (released Sept 2009)

Shipped with VS 12 as Fakes “Provide Microsoft Fakes w/ all Visual Studio

editions” got 1,457 community votes Code Digger (released on Oct 2008 for VS

08/10, on Apr 2013 in VS Gallery for VS 12) 22,466 download# (10 months, Apr 13-Jan 14)

How to make such successful case????

Lesson 1. Evolving “Dreams”

void TestAdd(ArrayList a, object o) { Assume.IsTrue(a!=null); int i = a.Count; a.Add(o); Assert.IsTrue(a[i] == o);}

Parameterized Unit Tests Supported by Pex

Moles/Fakes

Code Digger

Pex4Fun/Code Hunt

Surrounding (Moles/Fakes) Simplifying (Code Digger) Retargeting (Pex4Fun/Code

Hunt)

Lesson 2. Dealing with “Chicken and Egg” Problem

Developer/manager: “Who is using your tool?”

Pex team: “Do you want to be the first?” Developer/manager: “I love your tool but

no.”

Tool Adoption by (Mass) Target Users

Tool Shipping with Visual Studio

Macro Perspective

Micro Perspective

Lesson 2. Dealing with “Chicken and Egg” Problem cont.

Tackle real-world challenges Select/demo applying Pex on real-world cases (e.g.,

ResourceReader) beyond textbook examples Select/demo applying Moles to well address important

scenarios (e.g., unit testing of SharePoint code) Address technical and non-technical barriers

for technology adoption in industry (e.g., tool license)

Incremental shipping (e.g., shipping Code Digger before full-fledge Pex)

Find early adopters Provide quantitative info (reflecting tool’s

importance or benefit extent) Not all downloads are equal! (e.g., those from

Fortune 500)

Lesson 3. Human Factors – Generated Data Consumed by Human

Developer: “Code digger generates a lot of “\0” strings as input. I can’t find a way to create such a string via my own C# code. Could any one show me a C# snippet? I meant zero terminated string.”

Pex team: “In C#, a \0 in a string does not mean zero-termination. It’s just yet another character in the string (a very simple character where all bits are zero), and you can create as Pex shows the value: “\0”.”

Developer: “Your tool generated “\0”” Pex team: “What did you expect?” Developer: “Marc.”

Lesson 3. Human Factors – Generated Name Consumed by Human

Developer: “Your tool generated a test called Foo001. I don’t like it.”

Pex team: “What did you expect?” Developer:“Foo_Should_Fail_When_Bar_Is_Ne

gative.”

Lesson 3. Human Factors – Generated Results Consumed by Human

Object Creation messages suppressed (related to Covana by Xiao et al. [ICSE’11])

Exception Tree View Exploration Tree

View

Exploration Results View

Lesson 4. Best vs. Worst Cases

public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++)

if (y[i] == 15) x++;

if (x == 110) return true;

} return false;}

Key observations: with respect to the coverage target not all paths are equally promising

for branch-node flipping not all branch nodes are equally

promising to flip

• Our solution: – Prefer to flip branch nodes on the most promising paths

– Prefer to flip the most promising branch nodes on paths

– Fitness function to measure “promising” extents

Fitnex by Xie et al. [DSN’09]

To avoid local optimal or biases, the fitness-guided strategy is integrated with Pex’s fairness search strategies

http://research.microsoft.com/apps/pubs/default.aspx?id=81089

Lesson 5. Tool Users’ Stereotypical Mindset or Habits

“Simply one mouse click and then everything would work just perfectly” Often need environment isolation w/ Moles/Fakes or

factory methods, … “One mouse click, a test generation tool would

detect all or most kinds of faults in the code under test” Developer: “Your tool only finds null references.” Pex team: “Did you write any assertions?” Developer: “Assertion???”

“I do not need test generation; I already practice unit testing (and/or TDD). Test generation does not fit into the TDD process”

Lesson 6. Listening to Practitioners

Gathered feedback from target tool users Directly, e.g., via

MSDN Pex forum, tech support, outreach to MS engineers and .NET user groups

Indirectly, e.g., via interactions with MS Visual Studio team (a tool

vendor to its huge user base) Motivations of Moles

Refactoring testability issue faced resistance in practice

Observation at Agile 2008: high attention on mock objects and tool supports

Lesson 7. Collaboration w/ Academia

Win-win collaboration model Win (Ind Lab): longer-term research innovation, man

power, research impacts, … Win (Univ): powerful infrastructure,

relevant/important problems in practice, both research and industry impacts, …

Industry-located Collaborations Faculty visits, e.g., Fitnex, Pex4Fun Student internships, e.g., FloPSy, DyGen,

state cov Academia-located Collaborationshttp://research.microsoft.com/pex/community.aspx#publications

Lesson 7. Collaboration w/ Academia

Academia-located Collaborations Immediate indirect impacts, e.g.,

Reggae [ASE’09s] Rex MSeqGen [FSE’09] DyGen Guided Cov [ICSM’10] state coverage

Long-term indirect impacts, e.g., DySy by Csallner et al. [ICSE’08] Seeker [OOPSLA’11] Covana [ICSE’11]

http://research.microsoft.com/pex/community.aspx#publications

Summary

Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code

Hunt Lessons in transferring tools

Evolving Dreams “Chicken and Egg” Problem Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia

Thank you

http://research.microsoft.com/pex

Summary

Pex practice impacts Moles/Fakes, Code Digger, Pex4Fun/Code

Hunt Lessons in transferring tools

Evolving Dreams “Chicken and Egg” Problem Human Factors Best vs. Worst Cases Tool Users’ Stereotypical Mindset or Habits Practitioners’ Voice Collaboration w/ Academia