Tutorial

Pex4Fun:Teaching and Learning

Computer Sciencevia Social Gaming

http://pex4fun.com

Nikolai Tillmann, Jonathan de Halleux, Judith Bishop, Michal Moskal

Microsoft Research

Tao XieNCSU

http://bit.ly/ioa4qf

Pex4Fun:Teaching and Learning

Computer Sciencevia Social Gaming

http://pex4fun.comAgenda

• Background:Code Analysis byDynamic Symbolic Execution

• Writing Code in a Browser• Coding Duels• Social Experience• Teaching Platform• Conclusion

DYNAMIC SYMBOLIC EXECUTIONIN THE CLOUD

Pex – The Pex4Fun Engine – History

Before the cloud…

http://research.microsoft.com/pex/

http://pex4fun.com/CoverMe

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

T

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Generates test data systematically

Done: There is no path left.

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Dynamic Symbolic Execution Exercises

• ArrayIndexLengthPex knows about all implicit, exception-throwing control-flow branches

• ArrayHeapPex models the heap

• Assert, Assert123Assertions connect code coverage and correctness

http://pex4fun.com/DynamicSymbolicExecutionExercises

Limitations

• 16K characters of code (single file / editor window) *

• Single-threaded only• No environment interactions *• No non-determinism *• Try to avoid floating-point computations

* Limitation only applies to pex4fun.com, but not Pex tool

Dynamic Symbolic ExecutionSummary

• “Ask Pex” sends code to cloud• Code is compiled and analyzed in cloud• Dynamic Symbolic Execution automatically

finds relevant interesting test inputs that achieve high code coverage

• Results are shown in browser

WRITING CODEIN A BROWSER

Code Auto-Completion

Auto-Completion

Writing Code in BrowserExercise

1. Go to http://pex4fun.com2. Click “New”3. Write some code with public static “Puzzle” method

using System;class Vector { int x, y; public Vector(int x, int y) { this.x = x; this.y = y; } public int Length() { return (int)Math.Sqrt(x*x+y*y); } public Vector Normalize() { return new Vector(x/Length(), y/Length()); } public bool Equals(Vector other) { return this.x==other.x && this.y == other.y; }}public class Program { public static void Puzzle(Vector v) { var n1 = v.Normalize(); var n2 = n1.Normalize(); if (!n1.Equals(n2)) throw new Exception(“bug in normalize”); }}

4. “Ask Pex!” http://bit.ly/lOgPZJ

http://pex4fun.com/ParameterizedUnitTesting

Writing Code in BrowserSummary

• Code is compiled in cloud• Code is executed in cloud• Auto-completion via cloud/Javascript

in browser

CODING DUELSFun and Engaging Serious Game – Win by Writing Code

Coding Duels

Coding Duels

• Pex computes “semantic diff” in cloud code written in browser vs. secret reference implementation

• You win when there are no differences

secret

Coding Duelsclass VisibleProgram { public static int Puzzle1(int x) { return x;} }class SecretProgram { public static int Puzzle2(int x) { return x * x;} }

public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!");} }

Coding DuelsTeaching Debugging/Problem Solving Skills

• From Hypothesis to Theorem– Step 1: Inference of hypothesized (delta) requirement• When delta: Inference of failure-inducing conditions• If difficult: forced generation of new data points

– By adding data-point-fitting conditionals

– Step 2: Writing of requirement-satisfying solution• When delta:

– Derivation of hypothesized faulty line(s) of code– Derivation of patches on the faculty line(s) of code

– Step 3: “Ask Pex” to validate the hypothesis

Initial almost-complete (delta) working implementation helps teach debugging skills

Coding DuelsTeaching Abstraction Skills

• Step 1: Inference of hypothesized (delta) requirement from observed data points– Pattern recognition– Generalization from data points to formula– Inversion of constraint solving

Teaching Programming, Algorithm, and Program Understanding Skills

• Step 2: Writing of requirement-satisfying solution

• Many choices, e.g. recursion vs. loop implementation

Teacher can give guidance:• Providing hints in comments • Providing initial almost-complete

working implementation

Coding DuelsExercise

1. Go to http://pex4fun.com; optionally: sign in with Windows Live ID2. Click “New”3. Write secret implementation with public static “Puzzle” method

using System;public class Program { public static int Puzzle(int x, int y) { return x + y – 42; }}

4. “Ask Pex!” 5. Enter Coding Duel Name6. Click “Turn This Puzzle Into A Coding Duel”7. Open Permalink8. Edit Description9. Click “Permalink”10. Use it, or create ulta-short version, e.g. with http://bit.lyhttp://pex4fun.com/CreatingAndPublishingCodingDuels

Coding DuelsFun and Engaging

• Iterative gameplay• Adaptive• Personalized• No cheating• Clear winning criterion

Coding DuelsSummary

• Coding Duel: write function that matches specification given by another implementation

• Semantic equivalence checked in cloud• Student’s tasks:– Inference of hypothesized (delta) requirement– Writing of requirement-satisfying solution

http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels

SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment

Social Experience

Social ExperienceSummary

• Community• High score lists, leaderboard• Live feed

http://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx

PEX4FUN, A TEACHING PLATFORMPages – Courses – Automated Assessments

Teaching

TeachingSummary

• How to become a teacher:– Sign in, choose nickname– Send us email with nickname

• Teachers can– Create courses– Reuse existing or author existing pages– Get automated assessments of students– See students code

http://pex4fun.com/TeachingWithPexForFun

Outlook I

• Analyzing code submissions– Detecting duplicate submissions– Ranking submissions (how correct, how beautiful)– If faulty, give user guidance on “the next step”.

Outlook II

• Better code development experience– Debugging– Dynamic input/output table– More languages

• Beyond the web– Phone

More ReadingNikolai Tillmann, Jonathan de Halleux, and Tao Xie

Pex for Fun: Engineering an Automated Testing Tool for Serious Games in Computer Science

Microsoft Research, Technical reportMSR-TR-2011-41, March 2011

http://research.microsoft.com/apps/pubs/?id=147143

Conclusion• Taking programming into the browser, cloud– .NET: C# (with Intellisense), Visual Basic, F#– Execution, Code analysis with Pex

• Fun learning experience with Coding Duels• Social experience: Live feed, sharing duels• Teaching with automatic grading

• Create your own courses!

http://pex4fun.com