code hunt: experience with coding contests at scale judith bishop, r nigel horspool, tao xie,...

Code Hunt: Experience with Coding Contests

at ScaleJudith Bishop, R Nigel Horspool, Tao Xie,

Nikolai Tillmann, Jonathan de HalleuxMicrosoft Research, University of Victoria and UIUC

Working and learning for fun

Enjoyment adds to long term retention on a taskDiscovery is a powerful driver, contrasting with direct instructionsGaming joins these two, and is hugely popularCan we add these elements to coding?

Code Hunt can!

www.codehunt.com

Websites

GameProjectCommunityData ReleaseBlogs

www.codehunt.comresearch.microsoft.com/codehuntresearch.microsoft.com/codehuntcommunitygithub.com/microsoft/code-huntLinked on the Project page

The Code Hunt Game

Code Hunt

• Is a serious programming game• Works in C# and Java (Python coming)• Appeals to coders wishing to hone their programming skills• And also to students learning to code• Code Hunt has had over 300,000 users since launching in March 2014

with around 1,000 users a day• Stickiness (loyalty) is very high

The Story

Gameplay

1. User writes code in browser2. Cloud analyzes code – test cases show differences

As long as there are differences: User must adapt code, repeatWhen they are no more differences: User wins level!

secret

code

test cases

It’s a game!

1. iterative gameplay2. adaptive3. personalized4. no cheating5. clear winning criterionScore is based on • how many puzzles solved, • how well solved, and • when solved

secret

code

test cases

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

T

Constraints to solve

a!=null a!=null &&a.Length>0

a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Done: There is no path left.

Dynamic Symbolic Execution

Available in Visual Studio since 2010(as Pex and Smart Unit Tests)

Powered by a IntelliTest in Visual Studio 2015

Nikolai Tillmann, Jonathan de Halleux, Tao Xie:Transferring an automated test generation tool to practice: from pex to fakes and code digger. ASE 2014: 385-396

Code Hunt Contests

Code Hunt Contests

Code Hunt Properties Code Hunt Contests

Browser-based Have world-wide reach on all platforms

Automatic grading Are cost effective

Cloud based hosting Scale to 100,000s players

Clear scoring criteria Determine winners precisely

Results available Can identify top coders

The game is fun, fresh and different.

The ah-ha moment

Code Hunt can identify top coders

30 Contests in the past year

What the player sees

Dashboard and Leaderboard

Publically visible when the contest ends

Visible only to the organizer

Puzzles and Analytics

Puzzles

• About 400 puzzles• Classified and rated by difficulty• Puzzle Bank is highly protected

Puzzle Group Description SourceDifficult

yCalculat

ed Players

P249-1 numbers Compute 23*X-42 NH 2 1 335

P277 string Convert N to base 4 string NH 2 3 332

P115 array Convert array of scores to letter grades APCS 3 2 329

P002 array Count occurrences of a string in an array APCS 1 2 325

P260-1 bools Compute x && (y || !z) NH 1 2 325

Number: P067Group: ArraysDescription: Remove duplicates

from an arraySource: NHDifficulty: 2

Puzzle Bank Statistics

Modify the Calculated rating based on user experience.

D = a + tries * b + tries * c * distancewhere a, b and c are constants with values

a = 1 b = 0.02 c = 0.05

tries is the average attempts for all players who solved the puzzledistance is the number of levels solved so far.

Tries does depend on• Mix of students who enter• The internet speed• Whether students use an IDE in addition

Laser, Sept 2014

Original Difficulty

Average Calculated Difficulty s

Total Try Count

Average Try Count

Max Try Count

Total Solved Users

24 2.21 4.601.1 2 1.79 106 8.83 27 121.2 2 1.42 56 4.67 13 121.3 3 2.42 158 15.80 45 101.4 2 2.43 130 13.00 31 101.5 2 4.05 211 23.44 73 91.6 1 3.29 122 15.25 52 82.1 3 4.29 116 19.33 43 62.2 3 1.87 23 4.60 7 52.3 2 1.84 20 4.00 8 52.4 2 9.05 70 35.00 68 22.5 1 3.10 42 8.40 27 52.6 2 18.10 190 63.33 138 3

How players perceive difficulty

Beauty of Programming Subjective difficulty

Perceived difficulty

Players who started

Qualification 1.59 2.72 1511Preliminary A 2.17 1.84 251Preliminary B 2.50 1.84 247Semi-Final 2.60 2.22 140

CSTA and TEALS Subjective difficulty

Calculated difficulty

Players who scored

Students 1.96 5.22 61Teachers 1.96 4.38 14

Code Hunt - the APCS (default) Zone

• Opened in March 2014• 129 problems covering the Advanced Placement Computer Science course• By August 2014, over 45,000 users started.

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.80

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

APCS Zone, First three sectors, 45K to 1K

Sector and Level

Play

ers

Effect of difficulty on drop off in sectors 1-3

Yellow – DivisionBlue – OperatorsGreen - Sectors

Aug 2014 and Feb 2015

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8

-10

0

10

20

30

40

50

60

Effect of Puzzle Difficulty on Drop off

Aug Feb-A

Perc

enta

ge D

rop

Off

Puzzle Level Aug Feb-ACompute -X 1.1 17 22Compute 4 / X 1.6 18 21Compute X-Y 1.7 18 22Compute X/Y 1.11 32 38Compute X%3+1 1.13 15 18Compute 10%X 1.14 12 16Construct a list of numbers 0..N-1 2.1 37 48Construct a list of multiples of N 2.2 19 23Compute x^y 3.1 11 18Compute X! the factorial of X 3.2 16 19Compute sum of i*(i+1)/2 3.5 17 22

One week 3-10 October, 2014

One year May 2014 – April 2015

Survey results – before and after contest season How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution?

Till Sept 2014

Sept 2014 – May 2015

Survey Results cont.In your opinion, were your final solutions well-structured code?

Till Sept 2014

Sept 2014 – May 2015

Gender and Code Hunt Female Male

Respondents 12% 80%

Played as practice for a course in C#

11% 18%

Played as practice for a course in Java

40% 23%

Played for own enjoyment

47% 57%

Open survey with 850 respondents

Beyond contests

Towards a Course Experience

Total Try Count

Average Try Count

Max Try Count

Total Solved Users

13374 363 1306 1581

Public Data release in open source

For ImCupSept257 users x 24 puzzles x approx. 10 tries = about 13,000 programs

For experimentation on how people program and reach solutions

Github.com/microsoft/code-hunt

Upcoming events

PLOOC 2015 at PLDI 2015, June 14 2015, Portland, OR, USA

CHESE 2015 at ISSTA 2015, July 14, 2015, Baltimore, MD, USA

Worldwide intern and summer school contests

Public Code Hunt Contests are over for the summer

Special ICSE attendees Contest. Register at

aka.ms/ICSE2015

Code Hunt Workshop February 2015

Summary:Code Hunt: A Game for Coding

1. Powerful and versatile platform for coding as a game2. Unique in working from unit tests not specifications3. Contest experience fun and robust4. Large contest numbers with public data sets from cloud data

• Enables testing of hypotheses and making conclusions about how players are mastering coding, and what holds them up

5. Has potential to be a teaching platform• collaborators needed

Websites

GameProjectCommunityData ReleaseBlogsOffice MixICSE Contest

www.codehunt.comresearch.microsoft.com/codehuntresearch.microsoft.com/codehuntcommunitygithub.com/microsoft/code-huntLinked on the Project pagemix.office.comaka.ms/icse2015 to registerAka.ms/icse2015codehunt to play

Grazie!

Appendix

Results and rewards

Detailed results are presentedRanking is always based on score, not on time or attemptsScore is based on • how many puzzles solved, • how well solved, and • when solved

Special rewards for contests

Comparing Code Hunt

Code Hunt

A serious online coding game

Built on symbolic execution

Addressing various audiences

Data available in the cloud

IPCC

Annual programming contest

Scored by unit testing

Addressing university students

Questions available on the web

BoP China 2014