usaco: directions for the next decade more participants from a total of more than 65 countries on a...

USACO: Directions For The Next Decade

Rob Kolstad, Ph.D. – USACO Head Coach

ABSTRACT

The last decade of USACO and worldwide programming competitions saw huge growth inservices and participation. While USACO internet contests (via email) commenced in 1996 withjust a few countries participating, more recent widespread internet penetration has enabled onlinecompetitions around the world including the popular USACO contests that now garner a thousandor more participants from a total of more than 65 countries on a monthly basis.

The USACO effort appears to have reached a steady state for its current thrust towardalgorithmics contests, surprisingly with only four dozen USA participants at the highest (Gold)level and a total of fewer than 250 USA participants across all three competition levels. While thebest students acquit themselves well at the world level (USA teams consistently place in the topfive at the IOI, most often at #2 or #3 in the unofficial rankings), it is unclear that the overall goalsof USACO are being met at proper levels since participation is so very low.

This white paper examines the state of programming competitions in general, some of theissues that limit current participation, and suggests a host of new proposals for increasingUSACO’s ability to accomplish its mission and expand its participation.

USACO and Its Goals

The USACO1 is a volunteer organizationheaded by Director Dr. Don Piele, Head Coach Dr.Rob Kolstad, and Educational Director Dr. BrianDean. After successfully running the InternationalProblem Solving Competition (for high school stu-dents) for many years, Don Piele formed the USACOin 1992 to select and train the USA team for the newlyformed International Olympiad in Informatics (IOI),the ultimate international programming contest forpre-college students. The United Nations Educational,Scientific and Cultural Organization2 (UNESCO) cre-ated the IOI3 in 1989 to follow in the footsteps of thesuccessful international olympiads in mathematics,chemistry, physics, and others.

The extremely competitive IOI pits extraordinaryprogrammers (and some average programmers) fromaround the world against each other in two five-hourcompetitions to demonstrate mastery of algorithmicprogramming.

Traditionally, the USACO’s goals are to:

• Provide pre-college students around the worldwith opportunities to sharpen their computerprogramming skills to enable them to competesuccessfully at the international level.

• Enhance the quality of pre-college computereducation by providing students and teacherswith challenging problems, training materials,and competitions which emphasize algorithmdevelopment and problem-solving skills.

1http://www.usaco.org2http://www.unesco.org .3http://ioinformatics.org/index.shtml , http://olympiads.win.

tue.nl/ioi/ , http://en.wikipedia.org/wiki/International_Olympiad_in_Informatics .

• Recognize those students with outstandingskills in computer science and encourage themto pursue further opportunities in the profes-sion.

• Provide educational, motivational, and com-petitive materials in the form of programmingcompetitions and web-based training to studentsin the United States and over 75 other countriesvia the Internet.

USACO accomplishes these goals mostly successfully.

The background goal for all these items is theincrease of talented students entering computer sci-ence and allied fields (which covers a very large num-ber of areas of study). This includes provision of apeer group for talented high school computer scienceestudents, and stimulating pre-college students tostudy computer science along with considering acareer in that field.

Generally, the USACO goes about accomplish-ing the goals above with its training web site, train-ing camp, internet-based competitions, and interna-tional travel. The training camp serves as both rewardand a final set of competitions to choose the USA rep-resentatives at the IOI, which is held in a differentcountry every year. A trip to the IOI is the ultimateprize for a USACO competitor. Additionally, USACOoften takes a team to the Central European Olympiadon Informatics (as a ‘‘warmup’’ and first internationalcompetition experience).

While the USACO is indeed accomplishing itsgoals, the scale of those accomplishments makes a toosmall an impact across all the industries that needcomputer scientists and engineers. Participationremains at the ˜250 student level (out of a high schoolpopulation estimated at 16,100,0004 – a penetration

Version 0.9 Preprint; Limited Distribution 1

USACO: Directions For The Next Decade Kolstad

level of only 0.00155%). USACO should probablystrive for a penetration level closer to 100,000 stu-dents (0.62%).

Contest Participants by Country

1 ARE 25 CAN 1 ESP 23 IDN 6 MEX 2 SCG 16 TWN5 ARG 205 CHN 5 EST 19 IND 5 MKD 7 SGP 17 UKR5 AUS 3 COL 10 FRA 23 IRN 11 MNG 3 SVK 260 USA1 AUT 16 CUB 1 GBR 1 ISR 4 NED 2 SVN 2 VEN

15 BGD 3 CZE 46 GEO 18 KAZ 3 NZL 1 SWE 32 VNM39 BGR 16 DEU 1 GHA 1 KHM 1 PAK 4 SYR 17 YUG

1 BIH 1 DNK 5 GRC 6 KOR 2 PER 8 THA 7 ZAF38 BLR 1 DOM 3 HKG 7 LTU 10 POL 6 TKM 1 ZWE

1 BOL 1 ECU 8 HRV 10 LVA 34 ROM 1 TTO9 BRA 8 EGY 1 HUN 2 MDA 7 RUS 6 TUR

Table 1: 1,060 contestants from 68 countries who successfully submitted at least one solution for the December,2009 internet-based competition.

Figure 1: Historical participation in USACO internet competitions.

Current USACO Status

As an organization, the USACO currently hasthree long-term contributors:

• Don Piele, Ph.D., founder and currently direc-tor. Don has retired from the University of Wis-consin-Parkside and moving toward a goldenlife in Arizona.

• Brian Dean, Ph.D., Educational director. Brianorganizes the instructional and competitive as-pects of training camp and additionally admin-isters a grant for augmenting USACO instruc-tion at the introductory level.5

4http://nces.ed.gov/fastfacts/display.asp?id=65

• Rob Kolstad, Ph.D., head coach. Rob runs thecompetition and training websites, coordinatescompetitions, solicits funding, and organizesthe staffing and non-educational aspects oftraining camp.

Additionally, almost two dozen associate coaches

contribute tasks and solutions for competitions,including directing one of the three levels of the sixannual contests.

The $35-50K annual budget (principally pro-vided by donations from The USENIX Association,Booz Allen Hamilton, IBM, and ita Software) fundsthese ongoing activities:

5NSF CAREER award CCF-0845593, for which the ma-jor thrust of the educational component is the further ex-tension of USACO to younger students through develop-ment of effective on-line materials.

2 Version 0.9 Preprint; Limited Distribution

Kolstad USACO: Directions For The Next Decade

• Six popular monthly internet contests eachannual season. These contests draw anywherefrom 800-1,100 competitors from over 60countries. See Table 1 and Figure 1 to get anidea of the growth and participation levels.

• Annual bonus competitions with guest direc-tors (e.g., China’s or USA’s top competitors)

2002 2003 2004 2005 2006 2007 2008 2009 2010

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

Training Logins/Month

Contest Logins

Figure 2: Historic levels of student logins.

• Year-round internet-based training con-ducted in multiple natural languages. SeeFigure 2 for the level of student visits to thetraining site over time.

• An annual training camp and contest (theUSA Invitational Computing Olympiad –USAICO) for 15-30 of the nation’s best com-petitors (with occasional international guests. Itincludes another half dozen contests (two ofwhich are five hours long), these days with twodivisions. Additionally, multiple teams of atten-dees write AI players for one or more gamecompetitions.

• The Central European Olympiad on Infor-matics (CEOI) during mid-summer.

• The International Olympiad on Informatics(IOI) in late summer. The USACO generallystrives to send the ‘‘best representatives of ourcountry’’ in both technical and non-technicalways vs. simply ‘‘the best programmers.’’

• A new division for training college studentsusing extant ACM competitions from aroundthe world. Entrants compete virtually againstthe actual contestants with scores revealedincrementally as the contest progresses.

About Training

USACO has been training its members (and therest of the world) quite intensely for the IOI for severalyears. Those competitors who diligently solve problemscan see and solve as many as 1,000 tasks (includingmany at other sites) before the IOI. They generallyplace very high in the results. In fact, they do such agood job that the difficulty of the IOI tasks is escalatingas organizers try to differentiate the top finishers. Supe-rior competitors from just five years ago would proba-bly not do so well in the most recent IOI competitions;the task difficulty is growing that quickly.

USACO thus faces a dilemma: should USACOcontinue to train the world and expose all its best ped-agogical technologies to students from all differentcountries? Should USACO keep the best techniquessecret just for US students?

This is a very real issue because the amount oftime it takes to become a top-25 international-levelprogrammer now exceeds 1,000 hours of practicing.While this is commensurate with the cost of athleticsports excellence, it is exceedingly burdensome on‘‘intellectual athletes’’ who have other time-consum-ing interests, as well (including activities that poten-tially yield more ‘‘rewards for the time invested’’ thanUSACO). Increasing the level of competition isliable to reduce both USA’s ability to compete atthe international level and drive potential studentsaway from a contest that is already very difficult.Note that these results are antithetical to USACO’sgoals.



Assessing USACO Participation Challenges

The USACO faces a number of challenges toincreasing its participation:

• Available training time: Serious USACOcompetitors are maxed out on the time theywish to spend training. This negatively impactsUSACO’s ability to recruit students (they don’twish to invest so many hours) and is easilydemonstrated simply by counting the activecontestants, a number that rarely exceeds 250students for the entire USA (across all threedivisions). Fewer than 50 USA student qualifyfor, much less compete successfully at, thehighest level (‘Gold’) division.

• Insufficient critical mass upon which tobuild: In the recent January, 2010 contest, 45USA student participated in the Gold division,and nine of them failed to earn more than 100points out of 1,000 possible. To keep the pre-sentation balanced, note that five of the bestUSA students placed in the top 25 internation-ally. Only China has more students participat-ing in the USACO gold competition than theUSA; these participation issues are not limitedto America.

• Training and competition paradigms whichattract and reward loners rather than encour-aging collaboration and socialization: Tradi-tional algorithmics training techniques demandhundreds of hours focused on solo program-ming. Coupled with geographically dispersedonline competitions, this training style increas-ingly seems to motivate and encourage studentswho are loners, quiet students, and focused onlittle other than programming. It is these stu-dents who earn berths at our training camp.Their conversations at training camp concernalgorithms and mathematics ... and little else.Generally (with a few notable exceptions), theywould not be viewed as well-balanced or par-ticularly socially adept in their interactions withadults and as observed by adults at camp andon trips.

Evaluating Computing vs. Math Contests

One of my internet messaging correspondents isa 16+ year old high school junior of high ability whoattends a public school in silicon valley, California. Heis often quite excited about math contests.

I polled him and almost a dozen other students asto what it was that attracted folks to math contests (mycorrespondent participates in roughly 35 math contestsannually – and he is not active at the national level).

I discovered several significant differentiatorsbetween math contests and computing contests:

• Math contests foster a dramatically largercommunity. This means a competitor has some-

one with whom to share their interest(s) overlunch, someone local with whom to collaborateand compete, and generally is not ‘‘on theirown’’ to train and compete (unlike computingcompetitions).

• Math competitions attract the larger communityin several ways: they often offer team competi-tions; math teachers (who populate everyschool in numbers much greater than computerteachers) encourage competition, often citingthe mantra that ‘‘mathematics is the foundationof everything;’’ every student in almost everyUSA school system is required to take adecade or more of math instruction in orderto complete their high school diploma.The reduced number of computing contestcompetitors in a given community means thatthey are unlikely to congregate frequently (onlya few high schools supply more than one com-petitor to USACO).

• Math competitions offer a different experiencefrom computer competitions. Computer compe-titions (with the rare exception of ACSL) arehardly ever shorter than three hours and thusgenerally occur outside the classroom setting(perhaps via the internet at home or off-site ona weekend). Math competitions range fromunder one to three hours (until the nationallevel is reached). The mathematics mentoringcommunity (which includes high school teach-ers) dwarfs the computer programming mentor-ing community, which is almost non-existent.

• Computer contests dramatically exceed mathcontests in many sorts of difficulty. First of all,programming requires the formulation of an al-gorithm (a task some might argue is mathemati-cal) and then, once the proper algorithm isdeduced, a solution requires a working program(which some have called ‘mechanics’). Makinga small error on either one of these orthogonalrequirements can result in a task that earns nopoints at all.

Secondly, the ongoing training efforts ofUSACO’s site, contests, and other country’s con-tests has elevated the level required to differenti-ate among the top competitors to that of graduatestudents at leading research universities.

Finally, programming contests for high schoolstudents tend to have but a handful of tasks;math contests have 7-40 tasks; a talented stu-dent will have total confidence in getting atleast 2/3 of them correct. Programming contes-tants on challenging contests often have scoresin the 30-50% range, which sooner or latermust be frustrating to students accustomed toperfection.

• My IM correspondents argue about whetheralgorithmic programming contests tend to



focus on a small number of algorithm types,of which only the ‘ad hoc’ and ‘combination’varieties are not relatively standardized. Thismeans that the variety of programming problemsis smaller when compared to the relatively largerof math contest tasks. Simply knowing the maxi-mum value of ‘N’ in a programming task oftenreduces the number of solution types (perhaps toa small handful), thus further shrinking the vari-ety. Some of my correspondents (including inter-national-level programmers and international-level math competitors) believe that this vari-ance in variety is strongly evident; some argue itis a weaker proposition.

The combination of these factors appears to com-bine to yield a feeling that ‘‘Programming contestsare not as much fun as other activities’’ (likeFIRST or math contests) unless you are super-tal-ented and reaping benefits like trips. Furthermore,the properties listed above tend to attract (and cre-ate) loners instead of more socialized, team-ori-ented competitors who then gravitate elsewhere sinceother attractive activities are highly available.

Thus, I argue that the current emphasis onalgorithmic programming contests as a way to fos-ter interest in computing and computing careers isnot in the best interests of either the USACO, itsfellow contest organizers, or its contestants. Thisemphasis surely should not form the only basis foraccomplish USACO’s goals.

Algorithmics is not the only programming chal-lenge and surely is not the most appealing one to ouraudience; it’s also expensive to master.

History of Programming Contests

How did (algorithmic) programming contests getto their current state?

Many modern programming competitions findtheir roots in the ACM college student competitions ofthe 1970s that featured a single keypunch shared by ateam of three students who submitted batch jobswhose results were subsequently returned, printed onfan-fold printer paper.

Modern ACM competitions generally have 9-12tasks (vs. USACO/IOI at 3-4) in a five hour timeframe.ACM contests return some results in real-time, a two-edged sword that raises the excitement level but putstremendous pressure on organizers to supply correct testdata and judges’ solutions (a quality level which toomany ACM contests fail to reach, thus betraying andfrustrating the contestants, some of whom decline toparticipate in future ACM competitions).

In ACM contests, the amount of data returned inthe event of a ‘wrong answer’ is minuscule (e.g., ‘wehave some test data that makes your task fail’). Whileany contest can put forth any rules it likes, this partic-ular paradigm does not reflect real-world program-ming constraints and promotes perhaps excessive

stomach acid in trying to find the programming error.USACO/IOI contests return almost no data in realtime (just a verification that your program can solve asingle trivial case). This means that tiny errors canresult in little or no credit for a task. Neither of theseparadigms yields as much fun as they might (ACMcontests give too little data to find a problem;USACO/IOI contests don’t give results at all untilafter the contest). Few sports decouple performanceand reporting of scores this much (math contestsand other written competitions being the exception).

Appendix A documents a number of modern pro-gramming and other computer-oriented contests. Suchcontests are now widely available throughout theworld for pre-college competitors and others.

Recruiting New Programmers

Recruiting new interested constituents is thelifeblood of our organization. It’s not possible to fosterlong-term interest without starting at short-term interest.

Dutch IOI coach and database expert EljakimSchrijvers has championed the Lithuanian ‘‘Beaver6’’competition throughout central Europe. His examina-tion delivery system annually administers the multi-ple-choice aptitude test to over 80,000 students in Ger-many (and many students in other countries as well).A country’s best finishers take another exam wheresmall prizes are awarded to the highest scorers. Ofcourse, the big benefit is the identification of high-aptitude students to steer toward a training and compe-tition program.

USACO Coach and Clemson professor BrianDean has won a grant from the National ScienceFoundation that includes, as one component, the train-ing of computing newcomers (he has his own paper onthis). Holland’s superstar Jelle van den Hooff has hisown ideas on the creation of tutorials; that effort isongoing in the first part of 2010. It will also encom-pass the majority of USACO’s effort toward teachertraining and programs for educators.

Modern trends in novice training include a moveaway from Java to:

• MIT’s Scratch language – the much modern-ized version of Logo, at some levels: A drag-and-drop procedural language that processesinterrupts just like a real graphics system. Easyto make animations and games; kids pick it upinstantly and almost automatically.

• Python, a relatively new scripting language thatis good for teaching both object- and procedur-al-oriented programming; it includes somegraphics.

• GameMaker7 enables students to create gamesquickly and easily. This is used in some intro-ductory summer programs.

6http://www.bebras.org/en/welcome7http://www.yoyogames.com/gamemaker/



• CS Unplugged8 includes sets of classroom exer-cises (sometimes for very large groups) thatteachers really enjoy for computer-exposurestyle drills.

One of the real problems for early computer train-ing is the lack of teachers with skills and interest inteaching about computer programming and technology.The set of good materials for brick-and-mortar class-rooms appears to be too small. I am a big fan of comingup with a new set of ways to teach computer scienceand its allied concepts that avoid the quagmire of the APprogram with its goal of giving college credit for acourse that no two universities can agree on. Identifyingteacher needs (and teachers!) is a paramount challenge.

I am making no proposals that advocate competi-tive contests for novice programmers. That paradigmis anathema outside a fun Saturday morning teamcompetition where most entries get a perfect score. Abetter paradigm seems to be stepped completion andachievement levels that offer encouragement and fos-ter growth more like ‘‘merit badges’’ than rare com-petitive accolades that only a small fraction of stu-dents will garner.

Merit Badges

Not all rewards come from competitions. Work-ing through a set of requirements to achieve a standardyields a reward in the form of a certificate, patch,book, web-recognition, etc. that many students findrewarding. They work at their own pace and succeedor fail as they wish. Material that grows sequentiallymore difficult can lead to great self-generated rewardsas the challenges are conquered.

Participation Paradigms

While contests are not necessarily the be-all andend-all of promoting computers and their associatedtechnology, they are one strong methodology with aproven track record. Certain kinds of contests (e.g.,projects and associated authored paper competitions)do mimic many of the important qualities of compet-ing in the ‘‘real world.’’

This section discusses new sorts of contests (andother paradigms) that sport novelty and fun that differin small or large ways from the current algorithmiccontest format.

Proposal: Customizable Contests

The current algorithm programming system (andpresumably future systems for other styles of competi-tions) should have a feature for creating customizablecontests for multiple competitors and for trainingexercises.

For multiple contestants, a custom event mightfocus on an intraschool competition, dual meet (e.g.,two or three schools compete against other either on-site or in a distributed manner – and in lockstep time –

8http://www.csunplugged.com

or not), a regional competition (e.g., Cleveland CountyRegional Competition), or something even bigger.

For non-team events, USACO should considerimplementing one-on-one head-to-head ‘‘chal-lenges’’ or ‘‘mini-contests’’ (probably not on-siteand either in lockstep or close to it). These couldeasily lead to a challenge ladder as in tennis andother sports.

USACO should also implement customizablesingle-person practice contests with tasks chosenfrom an array of yet-unseen tasks from chosen cate-gories (perhaps with difficulties and/or ‘par’ times).These contests offer the flavor of competition (mostnotably time limits and interactive paradigms) withoutthe performance pressure of competing against (andpotentially losing to) other real humans.

Proposal: Increased Team Competitions

Perhaps the single most-requested feature forUSACO contests was the addition of teams (an optionwhich was implemented ca. 2008). When queried, stu-dents cited several reasons they like team contests:

• Collaboration (camaraderie)

• Ability to blend into a team (i.e., not stick outwith individual results)

• Reduction in the impact of one person having a‘bad day’

• Socialization, school spirit

• Greater participation

• Ability to bring less-experienced students intoserious competitions

Thus, it’s easy to assert: successful fun contests needto support both individual and team competitors.

Team competitions vary in their levels of collab-oration. Sometimes, a team is simply the sum of agroup of individual scores (perhaps the top four, per-haps a designated team, etc.). Other times, a team is agroup actively collaborating on solving problems (andallied programming issues). USACO should pursueboth of these as appropriate.

Programming Bee

This new style of online or on-site competitionformat includes a set of contestants on the internet anda clever server. The contest proceeds lockstep through aseries of ‘questions’ and ‘tasks’ announced in syn-chrony from a high-speed, well-connected site. Tasks(e.g., what is the value of 2100? Given this set of songtimes, what’s the most we can fit on a 66:00 minuteCD?) might have a time limit (perhaps two minutes for2100) or might have an advancement limit ‘‘the first 100solvers move to next round.’’ Problems progress in dif-ficulty and end up leaving a single champion, the lastperson standing. This competition has the advantages of

• Novelty – it feels like a spelling bee and canprovide great entertainment for observers.

• Fairness – everyone has to solve every problem(unlike a spelling bee where students each spella different word)



• Excitement – shorter time limits and display ofthose progressing make this a really high-pres-sure competition if one wishes, especially withteams (particularly if three team members sharea single keyboard/screen)

Programming Bowl

This is, as it sounds, a College bowl/Sciencebowl style contest for programming, implemented forteams or individuals. Conducted on-site, it includesverbal questions (‘‘What symbols are the block delim-iters in the C programming language’’) and program-ming or computer/net usage questions that mightrequire a computer. Unlike most of the other competi-tion styles, this one doesn’t scale past a dozen or soparticipants. It has the advantage, though, of beingparticularly adaptable to a stage setting with a largeaudience.

Monthly Projects

Jacob Steinhardt proposes another interestingidea for focused students. The organizers present orreference a paper or set of papers (e.g., ‘‘CurrentDirections in Machine Vision’’) and posit a project orchallenge that (teams of) students are to attack. Aftersome appropriate interval (probably a month or two),students submit results (e.g., programs, presentations,or papers) of their work for evaluation and potentiallyawarding of prizes. This sort of idea also works wellin the context of conferences for pre-college students.

AI Games

In AI game competitions, teams of 1-10 playerswrite cooperative (e.g., MIT’s sockey9) or competitive(e.g., Tron light-cycles10) programs that play with oragainst each other in some virtual arena, usually onethat is presentable as a nice real-time graphical display.

AI game competitions can be utilized bothshorter (2-3 hours-to-build) and more intense (3-20not necessarily contiguous hours) formats. USACOshould foster or collaborate with competitions likeKim Schrijvers’ CodeCup (perhaps during the Decem-ber holidays). For on-site games, both IBM’s TimdeBoer and I have frameworks for building newgames.

Students generally enjoy these sorts of competi-tions and do just fabulous work once they commit towinning a game.

Robotics

Robots and robot projects are extremely attractiveto students. Some believe robotics (as implemented bybattlebots and subsequently by Dean Kamen’s FIRSTorganization) is driving the incredible surge of studentsin the mechanical engineering sector. While the

9http://www.ai.mit.edu/lab/olympics/99/events/sockey/docs/how-to.html10In addition to the USACO implementation, U. Waterloo is

hosting a similar competition in early 2010 (http://csclub.uwa-terloo.ca/contest/).

population of CS students is just starting to increaseagain after the incredible decline following the dot combust,11 mechanical engineering schools are full.

I interviewed talented high school academicJacob Hurwitz (Intel Talent Search, programming con-test, Science Fair, and USACO winner) about his par-ticipation in the FIRST program. Among the distin-guishing characteristics that made robots fun, henoted:

• Robot contests create a tangible product thatyou can see and touch; robots do things

• Robot contests enable contestants to make coolresults without years of training

• Robot contests feature a project that can be fin-ished in, often, six weeks

• Robot contests almost universally feature ateam orientation with relatively large teams

• Robot contests are often mentored by skilledengineers who bring new ideas and viewpointsto the traditionally more staid academic setting

• Robot competitions can be made to be extreme-ly exiting (including real-time feedback, lots ofaction, and interaction among dozens of humancompetitors)

• Robot competitions are extremely well-fundedwith travel (a typical single high school FIRSTteam will spend $50,000 on 20 students; afflu-ent teams spend three times that much)

• Robot competitions promote the idea that it iseasy to grow skill

• Robot construction uses modular hardware (andsoftware) that requires problem-solving at verydifferent level from algorithmics

• Robot competitions often offer excellent prizes/rewards (e.g., big scholarships)

It’s easy to see that robot competitions address manyof the shortcomings of traditional USACO contests.

Robot competitions include not only FIRST butalso the Oklahoma-based Botball. I met with Botballfounder board member David Miller12 in late Decem-ber, 2009 to see if USACO’s and Botball’s interestscoincided. Not surprisingly, software is the genericweak point of almost all robot competitions. Despitean absolutely gorgeous embedded system box, theBotball folks suffer some of the same challenges asUSACO. We have agreed to talk in the future to findareas where we can cooperate.

Note that virtual robots can provide a lot of thefun (without tangible mechanical pieces) as they simu-late or model manipulations in a virtual world. TheNetherlands’ Kim Schrijvers knows of virtual compe-titions where winners’ programs are then run on thereal versions of very sophisticated robots that controlexotic devices.

11http://www.cra.org/govaffairs/blog/2009/03/computer-science-majors-increase-at-most-significant-rate-since-dot-com-boom/12http://faculty-staff.ou.edu/M/David.P.Miller-1/



‘‘Computer Clubs’’

Much like the Mu Alpha Theta folks provide anational-level umbrella organization for pre-collegemath clubs, USACO (or an allied organization) shouldperform the same purpose for a kind of computer club,information technology club, or similar association.

I propose a model that exploits the availability ofcomputers, data projectors, high speed bandwidth, andpreciious technology students that most high schoolspossess while acknowledging that teachers often haveneither the available time or advanced knowledge tocreate and present material applicable to an interestedgroup of technology enthusiasts.

I envision a club that meets at least every otherweek (and perhaps weekly). One set of meetings (per-haps every other one) typically runs like this:

• Opening statements along with schedulingreminders (e.g., for contests, trips, etc.)

• A 5-10 minute supplied video presentationfrom academia or industry. This presentationmight show how Seagate manufactures multi-terabyte disks that are never touched by humanhands from the inception of the manufacturingprocess until the disk is installed in a computersystem. Perhaps the video highlights MIT’snew research in digital mapping technologies orCalTech’s latest results in the exploration ofusing animal neurons to create better artificialintelligence. This part of the presentation hastwo purposes:

Show that the world of computer technol-ogy is broad, deep, and utterly fascinating.Pique students’ interests in a wide spec-trum of different sorts of computer- orengineering-oriented careers.

The beautiful side-effect is the promotion of thecompanies and academic institutions that pro-vide the videos. They can penetrate to preciselythe demographic of potential employees/cus-tomers they need over the next few years.

• After the short technical news, an informative,interesting ‘‘Powerpoint Presentation’’ is givenby an interested teacher, mentor, or student.The presentation might cover introductions todatabases, MIT’s Scratch language, system ad-ministration, a programming language (likeLISP, which would be unusual for the highschool set), computer graphics, artificial intelli-gence, networking, compilers, modern commer-cial computer game creation, embedded sys-tems, scripting languages, etc. The presentationis always followed by a hands-on experience toemphasize the concepts presented.

Clubs always need a year-end goal, so members of thisclub would attend a conference (in either western, cen-tral, or eastern USA). The conference would includeinspirational speakers, technical speakers, fun compe-titions, and camaraderie.

On alternate weeks, club members use onlinetutorials to learn programming languages, new tech-nology paradigms, or to construct software/hardwareprojects.

Contest Task Paradigms

The more popular contests these days offer eitheralgorithmic programming competitions or project com-petitions. We need a host of new styles/topics for pro-gramming competitions. A year-long casual survey ofprogressional programmers reveals that a typical pro-fessional programmer will, over the course of an entireyear, implement no more than a single instance of‘‘algorithmic’’ code similar to the sort that is the sub-ject of algorithmic programming competitions. I assertthat we can find interesting competitions and inspi-rational contests by examining other areas of pro-fessional programming.

Changing the layout and/or style of a competi-tion can yield a new contest with a different feel, evenif most elements are similar to the current algorithmiccompetition model (i.e., IOI). Consider:

• Function-only submissions perhaps not toodifferent from Topcoder. By submitting a solu-tion for only the heart of of the task, both theprogramming and the timing for pesky thingslike I/O are removed. This can enable extremelyaccurate timing, huge numbers of test iterations,and a slightly different feel for the contestants.Of course, this idea requires a framework foreach different supported language.

• Contests with many more than three tasks.More like ACM contests but perhaps with a setof tasks growing incrementally in difficulty.This might be most appropriate at lower levelsof competition.

• ‘‘No Limits’’ tasks. These problems have stor-age allocation specifications but no guaranteesabout the maximum size of ‘N’ as far as thealgorithm goes. An O(N4) task might limit N to4,000,000, even though no dataset that big isever solvable. This challenges students to fig-ure out algorithms from task statements ratherthan from the specified limits.

• Real-time scoreboard when appropriate. Justas ACM contests show real-time ongoingscores, our contests should do this too whencontestants are competing simultaneously. (Thisfunctionality is in place for USACO-conductedACM competitions.)

• Virtual scoreboard (including contests used astraining exercises by individuals). When con-testants do not compete cotemporaneously(even when they are participating in a years-oldcontest), show a virtual scoreboard whose con-tents are revealed in virtual lockstep with thecontestants’ clocks. (This functionality is inplace for USACO-conducted ACM competi-tions; watching folks compete against talented



Russians in a contest from 2007 is quite amus-ing.)

• We must increase the prizes and incentivesoffered to contest winners, especially when acompetition extends, e.g., throughout a schoolyear.

• USACO should support personal ‘‘results’’pages that provide scores and results for a stu-dent, enable the student to showcase some oftheir own work (via biographies, pictures, andthe like), and generically provide a bit of socialnetworking capabilities.

• USACO should create a standard recommen-dation program for colleges and universities sothat outstanding competitors can receive somesort of uniform recommendation based on theirperformance in elite USACO competitions.

I do not propose that USACO set up for ‘project com-petitions,’ which are well done by many other organi-zations.

New Task Topics

The refined and matured legacy of ACM contestsfrom the days of keypunches and line printers hasmushroomed into a large number of algorithmic-stylecontests that are, in my opinion, utterly inappropriatefor, e.g., most computing professionals.

We should explore a large set of task topics – notall in the vein of algorithmics – in order to broadenour offerings and demonstrate a wider spectrum ofinteresting programming and computer-oriented chal-lenges. Here’s a list of all sorts of task types:

• The original: algorithms. The most popularcontest-style for almost all 1-5 hour program-ming contests. This includes contests that reallyare about introductory ad hoc tasks.

• Simulations & modeling. The math folks (per-sonified by MAA director Steve Dunbar ofUniversity of Nebraska-Lincoln) are pursuingthis notion. Their prototype task: ‘‘Here’s amodel of a house; install N fire alarms in it sothat you maximize the likelihood a fire isdetected.’’ One can easily imagine modelingcontests for parking lots, baseball field mow-ing, embroidery layout patterns, forest mainte-nance, and a huge number of other real-worldsystems. These sorts of tasks have a relativelydifferent flavor and would probably attract anew set of aficionados.

• System programming, text, and file manipula-tions. Our algorithmic competitors almost uni-formly have no knowledge of efficiency andeffectiveness of the algorithms, techniques, andheuristics that buttress their user-level programs.A contest for efficient implementations of a‘‘ w o r d count’’ program, a ‘‘print a file back-wards program,’’ file manipulation tasks, andthese sorts of things would bring a completely

new flavor to our competitions. Likewise, theidea of writing parsers, lexers, and compilersfalls into this category.

• Databases/Statistics/data mining. The storage,retrieval, manipulation, and analysis of data is agoldmine for tasks that probably last a few hoursand are scored subjectively. Consider:

Find the region where expanded advertisingwill have its greatest impact in the next fis-cal year.Find the telephone traffic pattern that ismost likely to yield intelligence about terror-ist attacks.

• Supercomputing. Parallelism in all its forms isa hot topic in computer science. The supercom-puter conferences are beginning to experimentwith contests for using large supercomputers tosolve applicable tasks. Supercomputers have aspecial allure – just the promise of access couldattract new talent. Imagine: ‘‘Please create avideo of these two galaxies colliding.’’

• Scripting (including Python, PHP [mostly forthe web], Perl, and various shell scripting lan-guages). Potentially in conjunction with otherareas, this area is very rich in application; I per-sonally program 80-90% of my applications inPerl instead of C/C++/Java. Example: ‘‘Files1-N contain entrants in a science fair, teacherinformation, school information, judge infor-mation, and project scores. Determine theteachers with the best performance and reportthe judges whose scores most nearly matchedfinal results. You have 45 minutes.’’

• Artificial intelligence. AI contests tend to bemost attractive when designing collabora-tive/cooperative/competitive AI agents for (vir-tual) games. The previous discussion men-tioned some games already used and referred toKim Schrijvers’ CodeCup competition.

• Speculative: Tasks in a special programming

language. While this seemed a poor idea in thecontext of high speed algorithmic tasks, itseems a great idea in the context of interpretedstatements for a special purpose task or genre.MIT’s Scratch is one such language – it’s pro-grammed graphically with drag-and-drop. Oneof the fun parts here is that the language neednot be precisely procedural; it might be LISP-like or perhaps presented simply as object code(to be created or interpreted), or might utilizesome unusual or unorthodox representation orprogramming technique (think Snobol).

• Embedded applications and languages. Robots(see below) provide one of the most popularembedded applications, but almost any uniqueperipheral can afford some exciting program-ming (perhaps in assembly language, C, or aBASIC version). Plenty of robot contests exist;



see below for comments on cooperating withthe good folks at Botball.13

Conclusion

USACO must expand its offerings if it wishes toachieve its goals of promotion of computing careers inthe pre-college world. Until someone offers the sortsof expanded topics listed above, students will have noidea of the breadth of careers available in the com-puter technology arena (and thus will be diverted toother popular academic majors).

The sorts of competitions discussed in this papercan be built into various contest-administration sys-tems and, at the end of a season, provide a truly mean-ingful competition (perhaps the ‘‘Programming Pen-tathlon’’) where teams (individuals, too?) compete inmany areas to become real programming championsacross a variety of subjects, not just in the narrow andincreasingly deep world of algorithmics.

USACO must seek funding and contributors tobring these plans to fruition.

13http://www.botball.org

Appendix A: Overview of Some Current Com-puter-Oriented Contests

USACO is not the only organization promotingcomputer competitions. The number (if not the styles)of contests continues to grow as online graders (and,more rarely, training sites) proliferate owing to spread-ing skills, maturing tools, and availability of technol-ogy for system protective ‘jails’ that keep submittedcode from harming a host grading server (these jailsare now available on Linux and even, apparently, Win-dows).

The most popular contest sites include:

• USACO – intermediate through advanced train-ing and tasks

• Topcoder – lots of forums, divisions, and cashprizes

• Google Code Jam – cash prizes

• Codechef – India, cash prizes

• COCI – Croatia

• ACSL – Less advanced training (includes lec-tures for teacher assistance)

A list of popular training sites is presented later.

These are complemented by some slightlysmaller competitions:

• Regional contests (OI’s in Central Europe,Balkan area, Baltic area, Pacific)

• National contests in almost every nation partici-pating in the IOI

• The IOI itself

• Smaller regional contests like the Berkeley con-test, HP sponsored contests in Texas, UMd con-test, and dozens more

As further evidence of contest proliferation,2008-2009 USACO champion and IOI gold medalistNeal Wu finds enough online opportunities that hecompetes in 50-60 contests annually, generally devot-ing one to three hours for almost every competition(just a few are longer).

The combination of the higher number andincreased profile of these contests (some with cashprizes), the widespread availability of training, and theongoing penetration of ever-lower cost computing facili-ties has enabled a small number of programming prodi-gies to increase their skills to unprecedented levels.

Additionally, some geographical/political regionsare said to be investing heavily in academic competi-tions, even to the extent of hiring full-time coaches forhigh school competitions. The University of CentralFlorida has instituted a ‘fellowship’ program for itsundergraduates who wish to excel in the ACM ICPCcollege contests. Such a program is indistinguishablefrom paying students to train for competitions. CMUoffers a ‘contest programming course.’14

The concomitant blossoming of programmingproficiency has motivated elite competitions like theIOI to increase the overall difficulty of its six primary

14http://www.andrew.cmu.edu/course/15-295/



tasks significantly in order to differentiate among thetop competitors. The difficulty has increased enoughthat easier, lower-level tasks have been added to thecompetition in order to avoid the growing number ofstudent achieving a score of 0.

While acknowledging that correlation is not cau-sation, USACO conducted a statistical analysis thatcompared the number of training problems solved byUSAICO attendees with overall scores on USAICOtasks. With a single outlying datapoint, the result wasthat the overall performance was almost perfectly cor-related with the number of training tasks solved.

While this would seem to make algorithmic pro-gramming competitions appear to be attaining a level ofstrict meritocracy (which is often regarded as a positiveattribute), the resulting almost insurmountable chal-lenge is that becoming a high-level competitor at thehigh school level now requires somewhere in the rangeof 1,000 to 1,500 hours of practice. While this timecommitment is similar to that of elite athletes likeswimmers and gymnasts, it is a tremendous burden toplace on students who are almost uniformly interestedin allied fields like mathematics and physics which alsorequire similar time commitments. I have personalexperience with teachers who have advised their stu-dents against participating in USACO because the timerequirement is simply too high.

The world of algorithmic programming competi-tions evolves slowly from the viewpoint of competi-tors: programming language C++ was added manyyears ago; Java is now also used in some competi-tions. The administration and development systems(used behind the scenes) are maturing more quickly,but neither uniformly nor in public. Some formaliza-tion of these elite contests has emerged, as well: TheIOI publishes an annual journal of refereed papers onaspects of pre-college competition.

New Contest Styles

As the technology grows and evolves, enthusi-asts have created a host of new sorts of competitionsin a variety of allied fields.

X Prizes

These are serious high-powered, amazingly well-funded competitions. From the website:15

An X PRIZE is a $10 million+ award given tothe first team to achieve a specific goal, set bythe X PRIZE Foundation, which has the potentialto benefit humanity. Rather than awardingmoney to honor past achievements or directlyfunding research, an X PRIZE incites innovationby tapping into our competitive and entrepre-neurial spirits.

The Ansari X prize was won by Virgin Galacticwho won the contest for a non-government created

15http://www.xprize.org

spaceship. Google has upped the ante, also from thewebsite:

The Google Lunar X PRIZE is a $30 million com-petition for the first privately funded team to senda robot to the moon, travel 500 meters and trans-mit video, images and data back to the Earth.

The Progressive Automotive X prize offers $10million to the team that makes a reasonable 100 MPGvehicle. The Archon X Prize offers $7-10 million tothe team that can build a device to sequence 100human genomes in 10 days at a cost of less than $10Kper genome. Other prizes are in development. TheEnergy & Environment Prize Suite is to have a com-bined prize value of as much as $100 million.

DARPA Grand Challenge

Recent editions of the DARPA GrandChallenge16 have encouraged college-level and profes-sional competitors to create autonomous cars that cannegotiate urban driving courses at reasonable speedswithout accidents. First prize: $2,000,000. Secondprize: $1,000,000; third prize: $500,000.

Robot Competitions

Robot competitions are, these days, immenselypopular – and well they should be. Flash, tech, greatrewards, and just the right amount of hand-holdingcombine to make an unbeatable contest combination.The robots.net17 site lists 119 robot competitions; it’sno wonder university mechanical engineering schoolsare bursting at the seams. Mr. Wikipedia knows a largeset of competitions as well.18

Three popular competitions include:

• FIRST19 – an extensive competition whereteams of high school students construct, with amentor ’s guidance, a robot that will (usually)collaborate with another robot to compete in anarena against two other robots to see whichteam can complete a task like moving beachballs around. FIRST has a short autonomoussection of their competition, but is mostly forremote-controlled robots.Extremely well-funded (a team of 15-20 mightwell spend $50,000 annually), the competitionsare also lavishly produced (as is the website),complete with cheerleaders. USACO competi-tor Jacob Hurwitz says, ‘‘I idolize (competitionfounder) Dean Kamen the way other guys idol-ize sports figures.’’FIRST says they have $11,000,000 in scholar-ships this year. They have super participationwith programs for students from elementarythrough high school:

The total FIRST Community – with over196,000 students, 86,000 volunteers, and

16http://www.darpa.mil/grandchallenge/17http://robots.net/rcfaq.html18http://en.wikipedia.org/wiki/Robot_competition19http://www.usfirst.org



3,500 sponsors this year alone – is esti-mated at one million strong and growing.

• Botball20 – Botball has similar goals to FIRSTbut is much smaller and hosts contests withautonomous robots (much more technology)and much lower budgets.

• Robocup21 – Robocup focuses mostly on theuniversity set (though they have other contestsfor other age groups) for their theme which issoccer-playing robots. They have used modi-fied Aibo robot dogs as their players, TV cam-eras, and lots of software to play their robotgames. They are also fostering some other sce-narios like building-rescue for robots.

AI Game-Playing Contests

Some competitions ask students to write one ormore programs that will play in a collaborative orcompetitive setting. Few of these programs requireplayers to collaborate with strangers, generally non-teammates compete.

Probably the best-known AI game is computerchess.22 These days, programming competitive com-puter chess programs is mostly beyond scope of ama-teurs.

Kim Schrijvers and a Dutch team run the Code-cup competition each year. The Codecup developerspresent a new game each season; developers createprograms that play in an arena and periodically intournaments to determine the best program.

IBM’s Tim DeBoer’s CodeRuler, CodeRally,CodeInvaders were developed for the ACM ICPCcompetition. A number of competitors (e.g., four inCoderuler) vie to capture territory, win a race, oraccrue points in colorful arenas projected on largescreens for the enjoyment of the tournament entrants.

Perhaps one of the most interesting examples isMIT’s sockey23 (combination of soccer and hockey)game, where each of two competitors writes six col-laborating processes to control six players on thesockey field. Games are displayed graphically, withthe players moving all over the field, ‘kicking’ the ballin an effort to make a goal. Configurable game boards(e.g., faster or slower grass, different field dimensions)along with the requirement for intra-team cooperation(since many configurations do not have the ability to‘kick’ the ball all the way from one end of the field tothe other) make for fascinating game play.

Project Competitions

Some competitions focus on entrants’ creations(usually constructed over weeks or months, but occa-sionally in some limited time period like a weekend) ofa product or system.

20http://www.botball.org21http://www.robocup.org/22http://en.wikipedia.org/wiki/Computer_chess22http://www.ibm.com/developerworks/java/newto/23http://www.ai.mit.edu/lab/olympics/99/events/sockey/docs/

how-to.html

For high school students, the Intel InternationalScience and Engineering Fair is one of the most famousand lucrative such competitions, though by no meansdoes it focus on computing. Huge prizes and trips moti-vate competitors in a truly international environment.

Large corporations like Microsoft,24 Google, andTopcoder run project competitions for a variety of rea-sons. ACM’s special interest group SIGMOD con-ducts an annual design competition, for 2010 they arerequesting a design25 for a distributed database queryengine Design competitions are very common andsponsored for a variety of reasons, including productpromotion.

Programming Contests

The last decade has seen an explosion of thenumber of programming competitions and recordgrowth in established competitions at the high school,university, and professional level.

• The IOI sees 300+ participants from 75 coun-tries. Almost all those countries have feedercontests to determine who represents the coun-try at the IOI.

• The university-level ACM ICPC garners thou-sands of participants from around the world inlocal and regional contests leading up to aworld championships.

• The good folks at dmoz.org26 keep track ofmany dozens of programming competitions. Itis unknown why USACO does not appear onthis list.

• The American Computer Science League27 is along-running program that features teacherpreparatory materials and (relative to most)less-challenging team-oriented competition.Contests with both written answers and pro-gramming, multiple divisions, a registrationfee, and an annual on-site All Star Competitionare ACSL’s hallmarks.

• The ICFP functional programming contest28

focuses, of course, on functional programming.In 2009, this involved calculating orbits andtasks that determined how to intercept a satel-lite.

• The Matlab programming language puzzlesolving contest29 for 2009 involved implement-ing the solution to a puzzle a bit related todrawing bit related to drawing maps.

• The IEEE extreme contest for university stu-dents30 is a 24 hour marathon with a 25 page

24http://visualbasic.about.com/b/2009/07/15/new-microsoft-programming-contest.htm,25http://db.csail.mit.edu/sigmod10contest/26http://www.dmoz.org/Computers/Programming/Contests27http://www.acsl.org28http://icfpcontest.org/29http://www.mathworks.com/contest/flooding/home.

html30http://www.ieee.org/web/membership/students/xtreme/

index.html



instruction booklet containing ten problems,including both algorithmic tasks and morepragmatic tasks like a language interpreter.

• The Java 4K Programming Contest31 asks con-testants to write a game whose object code isno longer than 4,096 bytes.

• The Mario AI competition32 asks contestants towrite an AI agent to play a Mario console-based game with up to 24 steering updates persimulated second.

• The UMD high school competition33 is a long-running algorithmic programming contest thatinvites teams from the Washington, DC metro-politan area to compete.

• The TeraGrid supercomputer student program-ming contest34 (mostly for university students)asks teams of five students to ‘‘tackle scienceand parallel computation problems.’’

• The New Jersey Institute of Technology holdsan annual regional contest for high school stu-dents.35 Teams with three members; $50 regis-tration fee; puzzle (e.g., Sudoku), simulation,and pragmatic tasks.

• The University of Alabama/Birmingham isanother example of a regional high school com-petition.36 Prizes include a Dell Laptop. Thecontest features pragmatic and numerical tasks.

• The Dyalog company promotes its productswith a programming competition37 that appearsto use APL.

• The MIT Battlecode AI tournament38 features asophisticated real-time strategy game wheresoftware robots collaborate to take over terri-tory. This year offers $40,000 in prizes to stu-dents from perhaps 100 different universities.

• The University of Illinois at Urbana Champaignhas conducted a long-running ‘‘strategy game39

of war and money’’ to accompany their annualstudent computing conference.

• Florida International University conducts anannual high school ICPC-like team competi-tion40 with two divisions, 25 teams, and a $25registration fee. Tasks are relatively simple andoccasionally touch on algorithmics.

Wikipedia41 lists a host of sites that offer onlinetraining and contest judges:

31http://javaunlimited.net/contests/java4k.php;http://www.java4k.com/index.php?action=home32http://julian.togelius.com/mariocompetition2009/33http://www.cs.umd.edu/Outreach/hsContest.shtml34http://www.teragrid.org/tg09/index.php?option=com_

content&task=view&id=8035http://cs.njit.edu/news/programmingcontest/36http://www.cis.uab.edu/programs/hspc/37http://www.dyalog.com/contest_2010/index.html38http://battlecode.mit.edu/2010/39http://www.acm.uiuc.edu/conference/2009/mechmania.

html40http://users.cis.fiu.edu/˜acm/HScomp2009/index.php41http://en.wikipedia.org/wiki/Online_judge

• ACSL

• COCI

• Codechef – India, cash prizes

• Fuzhou University Online Judge, over 800problems. Popular among the Chinese.http://acm.fzu.edu.cn/

• Google code jam, cash prizes

• Harbin Institute of Technology Online Judge,over 1800 problems. Popular among Chinesepeople. http://acm.hit.edu.cn/

• Lviv National University Online Judge, over100 hard ACM-type problems. Only Ukrainianlanguage. http://acm.lviv.ua/

• Moscow Problem Archive with Online Judge,supports C, C++, Java, Free Pascal, Perl,Python, Haskell, Kylix, Lua, OCaml, Ruby, andScheme.http://acm.mipt.ru/judge/login.pl?lang=en

• Peking University Online Judge, over 2800problems including more than 200 originalproblems, supports C/C++/Pascal/Java/Fortran.http://acm.pku.edu.cn/JudgeOnline/

• Saratov State University Problem Archive withOnline Judge, over 300 problems http://acm.sgu.ru/

• Sphere Online Judge, over 5000 problems for-mulated in English, Polish, Vietnamese, Por-tuguese; supports 39 different programminglanguages, hosting own contests supported.http://www.spoj.pl/

• Tianjin University Online Judge, over 2000problems, supports C/C++/Pascal/Java, virtualcontests supported. http://acm.tju.edu.cn/

• Timus Online Judge, The Timus Online Judgeis the largest Russian archive of problems fromdifferent programming contests.Accepts solu-tions in C/C++, Java, Pascal and C#. http://acm.timus.ru/

• Topcoder, cash prizes

• USACO|USA Computing Olympiad TrainingSite, hundreds of problems with complemen-tary tutorials for incrementally expanding com-petitive programming skills; supports C, C++,Pascal and Java programming languages; tar-geted to pre-college students. Full ACM contestsupport, as well. http://train.usaco.org

• Valladolid Problem Archive with Online Judge,about 2500 problems, supports C, C++, Pascaland Java programming languages http://uva.onlinejudge.org

• z-trening contests & training, >500 tasks at endof 2009

• Zhejiang University Online Judge, over 2000problems, supports C, C++, Pascal and Java.http://acm.zju.edu.cn/

See the Wikipedia entry42 for other comments.

42http://en.wikipedia.org/wiki/Online_judge


usaco: directions for the next decade more participants from a total of more than 65 countries on a...

Documents