euklas supporting copy-and-paste strategies for ...natprog/papers/p13-doerner-plateau... · hci...
TRANSCRIPT
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the fullcitation on the first page. Copyrights for components of this work owned by othersthan ACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from [email protected]. PLATEAU'14, October 21 2014, Portland, OR, USA Copyright 2014 ACM 978-1-4503-2277-5/14/10…$15.00 http://dx.doi.org/10.1145/2688204.2688208
EUKLAS Supporting Copy-and-Paste Strategies for
Integrating Example Code
Christian Dörner
Senacor Technologies AG Erika-Mann-Str. 55
80636 München
Andrew R. Faulring
HCI Institute Carnegie Mellon University
5000 Forbes Avenue, Pittsburgh, PA
Brad A. Myers
HCI Institute Carnegie Mellon University
5000 Forbes Avenue, Pittsburgh, PA
Abstract Researchers have paid increasing attention in recent years to the fact that much development occurs though example modification. Helping programmers with some of the pit-falls and vagaries of working with example code is the goal of our tool, called Euklas. It helps developers to integrate JavaScript example code into their own projects by using familiar IDE interaction techniques of the Eclipse IDE. The Euklas plugin uses static, heuristic source code checks to highlight potential errors and to recommend potential fixes, when incomplete sections of code are copied from a work-ing JavaScript example and pasted into the program being edited. The most unique feature of the tool is the ability to automatically import missing variable and function defini-tions from an example file into a new project file. Our preliminary user study of Euklas suggests that it supports users in fixing errors more easily.
Categories and Subject Descriptors D.2.6 [Program-ming Environments]: Programmer workbench.
General Terms Algorithms, Design, Human Factors
Keywords Code Reuse, Copy-and-Paste, JavaScript, Eclipse, Natural Programming, Examples
1. Introduction Leveraging examples is an established technique in design [1], and has recently received increasing attention from
researchers focusing on programming tools [2-5]. With the rise of search engines and web repositories of code along with discussion threads, blogs, and code example websites, people often create new systems by copying and pasting code snippets from such sources [6]. Surveys and research show that looking for examples is often people’s preferred way to learn how to perform a task or to learn how to use application programming interfaces (APIs) [7].
Most of the research on reusing examples has focused on building improved search and data mining tools to help with finding the examples (e.g., [1, 3, 5, 7, 8]). However, there has not been much research on assisting users in reus-ing and integrating the examples after they have been
Figure 1. Euklas supports peoples’ integration attempts as it enhances Eclipse’s JavaScript editor by (#1) highlighting errors in the source code, (#2) providing quick fixes including the use of the context in the origin of any copied code, and (#3) including explanations for copy-and-paste errors based on the code of the example snippet that was used.
13
fb([ag
Uwpsapur
gvinsoswswswe
thshtyucirtinatapthavoe1fFv
1 n
found, with thebeen identified (e.g., copying s11-13]. Our a
attempts by prguidance to hel
This paper inUsers’ Keystrowhich helps uspaste strategiesspecific guidanand-paste errorpart of the requusers with reuresulting errors
JavaScript iguages for webvariety of usernteraction des
spite of this wioften provide lstatic code chewith other, nonson may be thaweak, dynamicstatic analyses well as provideerrors resulting
The followihe kind of cop
supported by Ehas found a coypes of enhan
use one in her code in more drrelevant pieceions, variables
necessary to mample. This isask especially
provide any hehe integration
ance. First, Euvariables, missiof CSS and Javerror markers 1). Second, Eukfied errors to Figure 1). For variable and fu
Euklas is Germa
nounced oy-class.
e exception of as a common
several lines, baim is to suproviding famililp them make tntroduces our
okes Lessened sers to more sus for reuse (sence for assistinrs that are typicuired code pat
using working s. is one of the b programmingrs, from end-uigners [6], to idespread use, less programmecking, limitedn-scripting langat analyzing Javc typing that mat edit-time. W
e more sophistg from copy-aning example ppy-and-paste reEuklas. Jamie ode snippet onnced combo-bo
website. Jamietail to separates. That is, sh, and imports o
make her code ws often a timey since Jamie’elp. In contrastprocess as it p
uklas highlighing function devaScript files,
and squigglyklas also comphelp Jamie eaexample, it s
unction definit
an for Euclase, w
[4, 9, 10]. Copusage pattern locks, or even
pport people’s iar, easy-to-ustheir copied cotool, called EUby Attaching uccessfully emee Figure 1). Eng users in fixcally caused byttern [11]. Thecode from ot
most populag [15] and is uuser developerprofessional pJavaScript de
ming support fod auto-completiguages such asvaScript is diff
makes it challenWe overcome tticated guidancd-paste. resents a typiceuse of JavaScis a JavaScrip
n the web that oxes from whiie has to explote relevant piec
he has to identof JavaScript awork the same-consuming an’s JavaScript t, Euklas can h
provides her whts errors, sucefinitions, and in her target coy underlines (s
putes quick fixeasily correct thsuggests copyitions from the
which is a gemsto
py-and-paste hafor reusing codwhole methodcopy-and-pas
e, and effectivde work.
UKLAS (Eclipfrom Samples)
mploy copy-andEuklas provide
xing some copyy copying onlye goal is helpinthers and to f
ar scripting lanused by a broars [16] such a
programmers. Ivelopment too
or users (e.g., nions), compares Java. One reficult since it hanging to perforthis limitation ace for correctin
cal use case focript code that pt developer an
creates variouch she wants ore the exampces of code frotify which funand CSS files ae way as the exnd cumbersomeditor does nhelp Jamie wiith editing guid
ch as undefinemissing imporode by insertinsee #1 in Figues for the identhem (see #2 ing the missinsource file an
one. Euklas is pr
as de
ds) ste ve
se )1, d-es y-
y a ng fix
n-ad as In
ols no ed a-as
rm as ng
for is
nd us to
ple m
nc-are x-
me not th d-ed rts ng re ti-in ng nd
ro-
offers and Ja
Eukusers tprojectpastedthe conwere cfeaturetion anheuristas chetions, awas coerrors.
Wepeopleresultsusableconfusful. Thwere ain the their ta
2. RThe relocatintarget findingthe weare toosystem
JDAusers tfrom Jwrite pieces UNIX Script not accEuklasusers ttheir c
D.MHTMLbles thexistinit follorepresements’D.MIXone st
to insert the mavaScript files.klas makes theto integrate Jats by identifyi
d code and recnsideration of
copied. The edes of the Eclipnd repair algortic edit-time so
ecking for uninand the analysiopied) to provi. e evaluated Euke, comparing s suggest that the and effectivesed that Euklashe study also sable to fix abo
control grouparget systems.
Related Weuse of examplng the examplsystem. Thereg relevant exaeb) [2, 3, 17, 18ols that suppo
ms. A (JavaScript that have no prJavaScript codesimple HTML of JavaScript systems. Howcode as “blac
cess or changes treats all Javato access and c
code structure.MIX targets wL and scriptinghem to build ng web sites. Dows a copy-andentation of the’ parameters cX’s editing envtep further, si
missing import
e following maavaScript examing some of thcommending pf the original cditing guidancepse IDE and exrithms. These ource code che
nitialized variabis of the exampide useful quic
klas in a prelimit to Eclipse’she features have manner, sincs can sometimsuggests that p
out two times ap, when integr
Work le code consistle code and 2
e are several syample code (e.8]. However, mort users in int
Dataflow Archrogramming ske that was founL commands t code in a wwever, JDA trk-boxes”, whi
e the internal coaScript code ascopy even inco
web designersg languages, suand share ma
D.MIX inspiredd-paste approae selected datacan be changevironment. Weince Euklas p
t statements fo
ajor contributiomple code intohe potential erpotential fixesode from whic
e uses familiar xtends it with nalgorithms are
ecks for JavaSbles and undefple file (from wck fixes for the
minary user stus JavaScript eve been implemce participants
mes be wrong oparticipants usas many errorsrating example
ts of two main2) integrating ystems that ass.g., in reposito
more relevant ttegrating code
hitecture) aimskills to build apnd on the web to connect th
way similar to reats the piecech means that ode structure. Is “glass boxes”omplete interna
s that are famuch as JavaScrashups created d the design ofach that works a [19]. In thated after pastine take the idea provides speci
or the CSS
on: It helps o their own rrors in the , based on ch sections interaction new detec-e based on
Script, such fined func-where code e identified
udy with 12 editor. The mented in a s were not or not help-ing Euklas s as people e code into
n phases: 1) it into the ist users in ories or on to our work e into their
s to enable pplications [10]. Users e different “pipes” in
es of Java-users can-
In contrast, ”, allowing al pieces of
miliar with ript. It ena-
from pre-f Euklas as on a richer t way, ele-ng them to
of D.MIX fic editing
14
oin
rthdoainupw
c[dineoetasdc
cgmdI
mlsOgaafSinaqck
Rurpfineth
operations, in tntegrate their c
The Lookingreuse functionahem to unders
dents to do thisof the programand end of thentegrate this f
uses the idea opaste-based reuwith familiar in
JIGSAW is copy-and-paste9]. It inspired
developers withnto the develop
example code aof the example ever, Jigsaw carget code tha
structures. Jigsdevelopers, whcumbersome an
JSLINT is acript code. JSLgoal is to helpmakes it differdo not restrict In addition, Euk
Many JavaSming support oevel, IDEs like
such as checkinOne of the mogramming Javaadvanced codeample, it offerfunctions haveStorm also offeng users to eas
a new empty quick fixes do copied, makingkind of quick fi
Euklas’s copRosson’s and uses in Smalltreuse strategy piece of code tfunctionality thnto the target
editing directiohe code work
the form of qucopied pieces og Glass IDE hality they find stand how the s by enabling th
m they are intee functionality functionality inof guiding useuse strategies,nteraction techn
a plug-in fore interaction ted the design ofh the integratiper’s own sourand the target code would fian only work
at have similarsaw was testedhich showed thand not yet wella popular tool Lint works on p programmerrent from Euklwhich featuresklas supports u
Script editors dother than syne NetBeans prong whether a vost advanced caScript is Webe inspections brs analyses to been defineders quick fixes sily declare andfunction definnot analyze th
g it impossiblixes that Euklapy-and-paste dCarroll’s obsetalk programmwhere progra
that they consihat they intend
context, theyons about whatk. This was o
uick-fixes, whiof code into theelps middle scin other progrcode works [4hem to 1) reco
erested in, 2) ithey are intern their new p
ers to support but integrateniques in Eclipr the Eclipse Ichnique for ref Euklas, sincion of the reusrce code. Jigsacode to sugge
it best in the tak with pieces or AST (Abstrad in a small at the resolutio supported by for detecting a subset of Jars to write belas’s error detes of the languausers in fixing tdo not provide ntax highlightiovide simple cvariable has becommercial prbStorm, whichby leveraging
check whethed. In contrast t
for the detected define a varianition. Howevehe files from we for WebSto
as provides. design is stronervations abouming” [13]. Thammers copiedidered to be prded to reuse.
y let the envirt would be nec
often accompli
ch help users eir projects. chool students rams by helpin4]. It guides sturd the executiodentify the sta
rested in, and 3program. Eukla
their copy-andes this guidancpse. IDE that uses
eusing Java code Jigsaw assissed source codaw compares thest which piecrget code. Howof example an
act Syntax Trestudy with tw
on of conflicts Jigsaw. errors in JavaS
avaScript and ietter code. Thection, since wage can be usethe errors. much programng. At the neode inspectioneen used or noroducts for proh provides moJSLint. For ex
er variables anto JSLint, Webed errors, allowable, or to creaer, the providewhere code wa
orm to offer th
ngly inspired but “the reuse ohey observed d and pasted romising for thAfter pasting onment providcessary to makished in sever
to
to ng u-on art 3) as d-ce
a de sts de he es w-nd e)
wo is
S-its his we ed.
m-xt
ns, ot. o-re x-nd b-w-ate ed as he
by of a a
he it
de ke ral
cycles until a
Kimenced ly coptionalievidenoften spend
Eukcopyinwill nofor sompastingare pehave b
3. EIn thissign. Aplemen
3.1 U
Returnexplaincode s
structocode fpastes two ercalls fThese not avcopiediar maerror min the
FigurJamie
of fixing andall of the copiedm et al. reportfields/constant
pied together bity [12]. In adnce that the idfails in the firtime on identif
klas supports ung-and-pastingot only mark eme of these errg additional pi
erformed iteratibeen fixed.
EUKLASs section we fiAfterwards wented and integr
User Interface
ning to the exan how Euklas hshe found on t
or (jawBar(idfor making ththe constructo
rrors in the cofindMatch(eare undefined
vailable in the td from the exaarker system inmarkers in tcode.
re 2. Euklas me pastes it into th
d waiting for nd code was fixet that related cts and caller/ca
because they bddition, Ko etdentification orst attempt, refying all relevausers when th
g it into their terrors, but it arors (which miieces of the exaively until all
first discuss Eue will present hrated into the E
e
ample discussehelps Jamie tothe web. Jamie
d), see Figure 2he combo-boxor into her targonstructor: thee) and init(d because theirtarget file, sinc
ample file. Eukncluding the squthe margin to
marks errors in he target file.
new editing sued. code snippets allee methods)
belong to the st al. providedof related codequiring progrant pieces of cohey try to reustarget code sinalso provides qight involve coample code). Terrors in the t
uklas’s user inhow the systemEclipse IDE.
ed in the introd integrate the ce identifies tha
2) is a promisinx work. She cget file. Euklas two undefine) (see #1 inr function defice they have noklas uses Eclipuiggle underlinindicate these
the function ja
uggestions,
(e.g. refer-) are usual-same func-d empirical de snippets rammers to ode [11]. se code by nce Euklas quick fixes opying and These steps target code
nterface de-m was im-
duction, we combo-box at the con-
ng piece of copies and s identifies ed function
Figure 2). initions are ot yet been
pse’s famil-nes and the two errors
awBar() after
15
s
eff3f
fthfthfbwbbfwato
FeoltTtisthu
3
EeH
Ffpsffw
In addition solutions for fi
that refers toerror, as shownfunctionality fofixes proposed 3) use Euklas’sfixes proposed
The first qufunction jawBahe example fi
function jawBahat Euklas pro
function, calledbody. This is swould not servbo-box work inbecause it resefered by Eclipwhich programalso a fallback o provide a bet
Jamie readsFigure 3) that error by using option is probaects it. Eukltype.init() This successfulion init().
still undefined he newly code
until the code is
3.2 Impleme
Euklas was imeditor as a pluHTML editors
Figure 3. Euklafined function inproposed fix is shshows a short pfollowed by a prefix has been appwhich is well-kn
to marking thixing them. Byo init(), Jamn in Figure 3. or showing theby Euklas (the
s icon to diby Eclipse itseick fix that Euar.prototyp
file (the file frar()) to the taoposes is a “dd init() that sufficient to ree Jamie’s goal n her website. embles the kinpse’s Java ed
mmers may be option for castter quick fix b
s the explanatidescribes howthe example c
ably the correclas pastes the
from the exally fixes the mHowever, the and Euklas id
e. Jamie contins fixed and the
ntation
mplemented onug-in to the IDdo not provid
as proposes the tnit(). More inforhown in the beigproposal explaineview of how thlied. Euklas augown by many pr
e errors, Euklay clicking on t
mie gets four opAgain, Eukla
e quick fix opte first two in tstinguish themelf (the last twouklas proposes pe.init() thfrom which Jaarget file. The sdefault fix” thawould have anemove the synof making theWe added thi
nds of quick fiitor and by Wfamiliar. The des in which Eu
based on the exion in the beigw Euklas propcode. She decidct fix for her se function j
ample file intomissing definiti
function findentified an adues to fix the r
e combo-box is
n top of EclipDE. Eclipse’s
de as much pro
top two quick firmation about thge pop-up at the ning what Euklhe code will lookgments Eclipse’srogrammers.
as also suggesthe error markptions to fix ths uses Eclipsetions. The quicthe list in Figu
m from the quico in the list). would copy that is defined
aime copied thsecond quick fat creates a nen empty functiontax error, but e enhanced comis default optioixes that are oWebStorm widefault option uklas is not ab
xample code. ge window (seposes to fix thdes that the firsituation and sawBar.proto
o the target filion of the funndMatch(e) dditional error remaining erros working.
pse’s JavaScriJavaScript an
ogramming sup
fixes for the undhe first (selecteright. The pop-uas intends to d
k after the selectes quick fix featu
sts ker his e’s ck re ck
he in he fix ew on it
m-on of-th is
ble
ee he rst e-o-le.
nc-is in
ors
pt nd p-
port. Isyntaxin file”
Javto reliavariounon-fuan und‘null’ for adoperatithat artionshiin the can bein a fuvariablmissindefinitwhich HTMLfunctio
EukSyntaxcode. pastes file. Etypes providbe the runtimerrors cause do not
Eukused in[21]. SvariablEuklasen to [22]. Tof thecorrecters a utype). errors undefidetectinumbeerrors
Theare monumbedynam
de-ed)up
do,ed
ure
In particular, thx errors and so” and “assign s
vaScript is a dyably detect err
us situations thaunction value (defined variabor ‘undefined’
dditional errorstions. We idenre specific to Jaip with HTMLfollowing list
e detected by Eunction’s paramle definitions,
ng CSS style tions), 5) missimight be loca
L elements beon getElemenklas employs hx Tree (AST) Euklas’s analya piece of co
Euklas providementioned ab
des quick fixes more importan
me errors in thein the HTMLproblems with
t necessarily leklas’s code anan other static Since JavaScrles, it is imposs instead emplobe a successf
The implemente used exampt, which mean
useful result (aEuklas uses thlisted above. T
ined variables ion algorithm er 3), and the tin the HTML c
e heuristics useore fragile thanered 1-3. The
mically, e.g. loa
hey only proviome basic quicstatement to neynamic languarors at edit timat can cause ru(e.g. undefinedble, and acces’ [20]. Startings that may arintified three adavaScript, and L and CSS codof potential co
Euklas: 1) missmeter list, 2) m, 3) missing sheet imports ing JavaScript ated on remoteing accessed bntById(HTML_heuristic, static
to find potenysis of the A
ode from an exes error highlibove. However
for numbers 1nt types of erroe JavaScript c
L part of the sh displaying thad to JavaScripalyses are diffcode analysis
ript does not ssible to run coys heuristic aful alternative tation of Eukla
ples is syntactns that the codea limitation of three different aThe first detect
(error numbechecks for un
third detection code (error numed for identify
n the heuristicsreason is tha
ading CSS cod
ide error highlk fixes, such a
ew local variabage, which mame. Jensen et auntime errors: d) as a functiossing a prope
g with this list, ise from copy dditional types
d arise from its de. Our analysopy and paste ing parameter missing local function defin(e.g. for genefile imports (e
e servers), and by the global _Element_ID analyses on th
ntial errors in ST is invokedxample file toighting for allr, Euklas curr1-3, which we cors, since they code. Numberssource code, whe webpage copt execution-timferent from tho
tools, such asprovide static
certain dataflowanalyses, which
approach for as assumes thatically and see is executablethe current Eualgorithms to ition algorithm
ers 1 and 2), tndefined functalgorithm che
mbers 4-6). ying errors nums used for detecat content can de within JavaS
lighting for as “rename
ble.” kes it hard al. identify invoking a
on, reading erty that is
we looked y and paste s of errors close rela-
sis resulted errors that definitions and global nitions, 4) eral layout e.g. scripts, 6) missing JavaScript
D). he Abstract JavaScript
d if a user o the target l six error rently only consider to may cause
s 4-6 cause which may orrectly but me errors.
ose that are s FindBugs
typing of w analyses. h has prov-JavaScript
at the code emantically and deliv-
uklas proto-identify the checks for
the second tions (error ecks for the
mbered 4-6 cting errors
be loaded Script code.
16
Trm
pmeCecttoctfto
athscrrEmeths
estiusus
seEvfsqfmfc
4TCtu
To visualize ouristics, error numarker inste
There are spositives or falmeans that the error, or miss Consider the ferrors in the evcaused by “hthat.findMato detect that
could be relatethat. It wouldfind the functioo the object th
Euklas not oalso can propohe links betwe
specific region code. Each targregion has exaregion in an exEuklas maintaimemory. It upedited to keep this metadata a
shuts down. Currently, E
example files, ispace. We deciion for the Euk
usability and ushould handle eual snippets ofsmall pieces of
The ASTs osolutions for serror in the taEuklas analyzevariable (underfile(s) is syntacsuitable definitquick fix propofile into the tarmarker in the tafix, Euklas parcopied AST pie
4. EvaluaThe evaluationCan users undures? Is the in
ur confidence umbers 4, 5, anad of an error mome cases in se negatives, danalyses can emarking some
following examvaluation. Euklahiding” variabtch(e);. In t
the function d to the object
d produce a falon definition ofhat . only finds potese fixes for theen the positioin the target f
get file can haactly one link xample file fromins the meta-dapdates the regithe metadata c
as part of the p
Euklas needs ti.e., they mustided that this wklas prototype,usefulness of example code f code (e.g., cf code in blog pf the example
some classes oarget file refees the exampler the assumptioctically and semtion in an ex
osal to copy tharget file, and aarget file. If therses the AST oece from the ex
ation n had the goalderstand and sntegration of e
level of the md 6 are markedmarker . which Euklas
due to the usedeither mark coe existing erromple that cauas cannot detec
bles: var ththis case Eukla
call that.ft this insteadse positive becf findMatch(
ential errors inhese errors. Euon in the examfile when usersave multiple reto exactly onem which the cata about theseions when the correct. Euklas project when E
to have accessbe present in
would be a re, to enable invits ideas, butfrom external
code copied frposts). files are analyz
of errors. For ers to an und file(s) for a d
on that the codemantically corrxample file, Eat definition fr
adds this quicke user selects aof the target fxample file.
l of answeringuccessfully us
example code
more fragile heud with a warnin
s produces fald heuristics. Thorrect code as aors in the codused one of thct errors that ahat = thisas is not be abindMatch(e) of to the objecause it does n(e) that belong
n the code, but uklas remembemple file and ths copy and pasegions and eace correspondinode was copiee connections
target files aloads and savclipse starts an
s to each of thEclipse’s workasonable limitestigation of tht a real systefiles or individom the web, o
zed for potentiexample, if thefined variabl
definition of the in the exampect). If there is
Euklas creates rom the exampk fix to the erroa proposed quicfile to insert th
g two questionse Euklas’s fefaster and mo
u-ng
se his an
de. he
are s; ble ;
ect not gs
it ers he ste ch ng ed. in
are es nd
he k-ta-he m d-or
ial he le, his ple s a
a ple or ck he
ns: a-re
correcteditor tor, wh
4.1 P
We fehave snot waTherefyear olanguaprojectWe relocal usated $to 37 verse softwa
4.2 A
The stgroundstallatiThe codoes nexperim“Euklatween the tarwhile chose able towanteding andJava ehavingtook th
Weused foin Firetest wh
4.3 P
The stversionEclipslite” aner the subjecimposswould run. Psince a
t with Euklas tor compared t
hich already of
Participants
eel that Euklassome experienant to have tofore, we recruof experience age), and whot in JavaScriptcruited 12 paruniversity com$15 for participyears (medianbackgrounds,
are engineering
Apparatus and
tudy was conds. We used aion including tontrol group unot highlight emental group as lite” and “Ethe two versio
rget file to iden“Euklas full” to have these
o separately stud to explore thd quick fixes w
editor. Secondg the more sophe example filee set up Eclipsfor the study. Eefox to allow phether their tar
Procedure
tudy used a bn as indepene’s JavaScriptnd “Euklas ful tasks were c
cts design wassible for the pa
d have known Participants weall subjects rep
than with Eclipto a more sophffers some erro
s is most appnce with usingo train people uited participausing Eclipse
o had done at t (using any devrticipants (10 m
mmunity. Each pating. Their an: 25, s.d. 5).
such as busig.
d Materials
nducted in ouan iMac runninthe WTP (Web
used Eclipse’s errors or provid
used one of Euklas full”. T
ons was that “Entify errors andalso analyzed two different
udy the followihe effects of pwith a user inte, we were int
phisticated anale(s) into accouse’s workspaceEach of the exaparticipants to erget code was w
between-subjecndent variable t editor (Conll”. The dependcompleted sucs chosen becaarticipants to rthe answer to
ere randomly ported approx
pse’s standard histicated Javaor detection fea
propriate for pg JavaScript, a
on how to usants who had (for developileast one pro
velopment envmale, 2 femaleparticipant wages ranged froThe participan
iness administ
ur lab on the ng a standard Eb Tools ProjecJavaScript edide quick fixestwo versions The main diff
Euklas lite” onld to compute q
the example versions of Euing two aspect
providing errorerface similar toterested in thelyses and quick
unt. e with the filesamples could bexplore the exaworking. .
cts design usinwith three c
ntrol conditiondent variable w
ccessfully. Theause it would redo the tasks, each task aftassigned to eaimately equal
JavaScript aScript edi-atures?
eople who and we did se Eclipse. about one
ing in any ogramming vironment). e) from our as compen-m 19 years nts had di-tration and
university Eclipse in-ct) plug-in. itor, which s, and each of Euklas:
ference be-ly analyzed quick fixes, file(s). We uklas to be s. First, we r highlight-o Eclipse’s effects of k fixes that
s that were be executed amples and
ng the tool conditions:
n), “Euklas was wheth-e between-have been since they
er the first ach group, JavaScript
17
programming experience. Participants in all groups re-ceived a spoken introduction to the study and signed the consent form. All participants were briefly introduced to Eclipse’s JavaScript editor. Members of the experimental groups received an additional introduction to Euklas’s extensions to Eclipse’s JavaScript editor, e.g. information about its error and warning markers and the quick fixes. All task descriptions explained which code should be copied and what the desired results would be after the code was pasted and all errors were fixed.
All participants performed the tasks in the same order. The tasks were designed to cover all cases in which Euklas provides support, as well as cases where it provides mis-leading help or does not help at all. Participants were al-lowed to work on each task for a fixed amount of time, which varied from 7 to 20 minutes, based on the task’s difficulty. The researcher who conducted the study meas-ured the time it took participants to work on each of the tasks and stopped them if they ran over the maximum time allowed for each task. Task No. of
Errors Types of Errors*
Source LOC
Copied LOC
Max. Time (min.)
1 1 2 112 10 7 2 3 2, 3 112 13 7 3 2 1, 2 122 29 7 4 3 2, 5, 6 103 5 10 5 5 3, syntax
error 218 125 12
6 10 2, 3, 4 1507 996 20 Sum 24 2174 1178 63
Table 1. Summary of the tasks (*types of errors ac-cording to the list in section 3.2
Table 1 shows the number of errors per task (errors that occurred after the first paste operation), the types of errors included in the tasks (based on the types of errors presented in section 3.2), the lines of code (LOC) of the source ex-ample files (HTML and JavaScript files together), the total number of lines that had to be copied to solve the tasks, and the maximum time participants were given to complete each of the tasks. We judged tasks to be finished success-fully if the code could be executed without causing any errors in Firefox and if it performed as required by the specification.
In task #1, participants had to choose and integrate the correct part of a larger function for setting a cookie, while they had to choose and integrate a different part of that function for getting a cookie in task #2. For task #3, partic-ipants had to integrate an enhanced pop-up menu into the target file. In task #4, participants had to add a pushpin to a ‘Bing’ map and integrate it into the target file. To complete task #5, participants had to integrate an enhanced combo box (the one that we described above as Jamie’s example) into their target file. Finally, in task #6, participants had to integrate an inner window into their target file.
At the end of the study, participants filled out a ques-tionnaire. The questions primarily used five-level Likert scales, but some were open answer.
4.4 Results
The analysis of the data shows that participants using “Euklas full” completed more tasks (Control: 7/24, “Euklas lite”: 13/24, “Euklas full”: 18/24.) and fixed about twice as many errors (average = 22.25) as the control group did (average = 11.75) when integrating the example code into the target system.
For the analysis we combined the tasks shown in Table 1 into an “easy” group (tasks 1-3) and a “difficult” group (tasks 4-6). There are several reasons for combining the tasks into the two groups. The median average success rate for all six tasks across all three conditions was 0.5. We defined “easy” tasks as those with an average success rate at or above the median, and “difficult” tasks as those below the median. Another reason was that the tasks had been designed with an increasing difficulty. We allotted more time for completing the more difficult tasks. Finally, with-out combining the tasks we would have not been able to perform any statistical analysis, due to the highly differen-tiated completion values. The same reasoning justifies the statistical tests on the number of errors fixed. “Easy” Tasks “Difficult” Tasks 1: 0.75 4: 0.33 2: 0.50 5: 0.17 3: 1.00 6: 0.42
Table 2. Average success rate per task (std. error: 0.13) We ran a logistic regression analysis of the tasks, which
showed a statistically significant difference for the success rate with respect to the tasks’ difficulty, i.e. between the “easy” and “difficult” tasks: likelihood ratio χ²[1] = 14.79, p < 0.0001. We also analyzed how the success rate of par-ticipants was affected by the tool used (Control, Euklas lite, Euklas full) and the task difficulty (easy tasks, difficult tasks). A nominal logistic regression analysis showed that the effect of the tool on the success rate was significant: likelihood ratio χ²[2] = 14.97, p = 0.0006. The effect of the task difficulty on the success rate was also significant: likelihood ratio χ²[1] = 19.34, p < 0.0001. The interaction was not significant. In other words, participants were more successful based on the tool used across all tasks. We per-formed pairwise nominal logistic regression tests to deter-mine which levels of tool use had a significant effect on the success rate: Control vs. “Euklas lite”: likelihood ratio χ²[1] = 3.21, p = 0.074; Control vs. “Euklas full”: likeli-hood ratio χ²[1] = 14.90, p < 0.0001; “Euklas lite” vs. “Euklas full”: likelihood ratio χ²[1] = 4.44, p < 0.035. In summary, “Euklas full” was better than both Control and “Euklas lite”. “Euklas lite” was slightly better than Control.
Looking at the number of corrected errors, i.e. the num-ber of errors that were fixed by the participants, the data
18
also shows that “Euklas full” participants fixed almost twice as many errors as participants in the control group We ran an ANOVA to test for the effect of the tool used and the task difficulty on the number of corrected errors, and found that the number of corrected errors differed sig-nificantly across the three tools, F[2,9] = 13.82, p < 0.002. We ran contrasts to compare the tools with each other. The Control group made fewer error corrections than the “Euklas lite” group (F[1,9] = 13.37, p = 0.005) and also corrected less errors than the “Euklas full” group (F[1,9] = 26, p < 0.001).
Unfortunately, we were not able to analyze differences for the timing data, i.e. how long participants took to com-plete each of the tasks, since there were so many tasks that participants failed to complete in the maximum allotted time. The analysis of the final questionnaires provides more details on the differences between the two Euklas versions. Participants who used “Euklas full” agreed that it usually provided helpful quick fixes (average 4 out of 5). One “Euklas full” user nicely expressed why he liked it: “Intel-ligent error messages and debugging makes it infinitely more useful, especially when it checks against the source of your copy.” “Euklas lite” got lower ratings in terms of helpfulness (average 3.25 out of 5) of its quick fixes from the participants who used it, which was not surprising. One of the “Euklas lite” participants suggested the following improvement, which reflects exactly the improvements in “Euklas full”: “Provide [a] pop-up menu which can suggest to copy blocks of code to resolve errors.” The questionnaire also asked whether Euklas speeds up the integration of JavaScript code compared to other editors. Participants using “Euklas full” strongly agreed with this statement (4.75 out of 5) while participants using “Euklas lite” did not share this view (3.75 out of 5).
5. Discussion The tasks that were combined in the “easy tasks” group contained a maximum of three errors and participants were allotted the same maximum time of seven minutes for each of the tasks. Even though the number of errors was low and the scripts were short, participants in the Euklas conditions were more likely to complete the three tasks and fixed more errors than participants in the control group. However, due to the rather low complexity of tasks in this group, we did not expect that “Euklas full” participants would have big advantages compared to “Euklas lite” participants, which was reflected by the very similar results in the two Euklas conditions.
For the three tasks in the “difficult tasks” group, howev-er, the situation is different. “Euklas full” users had a larger advantage than participants in the other two groups. The code they used was longer and more complex (1828 LOC instead of 346 LOC), contained more errors (18 instead of
6) and the errors were more difficult to find (see Table 1). Therefore, we increased the maximum time that partici-pants were allowed to work on each task. The task comple-tion rates were higher for participants using “Euklas full” than participants using “Euklas lite” and participants in the control condition. Also, the results show that participants using “Euklas full” were able to fix more errors than partic-ipants in any of the other two conditions.
Overall, the results show that the “Euklas lite” version, which provided error detection features and standard quick fixes, already brought many improvements in comparison with Eclipse’s JavaScript editor, which did not offer such features. This is not surprising, since we know that high-lighting errors helps users. “Euklas full”, which offered additional analyses and additional quick fixes, improved performance even more. “Euklas full” especially showed advantages when the pieces of copied code were longer and/or more complex (as in Tasks 4-6). Euklas’s approach of considering a broader context for the computation of potential quick fixes has implications for many other pro-gramming languages, such as Java.
Using Eclipse’s marker feature for highlighting and fix-ing errors seems to be an appropriate UI choice as partici-pants had a very positive attitude towards this approach and considered it to be easy to learn. The most important aspect about using this feature is that participants knew what to expect from the provided quick fixes. They knew that these were usually right in “Euklas full”, but that they could sometimes also be wrong and might not be helpful. Such a situation was simulated in task 5, where participants had to fix a syntax error. The task also included a missing function definition that was not detected by either of the two Euklas versions. Participants were generally able to distinguish between these cases where they got standard “quick fixes” from Eclipse and where they got more sophisticated quick fixes from Euklas, since they did some manual checks to see if a proposed quick fix was appropriate or not before they used it.
There were a total of 24 errors that occurred after partic-ipants pasted the code from the source files into the target files. “Euklas full” provided 20 helpful suggestions for fixing these errors, plus 33 generic suggestions (e.g. declare a function with an empty body). Eclipse generally added in two additional suggestions per error that were not helpful at all. Participants sometimes picked one of Euklas’s generic fixes (e.g., for one of the missing functions in task 5). They knew that this was not sufficient for making the code work and therefore looked at the example code to manually find the correct piece of code. In cases where Euklas did not show an error at all (e.g., for one of the missing functions in task 5), participants did not perform worse than the par-ticipants in the control group, so there appears to be no disadvantage to using Euklas. However, the evaluation has some limitations. First, the evaluation only had the rather
19
small number of 4 participants in each condition. Second, participants used the tools for only one hour. Third, the tasks may not have been representative of realistic tasks. Although the pieces of code were real-world examples that were taken from the web, we had to reduce the complexity of dealing with these examples to limit the amount of time spent on completing each of the tasks.
6. Conclusion and Future Work In this paper, we presented our new Eclipse plug-in, Euklas, which supports JavaScript programmers in some of the tasks when using copy-and-paste-strategies for reusing example code. This kind of reuse does not create clones in a codebase [14], but instead helps users to reuse working code from others to introduce new functionality to their codebases and understand how this functionality can be implemented. Euklas supports these strategies by analyzing the target code for errors and by suggesting fixes for these errors. An important innovation is augmenting the fixes through an analysis of the code from where the example was copied. Our evaluation shows that Euklas’s users were able to fix a much higher number of copy-and-paste related errors than participants who used Eclipse’s JavaScript edi-tor, which does not provide any debugging support.
Euklas’s main contribution is analyzing the file from where code was copied to provide more detailed error de-scriptions and much better quick fixes for these errors. We think that applying Euklas’s ideas to editors used for other languages (e.g., Python, Java, and C++) could increase programmers’ performance in these situations in the same way it did in our case for JavaScript.
In addition to reducing the limitations discussed above, future work could include implementing some additional features for Euklas. One idea for providing improved error detections would be to design a heuristic for analyzing the context of a variable to try to determine its runtime type. This would allow the implementation of better quick fixes, since the system could distinguish between variables with the same name, but of a different type.
Euklas points to a future where programming support tools better help developers by taking into account all of the available contextual information, and the provenance of resources used. The success of Euklas shows that this ap-proach is feasible and can be successful, and developers can make effective use of recommendations, even when they are heuristic. The incorporation of extended copy and paste support into different kinds of editors would be a first step into this future.
Acknowledgements This work was conducted in 2010/2011, when the first author was a postdoc at Carnegie Mellon University. The
authors would like to thank Sara Kiesler for her invaluable assistance in the data analysis. The first author thanks the Alexander von Humboldt-Foundation for his Feodor Lynen Research Fellowship for Postdoctoral Researchers. This research has also been supported by SAP, Adobe and the National Science Foundation, under grant CCF-0811610. Any opinions, findings and conclusions or recommenda-tions expressed in this material are those of the author(s) and do not necessarily reflect those of the sponsors.
References [1] Lee, B. et al., Designing with interactive example galleries. in
CHI '10, 2257-2266. [2] Bajracharya, S. et al., C., Sourcerer: An internet-scale software
repository. in ICSE SUITE Workshop '09, IEEE, 1-4. [3] Brandt, J. et al., S.R., Example-centric programming: integrating web
search into the development environment. in CHI '10, 513-522. [4] Gross, P.A. et al., A code reuse interface for non-programmer middle
school students. in IUI '10, 219-228. [5] Hartmann, B. et al., What would other programmers do: suggesting
solutions to error messages. in CHI '10, 1019-1028. [6] Myers, B. et al., How Designers Design and Program Interactive
Behaviors. in VL/HCC'08, 177-184. [7] Brandt, J. et al., Two studies of opportunistic programming. in CHI
'09, 1589-1598. [8] Stylos, J. and Myers, B.A., Mica: A Programming Web-Search Aid.
in VL/HCC '06, 195-202. [9] Cottrell, R. et al., Semi-automating small-scale source code reuse via
structural correspondence. in FSE-16, 214-225. [10] Lim, S.C.S. and Lucas, P., JDA: a step towards large-scale reuse on
the web. in OOPSLA '06, 586-601. [11] Ko, A. J. et al., Eliciting Design Requirements for Maintenance-
Oriented IDEs. In ICSE '05, 126-135. [12] Kim, M. et al., An Ethnographic Study of Copy and Paste Program-
ming Practices in OOPL. in ISESE ‘04, 83-92. [13] Rosson, M.B. and Carroll, J.M. The reuse of uses in Smalltalk pro-
gramming. ACM TOCHI, 3 (3). 219-253. [14] Rahman, F., Bird, C. and Devanbu, P., Clones: What is that smell?.
In MSR ’10, 72-81. [15] Crockford, D., JavaScript: The Good Parts, Sebastopol, CA: O'Reilly
& Associates, 2008. [16] Lieberman, H., Paternò, F. and Wulf, V. End User Development.
Springer, Dordrecht, 2006. [17] Sahavechaphan, N. and Claypool, K., XSnippet: Mining For sample
code. in OOPSLA '06, 413-430. [18] Holmes, R., Walker, R.J. and Murphy, G.C. Approximate Structural
Context Matching: An Approach to Recommend Relevant Examples. IEEE TSE, 32 (12). 952-970.
[19] Hartmann, B. et al., Programming by a sample: rapidly creating web applications with d.mix. in UIST '07, 241-250.
[20] Jensen, S.H., Møller, A. and Thiemann, P., Type Analysis for JavaS-cript. in 16th International Symposium SAS, Springer, 238-255.
[21] Ayewah, N. et al., Using Static Analysis to Find Bugs. IEEE Soft-ware, 25 (5). 22-29.
[22] Ko, A. and Wobbrock, J., Cleanroom: Edit-Time Error Detection with the Uniqueness Heuristic. in VL/HCC '10, 7-14.
20