USER-CENTERED LANGUAGE DESIGN, PART 2: TASKS

Michael Coblenz

YOUR TURN• Assume you have some tasks for the usability question from before.

• Who will you try to recruit?

• How will you:

• Recruit?

• Incentivize?

• Screen?

INFORMED CONSENT

• Online consent is OK (with IRB approval)

• We proposed paper-based consent for this study

DEMOGRAPHICS

• Collect information if you want it!

• Programming experience? Languages?

• If they tell you, you can use it…

• e.g. Gender__________

Pre-study questionnaire

1.How long have you been programming?______ years ______

months

2.Gender: _________

3.If you have any academic computer science background, what

degrees have you completed? If you are partway through a

degree, what degree and how far?

______________________________________

4.How much professional (paid) software development

experience do you have?______ years ______ months

5.List any programming languages in which you are currently

comfortable programming in DECREASING order of familiarity.

______________________________________________

6.How much experience do you have programming in Java?

Include time spent doing Java part-time, such as in a course.

______ years ______ months

7.Please rate your level of expertise in Java by circling one

option:

beginner intermediate advanced

8.Please rate your level of expertise in Rust by circling one

option:

none beginner intermediate advanced

9.Please rate your level of expertise in each of the following

blockchain programming languages/environments:

Solidity:

I don’t know what this is/none beginner intermediate

advanced

Hyperledger Fabric:

I don’t know what this is/none beginner intermediate

advanced

Other (specify which)__________________________________

Participant code_________

TRAINING• How will you prepare your participants?

• People don't read.

• People think they understand but in fact do not.

• Teach…and then assess.

• Or: decide that no training is necessary.

TASKS• This is the hardest part of study design.

• You will not get this right the first time.

• Solution: pilot repeatedly.

• But: you can use data from your "pilots" if you follow protocol.

• (a true "pilot" involves throwing the data out)

• What is the distribution over task times?

USABILITY STUDY TASKS

• Choose an interesting task

• One that you think might be hard

• One that is central to the usability of your design

• Can't test everything

TASK IDEAS• Write a program according to this specification.

• Are there bugs in this code? If so, what are they?

• Fill in the missing code…

• What does this code do?

• Answer these questions about this code.

PARSONS PROBLEMS

• Participant is given snippets of code, but they are out of order.

• Task: put them in the right order.

• "We find notable correlation between Parsons scores and code writing scores. We find low correlation between code writing and tracing and between Parsons and tracing." [1]

[1] Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the Fourth international Workshop on Computing Education Research (ICER ’08). Association for Computing Machinery, New York, NY, USA, 113–124. DOI:https://doi.org/10.1145/1404520.1404532

TASK DESIGN• Must carefully restrict tasks!

• People will get stuck on irrelevant things

• Decide how much help to provide

• Ideally: scope task to focus on the variable of interest

• Constrain the task as much as possible.

DECOMPOSING TASKSUsability Methods for Designing Programming Languages for So�ware Engineers 111:13

tt

(a) (b)

Fig. 2. The horizontal axis represents time; the vertical axis represents a dependent variable measured ina study. Part (a) shows how the variance increases over time. Shading shows how frequently a particularpoint in the space might be reached over many participants. In part (b), the task has been divided into threesubtasks to reduce the variance in each subtask.

3.9 SummaryTable 1 summarizes the approaches we have found e�ective when designing user studies ofprogramming languages.

4 USABILITY STUDIES FOR GLACIER4.1 Formative studiesWe used the Cognitive Dimensions of Notations framework [27] to reason about some of the designchoices. For example, including features that provided weaker guarantees than programmersactually needed could be error-prone if those features could be easily confused with stronger ones.Likewise, the inverse is error-prone too: if a programmer applied a weaker speci�cation than couldactually be applied, this could lead to undesirable tradeo�s. For example, if an interface is annotatedto return a read-only object (to an object that could be mutated through other references), theprogrammer might add locks to ensure safety in a concurrent context. But if the object is actuallyimmutable (that is, no reference could be used to mutate the object), then the locks would beunnecessary and reduce performance.Although the Cognitive Dimensions analysis was lightweight, it did not answer some of our

higher-level design questions. In order to narrow the space of possible language designs, weconducted semi-structured interviews with eight software engineers who were working on largesoftware projects at several organizations. Our participants had an average of �fteen years ofexperience, with a minimum of seven years, and had worked on projects with millions of lines ofcode and hundreds of people.

In order to both obtain unbiased data on problems with mutability in general as well as to obtainfeedback on concrete language designs, we carefully ordered the interview questions. First weasked general questions, such as “How do you make sure that state in running programs remainsvalid?” We got wide-ranging answers, including ones such as “We’ve essentially done away withmutability to avoid security and concurrency problems” as well as recommendations for regularuse of testing and assertions. Afterward, we asked about existing language features, such as const

Monolithic task Subtasks

DATA COLLECTION• Think-aloud

• Audio recordings

• Videos

• Screen capture

• Eye tracking

• Post-study survey

•Take lots of notes!, including timestamps! You do not want to watch the videos.

•Include a clock on the screen.

THINK-ALOUD

• Two varieties: concurrent and retrospective

• "Please keep talking."

• Can't use timing as a dependent variable due to effect of explanations.

TASK CONTEXTS

• Pencil/paper

• Text editor

• IDE

• Compiler?

• Debugger?

• Test cases?