human-centered methods for improving api usabilitynatprog/papers/myers wapi 2017... ·...

Human-Centered Methods for Improving API Usability

Brad A. MyersHuman-Computer Interaction InstituteSchool of Computer ScienceCarnegie Mellon Universityhttp://www.cs.cmu.edu/[email protected]

1© 2017 – Brad A. Myers

http://www.cs.cmu.edu/%7Ebammailto:[email protected]

APIs

• Application Programming Interfaces• Includes: libraries, toolkits, frameworks,

software development kits (SDKs), etc.• Today: web services, “middleware”• Also: internal APIs for large software systems• Provides some functionality for reuse by other

developers© 2017 – Brad A. Myers

2

Stakeholders &Their Goals• [Myers, Stylos, CACM , 2016]

© 2017 – Brad A. Myers

3

Why APIs?• Some design goals for APIs:

• Information Hiding – hide implementation• Provide device independence• Enable future changes to low level without requiring changes to application code• Protection of critical resources• Consistency for product consumer – toolkit can provide commonality• More robust code – toolkit implemented correctly• Run-time efficiency: provide services in an efficient way• Code reuse: Provide useful services only once• Programmer Productivity• …


My Goal: Allow API usability to be a first-class quality metric considered by API designers 4

APIs are Important and Valuable

• www.programmableweb.com – 17,508 APIs• Apigee says 77% of companies rating APIs

“important” to making their systems and data available• Total market for API Web

middleware was $5.5 billion in 2014

• Google recently boughtApigee for $625M


5

http://www.programmableweb.com/

Why Apply Human-Centered Techniques?

• Programming is a human activity• Take the human into account

• “DevX” – like UX for User Experience• APIs are the “interface” between the programmer and the functionality• Design should be close to user’s plan

• “Programming is the process of transforming a mental plan into one that is compatible with the computer.” — Jean-Michel Hoc

• Closeness of mapping — Green and Petre• If an API cannot be used effectively by developers, it doesn’t work!

• Even if it provides the right functionality• Using APIs incorrectly has resulted in bugs and security problems• Usability and quality are key influencers for the decision about which APIs to use


6

“Human Centered Methods” ̶̶More Than Just Lab User Studies


7

• Contextual Inquiry• Contextual Analysis• Paper prototypes• Think-aloud protocols• Heuristic Evaluation• Affinity diagrams• Personas• Wizard of Oz• Task analysis• A/B testing• Cognitive Walkthrough• Cognitive Dimensions • KLM and GOMS (CogTool)• Video prototyping

• Body storming• Expert interviews• Questionnaires• Surveys• Interaction Relabeling• Log analysis• Storyboards• Focus groups• Card sorting• Diary studies• Improvisation• Use cases• Scenarios• “Speed Dating”• …

Human Centered Approaches Across the Lifecycle

Exploratory Studies Contextual Inquiries Interviews Surveys Lab Studies Corpus data mining

Evaluative Studies Expert analyses Usability Evaluation Formal A/B Lab Testing

Design Practices “Natural programming” Graphic & Interaction

Design Prototyping

Field Studies Logs & error reports

[Myers, Ko, LaToza, Yoon. IEEE Computer, 2016]


8

HCI Techniques We Have Used for APIs• “Contextual Inquiry” & Field Studies

• What are the real problems & barriers that developers face?• “Natural Programming Elicitation”

• Let programmers express how they expect the functionality to be provided• How should this API be designed?

• Expert analyses• What are some potential problems with this API?• Heuristic Analysis – evaluate based on guidelines• Cognitive walkthrough – how hard will this specific task be to learn?

• Lab studies of programmers using an API• Does my API work for programmers?• What problems do the target developers have with my API?• Is this design better than that one?


9

“Natural Programming” Elicitation• Technique developed by my group to elicit developer’s

“natural” expressions• Mental models of tasks, vocabulary, etc.

• Blank paper tests• Must prompt for the tasks in a way that doesn’t bias the

answers• Examples:

• API Architecture• Words used• Which methods are on which classes


10

Context:Natural Programming Project• Researching better tools for programming since 1978• Natural Programming project started in 1995 • Make programming easier and more correct by making it more natural

• Closer to the way that people think about algorithms and solving their tasks• Methodology – human-centered approach

• Perform studies to inform design• Provide new knowledge about what people do and think, & barriers

• Guide the designs from the data• Design of programming languages and environments

• Iteratively evaluate and improve the tools• Target novice, expert and end-user programmers


11

http://www.cs.cmu.edu/%7ENatProg/index.htmlhttp://www.cs.cmu.edu/%7ENatProg/index.html

End User Programming• People whose primary job is not programming• [Scaffidi, Shaw and Myers 2005]

• 90 million computer users at work in US• 55 million will use spreadsheets or databases at work (and therefore may

potentially program)• 13 million will describe themselves as programmers• 3 million professional programmers

• All of these people use APIs!


90,000,000

55,000,000

13,000,0006,000,000 3,000,000

0

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

Users Spreadsheets andDBs

Self-DescribedProgrammers

Scientists &Engineers

ProfessionalProgrammers

12

Program Complexity and Sophistication

Goal: Gentle Slope Systems

Difficultyof

Use

Goal

Flash

ActionScript

C Programming

Visual Basic

Basic

C or C# ProgrammingSwing

Java

LowThreshold

HighCeiling


Web Development

CSS & HTML

JavaScript

editor

Server-side

Our Studies of APIs• Our work started with Jeff Stylos’s PhD, 2005-2009

• Interned in Microsoft’s API Usability group with Steven Clarke, et. al.• Which programming patterns are most usable?• Measures: learnability, errors, preferences• Studied:

• Required parameters in constructors• Factory pattern• Object design• SAP’s APIs

• Tools to help with APIs• New work: API designers’ needs


14

What can be addressed?

• All API designdecisions

• Tools &documentationfor APIs


15[Stylos & Myers, VL/HCC ‘2007]

Required Constructors Study• Compared create-set-call (default constructor)

var foo = new FooClass();foo.Bar = ;foo.Use();

• vs. required constructors (immutable classes):var tempBar = ;var foo = new FooClass(tempBar);foo.Use();

• All participants assumed there would be a default constructor• Required constructors interfered with learning

• Users wanted to experiment with what kind of object to use first• Preferred to not use temporary variables• Tradeoff with the security and reliability of immutable classes

• See [Coblenz, Nelson, Aldrich, Myers, Sunshine: “Glacier: Transitive Class Immutability for Java”], Wed @ 11:00© 2017 – Brad A. Myers

16

[Stylos & Clarke, ICSE ‘2007]

“Factory” Pattern Study• Instead of “normal” creation: Widget w = new Widget();• Objects must be created by another class:

AbstractFactory f = AbstractFactory.getDefault();Widget w = f.createWidget();

• Used frequently in Java (>61) and .Net (>13) and SAP• Results:

• When asked to design on “blank paper”, no one designed a factory• Time to develop using factories took 2.1 to 5.3 times longer compared

to regular constructors (20:05 v. 9:31, 7:10 v. 1:20)• All subjects had difficulties getting using factories in APIs


17

[Ellis. Stylos & Myers, ICSE ‘2007]

Object Method Placement Study• Where to put functions when doing object-oriented

design of APIs when multiple classes work together• mail_Server.send( mail_Message )

vs.mail_Message.send( mail_Server )

• When desired method is on the class that they start with, users were between 2.4 and 11.2 times faster (p < 0.05)

• Starting class can bepredicted based on user’s tasks

• More general terms should beused most commonly• Mail vs. Mail_server class• Java File class


18

Time to Find a Method

0

5

10

15

20

Email Task Web Task Thingies TaskTi

me

(min

)

Methods onExpected ObjectsMethods onHelper Objects

[Stylos & Myers, FSE 2008]

Chart2

Email TaskEmail Task

Web TaskWeb Task

Thingies TaskThingies Task

Methods on Expected Objects

Methods on Helper Objects

Time (min)


1

11.2

2

15.2

2

6.8

Sheet1

Average Time in Minutes to Find a Method (FSE paper) (p < 0.05)Email TaskWeb TaskThingies Task

Methods on Expected Objects122

Methods on Helper Objects11.215.26.8

Average Time in Minutes to Construct (Factory Pattern) (p < 0.05)Sockets TaskThingies Task

Using a Default Constructor9.51666666671.3333333333

Using a Factory Pattern20.08333333337.1666666667

Sheet1

Methods on Expected Objects

Methods on Helper Objects

Time (min)


Sheet2

Using a Default Constructor

Using a Factory Pattern

Time (min)

Time to Construct Objects

Sheet3

Study of APIs for SAP

• Study APIs for Enterprise Service-Oriented Architectures (“Web Services”)

• Naming problems:• Too long• Not understandable• Differences in middle are frequently missed


19

[Jeong, Xie, Beaton, Myers, Stylos, Ehret, Karstens, Efeoglu, Busse, IS-EUD'2009]

eSOA Documentation Results

• Multiple paths: unclear which one to use• Some paths were dead ends• Inconsistent look and feel

caused immediate abandonmentof paths

• Hard to find required information• Business background helped• Many other studies have reported

documentation problems


20

(IS-EUD'2009)

Usability study of an API from SAP• Jeff Stylos as summer intern at SAP• SAP “Business Rules Framework Plus” API (BRFplus)• Interviews with users

• Identified a mismatch of abstraction level• API was very flexible, but users had simple use cases

• Natural programming techniques to identify expected designs

• User studies of redesigned APIs• Showed were successful

• Three months total work© 2017 – Brad A. Myers

21

[Stylos, Busse, Graf, Ziegler, Ehret, Karstens, VL/HCC’2008]

Evaluation based on Guidelines

• Nielsen’s Heuristics, Cognitive Dimensions• Also Cognitive Walkthroughs• Example: consistency violation:


22javax.xml.stream.XML.StreamWriter – [Rama, Kak, 2013]

SAP’s NetWeaver® Gateway Developer Tools

• Plug-in to Visual Studio 2010 for developing using certain SAP APIs

• We used the HCI methods of heuristic evaluation and cognitive walkthroughs to evaluate early prototypes

• Our recommendations were quickly incorporated due to agile software development process


23

Automated Tools (by Others)• Apply nine metrics to APIs

[Rama, G.M. and Kak, A. Some structural measures of API usability. Software: Practice and Experience 45, 1 (Jan. 2013), 75–110]

• E.g., consistency; avoid lists of strings; factory pattern; generic exceptions; …

• API Concepts Framework takes uses of APIs into account[Scheller, T. and Kuhn, E. Automated measurement of API usability: The API concepts framework. Information and Software Technology 61 (May 2015), 145–162]

• Interface Complexity; Implementation Complexity;Setup Complexity


24

Our Tools to Help with APIs• If cannot change API, then fix the documentation and tools


25

• Mica

• Jadeite

• Calcite

• Dacite

• Euklas

• Graphite

• Apatite

http://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=Nhttp://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=N

Mica Tool to Help Find Examples

• MICA: Makes Interfaces Clear and Accessible

• Use Google to find relevant pages• Match pages with Java keywords• Also notes which pages contain

example code or definitions


26

[Stylos, Myers VL/HCC’2006]

Jadeite: Improved JavaDoc• JADEITE: Java API Documentation with Extra

Information Tacked-on for Emphasis• http://www.cs.cmu.edu/~jadeite

• Mine the web for usage of Java APIs• Fix JavaDoc to help address problems

• Focus attention on most popular packagesand classes using font size

• “Placeholders” for methods that userswant to exist

• Automatically extracted code examplesfor how to createclasses

• Related classes© 2017 – Brad A. Myers

27

[Stylos, Faulring, Yang, Myers, VL/HCC’2009]

http://www.cs.cmu.edu/%7Ejadeitehttp://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=Nhttp://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=N

Calcite: Eclipse Plugin for Java• CALCITE: Construction And Language Completion

Integrated Throughout Eclipse• http://www.cs.cmu.edu/~calcite

• UI = Code completion in Eclipse since familiar and usable• Code completion in Eclipse augmented with Jadeite’s

information• How to create

objects of specificclasses?SSLSocket s = ???


28

[Mooty, Faulring, Stylos, Myers, VL/HCC’2010]

|

http://www.cs.cmu.edu/%7Ecalcite

Dacite: API Designer Annotates• DACITE: Design Annotations for Complementing Interfaces Targeting Effectiveness• Visiting Professor André L. Santos from University Institute of Lisbon, Portugal• Use Java annotations to declare properties of APIs

• Instead of needing to search the web for them• More accurate & works for APIs with small user bases

• Processed by Eclipse plugin to help with API discoverability• Unifies what Calcite did through crawling the web:

• Supports static factories, factory methods, object builders, helper methods• Also adds additional patterns: decorators and composite classes• API designers know better what

should be annotated

• Lab user study showed effective• Twice as many tasks finished© 2017 – Brad A. Myers

29

[Santos, Myers, Journal of Systems & Software, April, 2017]

Euklas: Eclipse Plugin for JavaScript

• EUKLAS: Eclipse Users’ Keystrokes Lessened by Attaching from Samples• http://www.cs.cmu.edu/~euklas

• Postdoc Christian Dörner• Brings Java-like analysis to

JavaScript• People often copy from examples

in documentation• Auto-correct uses copy source

context for errors due to copy &paste


[Dörner, Faulring, Myers, PLATEAU'2014]

30

http://www.cs.cmu.edu/%7Eeuklas

Graphite: Eclipse Plugin for Literals

• GRAPHITE: GRAphical Palettes Help Instantiate Types in the Editor.

• Pop up a custom palette forspecialized constants (literals)in Eclipse• Regular

expressions• Color palettes

• Customizable© 2017 – Brad A. Myers

31

[Omar, Yoon, LaToza, Myers, ICSE'2012]

Apatite Documentation Tool

• APATITE: Associative Perusing ofAPIs That Identifies Targets Easily• http://www.cs.cmu.edu/~apatite

• Start with verbs (actions) andproperties and find whatclasses implement them

• Find associated items• E.g., classes that are often used together• Classes that implement or are used by

a method© 2013 – Brad A. Myers

32

[Eisenberg, Stylos, Faulring, Myers, VL/HCC'10]

http://www.cs.cmu.edu/%7Eapatite

New Project• Funded by a grant from Google• Interview and survey API designers

• Processes used• Barriers to high usability• Information needs about API users• Appropriate signals of API usability

• Preliminary discussions with Google, IBM, Amazon, Bloomberg, Microsoft• Appear to have quite different processes• Different levels of sensitivity to API Usability

• Starting next week!© 2017 – Brad A. Myers

33

Open Challenges

• What other design patterns in APIs are problematic or beneficial for usability?

• How to make coordinating multiple APIs easier?• What other design or evaluation methods are

needed?


34

Open Challenges• Identify best practices in API Design

• How to insure that usability is a key quality metric that API designers always consider?

• What process results in the most usable APIs?• What kinds of testing should be done on APIs for usability?• How should the API design team be organized?

• What guidelines should be followed?• Does having guidelines even work?• Level of specificity of guidelines?

• E.g., Smith and Mosier’s 1986, 486 pages of guidelines, vs. Nielsen’s 10


35

A Few Resources• Brad A. Myers and Jeffrey Stylos, "Improving API

Usability", Communications of the ACM, vol 59, No. 6, June, 2016, pp. 62-69, Official ACM DL entry; html or local pdf.

• www.apiusability.org• http://www.cs.cmu.edu/~NatProg/apiusability.html

• https://www.programmableweb.com/


36

http://dl.acm.org/authorize?N03391http://cacm.acm.org/magazines/2016/6/202645-improving-api-usability/fulltexthttp://www.cs.cmu.edu/%7Enatprog/papers/p62-myers-CACM-API_Usability.pdfhttp://www.apiusability.org/http://www.cs.cmu.edu/%7ENatProg/apiusability.htmlhttps://www.programmableweb.com/

Euklas: EclipseUsers’ Keystrokes Lessened by Attaching from Samples

Acronyms are fun!And there are lots of Gemstones!! C32CMU's

Clever and Compelling Contribution to Computer Science in CommonLisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of CreativeConstructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, andCommunicated using Columns of Cells, that are Constantly Calculated so they Change Continuously, and Cancel Confusion

PebblesPDAs for Entry of Both Bytes and Locations from External Sources

GARNETGenerating anAmalgam ofReal-time,NovelEditors andToolkits

For more, see: www.cs.cmu.edu/~bam/acronyms.html

Azurite:AddingZest toUndoing and Restoring ImprovesTextual Exploration

Fluorite:Full ofLow-levelUserOperations Recorded InTheEditor

Apatite: Associative Perusing of APIsThatIdentifies TargetsEasily

Graphite: GRAphical PalettesHelpInstantiate Types in the Editor

Calcite: Construction AndLanguage Completion Integrated Throughout

Jadeite:JavaAPIDocumentation with ExtraInformationTacked-on for Emphasis

Mica:Makes InterfacesClear and Accessible

Jasper:JavaAid withSets of Pertinent Elements for Recall

Crystal:Clarifications Regarding YourSoftware using aToolkit, Architecture and Language

Euclase:EndUserCentered Language, APIsSystem and Environment

Aquamarine: AllowingQuickUndoing ofAnyMarksAndRepair Improving NovelEditing

Gneiss: GatheringNovelEnd-userInternetServices usingSpreadsheets

GlacierGreatLanguagesAllowClass Immutability EnforcedReadily

Variolite: Variations Augment Real Iterative Outcomes Letting Information Transcend Exploration

SugiliteSmartphone UsersGenerating Intelligent LikeableInterfaces ThroughExamples


37

http://www.pebbles.hcii.cmu.edu/index.phphttp://www.pebbles.hcii.cmu.edu/index.phphttp://www.cs.cmu.edu/%7Ebam/acronyms.htmlhttp://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=Nhttp://images.google.com/imgres?imgurl=http://www.orientaljadejewelry.com/DSCN0187.JPG&imgrefurl=http://www.orientaljadejewelry.com/bangles.htm&h=480&w=640&sz=33&hl=en&start=80&usg=__8TwN8oO0hOBrRBDl-NNOisyBsi4=&tbnid=7Hnu7F-4FzBLHM:&tbnh=103&tbnw=137&prev=/images?q=jadeite&start=60&ndsp=20&hl=en&rls=IBMA,IBMA:2006-17,IBMA:en&sa=N

Thanks to:


38

• Funding:• NSF under IIS-1116724, IIS-0329090, CCF-0811610, IIS-0757511 (Creative-IT),

ITR CCR-0324770 as part of the EUSES Consortium• SAP• Adobe• IBM• Microsoft Research• Yahoo! InMind• Google

• >38 students & visitors: Htet Htet Aung Jack Beaton Ruben Carbonell John R. Chang Kerry S. Chang Polo Chau Luis J. Cota Michael Coblenz

Christian Dörner Dan Eisenberg Brian Ellis Andrew Faulring Aristiwidya B. (Ika)

Hardjanto Erik Harpstead Amber Horvath Sae Young (Sophie)

Jeong Mary Beth Kery

Andy Ko Thomas LaToza Joonhwan Lee Toby Li Leah Miller Steven Moore Mathew Mooty Gregory Mueller Yoko Nakano Stephen Oney John Pane

Sunyoung Park Michael Puskas Chotirat (Ann)

Ratanamahatana André L. Santos Christopher Scaffidi Jeff Stylos David A. Weitzman Yingyu (Clare) Xie Zizhuang (Zizzy) Yang YoungSeok Yoon

http://www.nsf.gov/http://www.nsf.gov/

Human-Centered Methods for Improving API Usability

Brad A. MyersHuman-Computer Interaction InstituteSchool of Computer ScienceCarnegie Mellon Universityhttp://www.cs.cmu.edu/[email protected]


http://www.cs.cmu.edu/%7Ebammailto:[email protected]

Human-Centered Methods for Improving API UsabilityAPIsStakeholders &�Their GoalsWhy APIs?APIs are Important and ValuableWhy Apply Human-Centered Techniques?“Human Centered Methods” ̶̶ � More Than Just Lab User StudiesHuman Centered Approaches Across the LifecycleHCI Techniques We Have Used for APIs“Natural Programming” ElicitationContext:�Natural Programming ProjectEnd User ProgrammingGoal: Gentle Slope SystemsOur Studies of APIsWhat can be addressed?Required Constructors Study“Factory” Pattern StudyObject Method Placement StudyStudy of APIs for SAPeSOA Documentation ResultsUsability study of an API from SAPEvaluation based on GuidelinesSAP’s NetWeaver® Gateway Developer Tools Automated Tools (by Others)Our Tools to Help with APIsMica Tool to Help Find ExamplesJadeite: Improved JavaDocCalcite: Eclipse Plugin for JavaDacite: API Designer AnnotatesEuklas: Eclipse Plugin for JavaScriptGraphite: Eclipse Plugin for LiteralsApatite Documentation ToolNew ProjectOpen ChallengesOpen ChallengesA Few ResourcesAcronyms are fun!Thanks to:Human-Centered Methods for Improving API Usability

human-centered methods for improving api usabilitynatprog/papers/myers wapi 2017... ·...

Documents