user interface toolkits and end user programming for uis vishal dwivedi institute for software...

User Interface Toolkits and End User Programming for UIs

Vishal Dwivedi

Institute for Software ResearchCarnegie Mellon University

[email protected]

Advanced User Interface Software

1

Carnegie Mellon University, School of Computer Science

2

“End-user programming enables end users to create their own programs. Today, millions of end users create numerous programs [Scaffidi et al, 2005]

And they make a mess out of it

Christopher Scaffidi, Mary Shaw, Brad A. Myers: Estimating the Numbers of End Users and End User Programmers. VL/HCC 2005: 207-214


Agenda

Motivation/Definitions Examples (of EUP toolkits)

Alice, SUIT, Garnet, HANDS, SWiFT, … Historical trends Key techniques

VPL, PBD, Natural Programming, Application specific languages,…

Challenges

3


Definitions “Programming”

‘‘The process of transforming a mental plan of desired actions for a computer into a representation that can be understood by the computer’’ – Jean-Michel Hoc and Anh Nguyen-Xuan

“End-User Programming” Programming by people who write programs, but not as their

primary job function.

Instead, they must write programs in support of achieving their main goal, which is something else.

Covers a wide range of programming expertise Business analysts, Neuroscientists, Physicists, Teachers, Accountants, etc.

4


Definitions

“End User Programming toolkits” An assembly of tools; set of basic building units that help end

users do end user programming, mostly by manipulating or creating graphical user interfaces.

Examples: Toolkits for:

Learning to write code Creating web pages Simulations Email filtering Other user-created functionality.

5


Major Techniques “Visual Programming”

“Programming in which more than one dimension* is used to convey semantics.” - Myers, 1990

A programming language that lets users create programs by manipulating program elements graphically rather than by specifying them textually - Golin , 1990

“Programming by Demonstration” An end user development technique for teaching a computer or a

robot new behaviors by demonstrating the task to transfer directly instead of programming it through machine commands.

- Dan Halbert, 1984 “Natural Programming”

Set of techniques to make making programming languages and environments easier to learn, more effective, and less error prone. - Myers, 1990-

6

* Multi dimensional graphical objects, spatial relationships, time relationships (before/after)

Carnegie Mellon University, School of Computer ScienceExample ToolkitsAlice [Randy Pausch, 2000]

7

Youtube Video

https://www.youtube.com/watch?v=K2XA8mx3sKc

Carnegie Mellon University, School of Computer ScienceExample ToolkitsGarnet [Myers, 1988]

8

One of the earliest GUI development toolkit

Aimed at tools to help designers create, modify and maintain highly interactive, graphical, direct manipulation interfaces.

Carnegie Mellon University, School of Computer ScienceExample ToolkitsSUIT [Randy Pausch, 1992]

9Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User Interface Toolkit. ACM Transactions on Information Systems, 10(4), 320-344.

Provides a collection of screen objects described by its: state a C procedure that displays it a C procedure that handles input and updates the object state

Similar toolkits : UIMX, Interviews, Motif widgets

Carnegie Mellon University, School of Computer ScienceExample ToolkitsHANDS [J.F. Pane, 2002]

10

J.F. Pane, "A Programming System for Children that is Designed for Usability," Ph.D. Thesis, Carnegie Mellon


Example ToolkitsOzone – a publish-subscribe based widget composition and creation toolkit

A popular framework that enables rapid creation, assembly and configuration of rich, lightweight widgets. Promoted to create a common widget repository for the US

military – similar to ‘Android Market’.

Link11

https://www.owfgoss.org/


Programming Paradigm for Programming with Pictures, particularly visual Simulations and rules.

Rule based design to:

animate agents play sounds and MIDI instruments speak text react to mouse and keyboard events visualize values as colors and plots query web pages, control web

browsers

Example ToolkitsRule based programming (Agent Sheet, MOSL, VISPATCH, SAM, VXT, …)

12Agent Sheet: http://www.agentsheets.com

http://www.agentsheets.com/


http://www.agentsheets.com/Applets/frogger/index.html

Carnegie Mellon University, School of Computer ScienceExample ToolkitsDataflow based (PROGRAPH, ObjectFlow, The Cube Language, SWiFT,…)

General-purpose Data-flow driven languages

Information flows through components

Data is transformed at certain points in the program

Operations

Method case

13


Example ToolkitsSWiFT– a web-based toolkit web service composition (Neuroscience example)

ALIGN

TEMPORAL FILTERING

SPATIALFILTERING

/usr/local/fsl/bin/flirt -ref standard -in example_func -out example_func2standard -omat example_func2standard.mat -cost corratio -dof 12 -searchrx -90 90 -searchry -90 90 -searchrz -90 90 -interp trilinear

Program (a large number of binaries that perform one or more functions)

Parameters (numbers range from 5 to 25)

A large script file that contains program calls

14

Carnegie Mellon University, School of Computer ScienceHistorical trends

15

1960 1980 1990 2000

oAMBIT/G/LoGrailoGALoGraphical Program

EditoroQuery by ExampleoPygmaliono I/O Pairs

oAction Graphics

oFORMALoThingLaboHi-VisualoLabViewoPROGRAPHoPIGSoPictoRehearsaloSmallStar

oFormsoEditing by

ExampleoPICToLotus 1-2-3oSIL-ICONoVisiCalcoHiGraphsoMirooStateMaster

oCubeoCantataoSchemePaintoCODE 2.0o IconicodeoMViews

Techniqueso Graphso Flowchartso Flowchart derivativeso FORMSo Demonstrational

Techniqueso Graphso Flowchartso Flowchart

derivativeso FORMSo Demonstrationalo Data Flowso Spreadsheetso Matriceso Jigsaw Puzzleso Petri netso Flowchart

derivatives

oAVSoMondrianoChemTrainsoVampireoVIPRoSPE

Techniques/Goalso 3D Renderingo Visual Hierarchyo Procedureso Control Structureso Programmable Graphicso Animationso Video Imagery Exploitationo General purpose, declarative languageo Audio, video and image processingo Graphical models from behavioral modelso Learning and Cognitive abilities in vision

processeso Handling Scalability, typing, and

imperative designo Collaborative Software Development

oLOFI/HIPIoFOXQoVMQLoGXLoEuler ViewoYahoo PipesoPopfly

Techniques/Goalso Child Learningo Xquery by FORMSo Spreadsheet Analysiso Visual Model Queryo Layoutso Specification and Interchangeo Mashupso Web-based designo Programming for end-users

(non-Professionals


Historical trends

16

1960 1980 1990 2000

oMake programming more accessible

oNatural programmingoSupport domain -

specific designs

oLet users program in Visual Languages

o (Almost) Make textual languages redundant

oStrive for improvements in programming language design

oSupport the cognition aspect of Programming


“Programming by Demonstration” An end user development technique for teaching a computer or a

robot new behaviors by demonstrating the task to transfer directly instead of programming it through machine commands.

Many toolkits based on this technique: COCOA, HyperCard, Grammax, Mail Filtering

17

Halbert, Dan (November 1984). Programming by Example. U.C. Berkeley (PhD dissertation)

Key TechniquesProgramming by demonstration [Dan Halbert, 1984]

http://danhalbert.org/pbe.pdf



“Visual Programming” “Programming in which more than one dimension* is used to

convey semantics.” - Myers, 1990

Many, many toolkits based on this technique. Myers’ Taxonomy - Myers, 1990 Burnett’s Taxonomy - Burnett, 1994

18

Key TechniquesVisual Programming by demonstration

Brad Myers, Taxonomies of Visual Programming and Program Visualization, 1990Margaret Burnett, A Classification System for Visual Programming Languages, 1994.


“Natural Programming” Set of techniques to make making programming languages and

environments easier to learn, more effective, and less error prone. - Myers, 1990 - Present

“Brad Myers Natural Programming project” http://www.cs.cmu.edu/~NatProg/index.html

19

Key TechniquesNatural Programming

http://www.cs.cmu.edu/~NatProg/index.html

http://www.cs.cmu.edu/~NatProg/index.html


“Direct Manipulation” A technique that involves users to directly manipulate objects

presented to them, using actions that correspond at least loosely to the physical world.

“Central Ideas” Visual Model of the world Visual objects that can be operated on Results of actions are reflected in the objects immediately. Objects, once operated on, can be further operated on. - Shneiderman, 1983

20

Key TechniquesDirect Manipulation

Shneiderman, Ben. "Direct Manipulation. A Step Beyond Programming Languages" (HTML). IEEE Transactions on Computers


“Application/Domain specific Languages” A popular trend is to provide toolkits that allow end users to

create domain/application specific platforms. Many of them are VPL platforms

Examples: Dataflow based: Yahoo Pipes Pub-sub based (widget composition): Ozone, Synapse Service composition based: Loni Pipeline, Taverna Ontology based: WINGS Scripting based: TCL-TK, various domain-specific scripts

21

Key TechniquesApplication/Domain specific Languages


22

ChallengesUnderstanding the end user ecosystem

Scientists,Physicists,

Astronomists

Accountants,Moms and Pops

The goal of End user programming is to support their professional task. But it is not always that they perform all these roles themselves.

People with Professional end user software developer role [Segal, 07]

People with a(technically) novice role

Domains that involve using or adapting turn-key software

Domains that involve writing a lot of code

Domains that involve adaptations of software,

and change in configurations


23

ChallengesUnderstanding the end user ecosystem…

“Multiple roles in Component based software development” Framework builders,

Component developers, Application assemblers, and end users

Additional role: end user developers – mainly involved with tailoring the applications.

Successful tools like Taverna, WINGS, LONI-pipeline are built on such ecosystems.

Morch, A., Stevens, G., et al. (2004) Component-based technologies for end-user development. Communications of the ACM, 47(9), pp. 59-62


Supporting reuse of existing end user code requires efficient packaging and search.

Current work*: Package new capability as plugins [Scaffidi 2008, Sestoff 2002] Finding resources through hierarchy of repositories from personal to

local to global

[Scaffidi, 2009]

24

ChallengesReuse is hard

*mostly in context of spreadsheets


Most toolkits generate or select code as a result of user interaction. Users typically have limited abilities to make changes in that code.

For example in the SUIT toolkit: Creating custom widgets was hard Teaching callbacks was not possible

Studies have demonstrated that programs produced using various end user programming toolkits are erroneous up to the level of 90% [Leventhal et al. 1994,

Stolee et al. 2010] There is limited support to analyze and fix such errors.

25

ChallengesTesting, verification and analysis is even harder (and hardly available)


Many end user programming tools are developed from scratch, leading to: High cost of development, and Low quality

Instead of many tools, what is needed is frameworks to support: Component level reuse Testing and analysis Execution support, and finally Packaging and sharing

26

ChallengesHigh Cost and low quality end user programming tools


Next…

We discuss an approach called End User Architecting that is aimed to address some of these challenges…

27


References

28

Agent Sheet: http://www.agentsheets.com Ben Shneiderman. "Direct Manipulation. A Step Beyond Programming Languages"

(HTML). IEEE Transactions on Computers Brad A. Myers, John F. Pane and Andy Ko, "Natural Programming Languages and

Environments". Communications of the ACM. 47(9), pp. 47-52. Brad A. Myers. "Taxonomies of Visual Programming and Program Visualization," Journal of

Visual Languages and Computing. vol. 1, no. 1. March, 1990. pp. 97-123. Christopher Scaffidi, Christopher Bogart, Margaret M. Burnett, Allen Cypher, Brad A.

Myers, Mary Shaw: Predicting reuse of end-user web macro scripts. VL/HCC 2009: 93-100 Conway, M., Audia, S., Burnette, T., Cosgrove, D., Christiansen, K., Deline, R., et al. (2000,

Apr 1-6). Alice: Lessons Learned from Building a 3D System For Novices. Paper presented at the Proceedings of

CHI 2000, The Hague, The Netherlands. David Garlan, Bradley Schmerl, Vishal Dwivedi, Aparup Banerjee, Laura Glendenning, Mai

Nakayama, and Nina Patel. Swift: A tool for constructing workflows for dynamic network analysis. http://acme.able.cs.cmu.edu/pubs/show.php?id=333, 2011.

F.H. Post and W. Barth, Construction Techniques of Graphic, Direct-Manipulation User Interfaces, EUROGRAPHICS ’91 / Elsevier Science Publishers B.V. Eurographics Association, 1991.



References…

29

Halbert, Dan (November 1984). Programming by Example. U.C. Berkeley (PhD dissertation)

J.F. Pane, "A Programming System for Children that is Designed for Usability," Ph.D. Thesis, Carnegie Mellon

Joseph Lawrance, Steven Clarke, Margaret M. Burnett, Gregg Rothermel: How Well Do Professional Developers Test with Code Coverage Visualizations? An Empirical Study. VL/HCC 2005: 53-60

K. T. Stolee and S. Elbaum, "Refactoring Pipe-like Mashups for End User Programmers," International Conference on Software Engineering (ICSE), Honolulu, Hawaii, May 2011. to appear.

Margaret M. Burnett, “Visual Programming” In the Encyclopedia of Electrical and Electronics Engineering (John G. Webster, ed.), 1999

Morch, A., Stevens, G., et al. (2004) Component-based technologies for end-user development. Communications of the ACM, 47(9), pp. 59-62.

Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User Interface Toolkit. ACM Transactions on Information Systems, 10(4), 320-344. ACM DL Ref

Scaffidi C., Myers B.A., AND Shaw M. 2008. Topes: Reusable abstractions for validating data. International


user interface toolkits and end user programming for uis vishal dwivedi institute for software...

Documents

programming languages

end user programmers

user interface toolkits

end user development

numbers of end users

millions of end users

graphical user interfaces

computer jeanmichel