Feiner, COMS W4170, Fall 2017

1

COMS W4170Programming by Demonstration 3Steven FeinerDepartment of Computer ScienceColumbia University New York, NY 10027

November 28, 2017

2

Chimera D. Kurlander 1988–92

Graphical history can be used to select past actions

7:00-14:30https://www.youtube.com/watch?v=JbrJQW25ekI

Feiner, COMS W4170, Fall 2017

3

Chimera D. Kurlander 1988–92

Macro creation

Confirmation of inferences Macro Invocation

4

Chimera D. Kurlander 1988–92

Testing

Feiner, COMS W4170, Fall 2017

5

Chimera D. Kurlander 1988–92

Debugging

6

Chimera D. Kurlander 1988–92

Testing again

Feiner, COMS W4170, Fall 2017

14

Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009

15

Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009

Implemented in GimpCapture UI state and app state

Prune changes that don’t make a difference

Use existing vision-based recognition/labeling techniques to classify parts of images

E.g., faces, landscapesUse heuristics to generate related text

Relative location, “nose”, “eyes” (and parts: “left pupil” “right iris”, “sky”)

Feiner, COMS W4170, Fall 2017

17

Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009

Tutorial GenerationOne step per major image processing op

Aggregate related and repeated actions (e.g., parm tweaking)

Annotate screenshots with arrows and highlights corresponding to user actions

Crop to include only “relevant” portionsGenerate text descriptions

Template-based “fill in the blanks” approachLay out screenshots and text in grid

Use text (on left)-screenshot (on right) pairsUse before-after screenshot pairs to show consequencesUse screenshots of multiple actions when necessary

Stack pairs in columns, laid out left-to-rightInclude before-after screenshot pair at left for entire task

Berthouzoz et al., ACM Trans. on Graphics 2011 generalizes to create parameterized macros

18

Generating Photo Manipulation Tutorials F. Grabler et al. SIGGRAPH 2009

Follow-on work: Adobe Tutorial Builder for Photoshop

http://labs.adobe.com/technologies/tutorialbuilder/

Feiner, COMS W4170, Fall 2017

19

Excel 2013 (, 2016,…) “Flash Fill”S. Gulwani

Infers iterations over cells based on one or a small number of examples

Type and press <enter> to accept suggestionsOtherwise press <ESC>Or use ctrl-E to execute

http://office.microsoft.com/en-us/excel-help/split-a-column-of-data-based-on-what-you-type-HA102809794.aspx?CTT=5&origin=HA102809308

https://www.microsoft.com/en-us/research/project/flash-fill-excel-feature-office-2013/

0:10−

20

Apparition W. Lasecki et al., CHI 2015

Crowd-poweredrefinement of UI prototypeCrowd-powered “wizard-of-Oz”animation of prototype inresponse to designer’s spec

Crowd-powered refinement of prototypehttps://www.youtube.com/watch?v=tBCB6P7FwWY

Feiner, COMS W4170, Fall 2017

21

SketchExpress S. Lee et al., UIST 2017(built on Apparition)

Crowd-powered creation, editing, and saving of animated behaviors for UI prototypes by workers“Demonstrate–remix–replay” approach

Worker candemonstratebehavior that isrecorded,remix it to modifyparameters, andreplay it whenneededRecordedbehaviorsare saved

https://www.youtube.com/watch?v=A_Pngz1mbDs