building whole applications using only programming-by- demonstration richard mcdaniel january 14,...
Post on 19-Dec-2015
216 views
TRANSCRIPT
![Page 1: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/1.jpg)
Building Whole Applications Using Only Programming-by-
Demonstration
Richard McDaniel
January 14, 1999
![Page 2: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/2.jpg)
2
Purpose
Allow nonprogrammers to build applications with original, dynamic behavior Use programming-by-demonstration No written language at all - programs
created solely through demonstration Implement in system called Gamut
![Page 3: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/3.jpg)
3
State of the Art
Full-fledged languages and libraries: Visual C++, Java
Simplified languages with syntax-supporting editor: Authorware, Agentsheets
Tools with macro languages: Director / LINGO, Excel / Visual Basic
![Page 4: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/4.jpg)
4
Outline
Introduction Interaction Techniques Video Inferencing User Study Conclusion
![Page 5: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/5.jpg)
5
Domain
“Board Games” Has background “board” “Pieces” above board show state Pieces follow rule-like behavior Autonomous behavior controlled by timers
Examples Tic-Tac-Toe, Pacman, Reader Rabbit
![Page 6: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/6.jpg)
6
Example: Pacman
![Page 7: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/7.jpg)
7
Innovations in Gamut
Interaction Techniques Allows developers to specify important
details Inferencing
Infer conditions from sources unaffected by the behavior
Infer chains of expressions
![Page 8: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/8.jpg)
8
Inferencing - Conditions
Relationship can be formed between arbitrary objects
– Relating a toggle switch to decide which way to move
Run Away
?
![Page 9: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/9.jpg)
9
GO
Expression Chain
Expressions form chain of relationships– In Monopoly, the position to move a piece is, “the square
which is the sum of the dice’s number of squares from the original position”
Red’s Turn
![Page 10: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/10.jpg)
10
Interaction Techniques
Nudges Do Something, Stop That Hint highlighting
Guide Objects Temporal Ghosts Cards and Decks
![Page 11: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/11.jpg)
11
Nudges
![Page 12: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/12.jpg)
12
Negative Examples
Standard technique for demonstrating examples: Macro Recorder Requires special controls to make a
negative example Negative examples in Gamut provided
by Stop That No special set-up required
![Page 13: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/13.jpg)
13
Guides, Ghosts, and Hints
![Page 14: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/14.jpg)
14
Cards and Decks
![Page 15: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/15.jpg)
15
Video
![Page 16: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/16.jpg)
16
Stages of Inferencing
TranslationMatchingActions
CodeGeneration
![Page 17: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/17.jpg)
17
Translation
Read events from undo history list Convert events to actions Eliminate redundant and
inconsequential events
![Page 18: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/18.jpg)
18
Matching Actions
Matches actions in trace to original behavior
– Unordered– Typically not many (three or four)
Considerations– Degree to which actions match– Priority of actions in original behavior– Actions which create new objects
![Page 19: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/19.jpg)
19
Recursive Difference
Propagates changes to descriptions Each description contains heuristics for
comparing values Builds an And/Or tree result
![Page 20: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/20.jpg)
20
Code Generation
Resolves each value difference by creating a new description Search for previously defined description Use heuristics to generate description from
highlighted objects Assemble values into a table and use
Decision Tree learning
![Page 21: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/21.jpg)
21
Decision Trees
Created at places where Gamut chooses between different behaviors
Generate attributes using heuristics and hint highlighting
Build table of examples that relate behaviors to attributes
Is Eaten
RunHome
Is Blue
ChasePacman
RunAway
Y N
Y N
Condition For Pacman Monster
![Page 22: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/22.jpg)
22
User Study
Goal Determine if nonprogrammers can use
Gamut Perform an informal study
Participants Nonprogrammers, CMU community
Location HCI Institute’s usability lab
![Page 23: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/23.jpg)
23
Tasks
Safari
G-bert
Pawn Race
![Page 24: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/24.jpg)
24
Expertise with Techniques
techniques P1 P2 P3 P4
Do Something X X X
Stop That X X
Highlights Ghosts X
Guide Objects X X
Cards and Decks X X
Player MouseIcons
X
![Page 25: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/25.jpg)
25
Problems
Reluctance to create guide objects Needed explicit instructions in tasks Once told, participants would create
appropriate objects Highlighting inappropriate objects
Some objects seem too obvious to highlight, like ghosts
![Page 26: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/26.jpg)
26
Future Work
Apply to other domains Feedback for just inferred relationships Display for language Higher-level inferencing
![Page 27: Building Whole Applications Using Only Programming-by- Demonstration Richard McDaniel January 14, 1999](https://reader036.vdocuments.us/reader036/viewer/2022062421/56649d3a5503460f94a157f4/html5/thumbnails/27.jpg)
27
Conclusion
Builds entire applications without a written programming language
Infers a broader range of behavior than prior systems
Is usable by nonprogrammers