henry lieberman mit media lab ai in end-user software engineering henry lieberman, mit (presenting)...
TRANSCRIPT
![Page 1: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/1.jpg)
Henry Lieberman • MIT Media Lab
AI in End-User Software Engineering
Henry Lieberman, MIT (presenting)
Raphael Hoffmann, Michael Toomim, U. Washington,
Seattle
![Page 2: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/2.jpg)
Henry Lieberman • MIT Media Lab
Roles for AI in End-User SoftEng
Supply knowledge end-user doesn't have
Recognize user intent
Plan generation and plan recognition
Automate common patterns of use
Generalize & Specialize
Manage multiple representations
Error checking and sanity checking
![Page 3: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/3.jpg)
Henry Lieberman • MIT Media Lab
U. Washington SoftBots & more
Planning-based software agents
Version-space Programming by Demonstration
Sequence prediction via Logic & Probability
Recommendation from software libraries
![Page 4: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/4.jpg)
Henry Lieberman • MIT Media Lab
SmartEdit - Lau
![Page 5: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/5.jpg)
Henry Lieberman • MIT Media Lab
Version Spaces
![Page 6: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/6.jpg)
Henry Lieberman • MIT Media Lab
Linked Editing
Alternative to content abstraction facilities
Powerpoint Master Slides
Style Sheets
"Symbols" vs. instances in Flash
Web templates, components
User stays within WYSIWYG paradigm
![Page 7: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/7.jpg)
Henry Lieberman • MIT Media Lab
Linked Editing
End-users "underuse" abstractions
Linked Editing
User edits concrete example
Generalizing over content rather than actions
Finds other instances of description
Performs "analogous" edits
Program code, Web pages, Vector graphics, Sheets
![Page 8: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/8.jpg)
Henry Lieberman • MIT Media Lab
Linked Editing
![Page 9: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/9.jpg)
Henry Lieberman • MIT Media Lab
Linked Editing
![Page 10: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/10.jpg)
Henry Lieberman • MIT Media Lab
Assieme - Customization Recommendation
Goal: Recommending scripts for application extensions: Emacs, Firefox
Collect keystrokes, file names, commands
Provide Web service for recommendation
Collab filtering, learning classifier
![Page 11: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/11.jpg)
Henry Lieberman • MIT Media Lab
Assieme - Customization Recommendation
![Page 12: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/12.jpg)
Henry Lieberman • MIT Media Lab
The crazy dream: Programming in English
Programming languages are too difficult for human beings
People can express procedures in English (or other natural language)
Why not program in natural language?
That was the original idea behind Cobol
(whereas Fortran tried to make programming like math)
![Page 13: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/13.jpg)
Henry Lieberman • MIT Media Lab
Childrens’ description of Pac-Man
![Page 14: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/14.jpg)
Henry Lieberman • MIT Media Lab
But what can you do if you can only partially understand English?
Need to make the interaction fail-soft
Rely on dialogue to resolve ambiguities and errors
Maybe don’t eliminate code entirely
Emphasize correspondence between language and code
An "outliner" for code
Application: Programming MOOs
![Page 15: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/15.jpg)
Henry Lieberman • MIT Media Lab
New resources for Common Sense Reasoning
Open Mind Common Sense knowledge base
ConceptNet Semantic Net
MontyLingua/LangUtils
![Page 16: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/16.jpg)
Henry Lieberman • MIT Media Lab
Metafor Interface
![Page 17: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/17.jpg)
Henry Lieberman • MIT Media Lab
Programmatic Semantics
English to “scaffolding” code
Roughly,
Turns nouns into classes and/or instances;
Verbs into functions
Adjectives into instance variables
Resolves anaphora
Untangles conditional and loops
Propagates context
![Page 18: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/18.jpg)
Henry Lieberman • MIT Media Lab
Narrative stances
I want to make a bar with a customer. (1st person programmer)
There is a customer in the bar. (3rd person objective)
I am a customer sitting on a stool. (1st person program)
The bartender said, “Here is a customer” (3rd person subjective)
![Page 19: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/19.jpg)
Henry Lieberman • MIT Media Lab
Refactoring
There is a bar. (atom)The bar contains two customers. (unimorphic list)It also contains a waiter. (unimorphic wrt. persons)It also contains some stools. (polymorphic list)The bar opens and closes. (class / agent)The bar is a kind of store. (inheritance class)Some bars close at 6pm. (subclass or instantiatable)
![Page 20: Henry Lieberman MIT Media Lab AI in End-User Software Engineering Henry Lieberman, MIT (presenting) Raphael Hoffmann, Michael Toomim, U. Washington, Seattle](https://reader035.vdocuments.us/reader035/viewer/2022062618/5514bbfa550346d36e8b66ea/html5/thumbnails/20.jpg)
Henry Lieberman • MIT Media Lab
Shameless Plug