symmetric multi-modal intelligent interactive development environment (smiide) seedling update as of...
TRANSCRIPT
Symmetric Multi-modal Intelligent Interactive Development
Environment (SMIIDE) Seedling
Update as of
September 21, 2009
BAE AIT: Greg Sullivan, Basil Krikeles, Mike Cook
MIT CSAIL: Randy Davis, Howie Shrobe, James Oleinik, Kyle Mill
September 21, 2009 SMIIDEling Overview 22Distribution authorized to U.S. Government Agencies only
Symmetric Multimodal Interactive Intelligent Development Environments
•Dramatic reduction in time and cost of constructing software designs and related engineering artifacts
•Production of higher quality artifacts through interactive design.
•Metrics to be refined during seedling.
Multi-modal User Interfaces can now integrate sketch, gesture,& natural language to capture design, rationale more naturally.
Mixed-Initiative User Interfaces enable symmetric collaboration between human and computer-based designers.
Adaptive, Domain-Specific Model-Based Software Engineering tools enable flexible collaboration at a human level of abstraction.
Symmetric multimodal interaction brings intelligence to model-based development
Multi-modal Mixed-initiative User Interface
• Supports sketching, gestures, natural language.
• System can query human for clarification, suggestion, etc.
• System can take initiative in design and implementation, freeing the human from detailed specification of lower-level details
Adaptive Domain Specific Model-Based Software Engineering tools
• Support evolution of domain “vocabulary”
• Support diagramming using symbology specific to a domain.
ASSUMPTIONS AND LIMITATIONS:
• Anticipated productivity gains can be demonstrated on well-defined domains.
• Current natural language technology is sufficient.
• Current Integrated Development Environments (IDEs) are developer-centric, detail-intensive, and error-prone
•Modality is essentially paper-based and human developer-centric
•IDEs support limited consistency checking and debugging
• Demonstrate program feasibility and develop program structure
•Prototype combination of symmetric multimodal HCI with model-based code generation technology
•Determine feasibility of program leading to mixed-initiative high-productivity environments
•Identify technology gaps and refine DARPA-hard program challenges.
•Demonstrate on a DoD-relevant domain
•Survey and position with respect to previous research
QU
AN
TIT
ATIV
E IM
PA
CT
EN
D-O
F-S
EED
LIN
G G
OA
L
STA
TU
S Q
UO
NEW
IN
SIG
HTS
Code
Com
pile
Test
System Complexity
Co
st
(Tim
e)
Savings
With MMIIDE
Current IDEs
Does this process need access to the
database?
Let’s create a data interface component
September 21, 2009 SMIIDEling Overview 3
Some seedling goals (from Jim’s slides at August meeting)
• Arrive at a focused set of technical challenges that form the basis of a new program
• The purpose of the seedling is to create a new program initiative brief
• A “Scientific American” statement of the technical goals of the program
• A clear connection to compelling operational need• A convincing case for how this can be achieved• The nature of the technical challenges, and which
focused challenges will form the basis for a program?• How far can we reach in terms of operational capability
in the scope of a program?• What is the potential operational payoff?• What options for demonstrating the concept and
measuring success?
September 21, 2009 SMIIDEling Overview 4
Outline
1. The Heilmeyer Questions2. A “Scientific American” statement3. Compelling operational need4. Research Topics for proposed program
1. Quick review of topics from August meeting.2. “Lots of Little Languages”.3. Rationale Capture via Natural Interaction4. Related research (“Why now?”)
5. Sample Scenarios1. Geospatial reasoning in Deep Green2. Building a wiki for after-action reports
September 21, 2009 SMIIDEling Overview 5
1. The Heilmeyers…1. What is the problem, why is it hard?
Software design, development, and maintenance. Low level, error-prone, complex, hard to verify. Slow to adapt to mission needs.
2. How is it solved today? Code-level “forward engineering”. That is, human-intensive high-to-low level specification with
computer “automaton” support for detail tracking (e.g. parsing) and high-to-low level translation (e.g. compilation).
3. What is the new technical idea; why can we succeed now? Natural interaction and DSML interpretation combine to enable mixed initiative interaction, with
useful initiative from computer, augmented by captured rationale. Drill down and machine learning enable both computer and human input at all levels of abstraction.
4. What is the impact if successful? Revolutionary increase in productivity for creating software-intensive systems. Enables fast
development and deployment of latest technology, rather than traditional slow DoD software acquisition process.
Dramatic increase in development / adaptation allows rapid response to mission needs.
5. How will the program be organized?
6. How will intermediate results be generated?
7. How will you measure progress?
8. What will it cost?
September 21, 2009 SMIIDEling Overview 6
A “Scientific American” statement of the technical goals of the program
• Every day, countless conversations take place between software engineers in front of white boards. The engineers take turns grabbing markers, and talk while drawing. In response to questions from each other, the engineers elaborate or erase parts of their diagrams. The interactions are highly verbal, and the drawings include text, traditional software diagrams, and ad-hoc notations made up on the spot.
• Now, what if one of those engineers was a computer?
• The goal of the SMIIDE program is investigate whether the combination of several cutting edge research areas can enable computer-based agents to become active participants in the software design, development, and maintenance process.
• The computer should be able to ask relevant questions, suggest useful approaches and designs, and thereby substantially contribute to the implementation of the eventual system.
Research Areas: Sketch recognition, Natural Language, Rationale Capture, Domain Specific Modeling, Dialogue management.
September 21, 2009 SMIIDEling Overview 7
Compelling Operational Need
• The problem: in a fast-changing world, it’s easy to miss the mission window
• Response: Rapidly tailor systems to mission needs– Can leverage rationale captured in natural modalities.
• Why were certain decisions made?• What were assumptions?
• Increased speed of development while maintaining correctness
• Decreased costsWhat current operational needs are addressed by a dramatic increase in speed and flexibility?
September 21, 2009 SMIIDEling Overview 8
Research Topics identified during August meeting
• Grounding complex abstract diagrams in sufficient meaning to support the design task;
• Determining how complex specifications can be segmented in ways that are functionally meaningful for reasoning and dialog;
• Determining what kind of human-computer interactions are needed at any given moment and mediating that interaction effectively;
• Determining the HCI principles that lead to productivity enhancements not attainable with model-based software engineering tools alone (including the advanced versions produced in the current DMT-SWP program).
• Is there a body of kn about MBSE applicable across domains?– Dev of DSMLs suggest any common body of language, lexicon,…– How wide a variety of specification forms to consider? tabular, text,
graphical
• Graphical models vs. behavioral specifications? How do we get the benefits of graphical structural models for behavioral specs?
September 21, 2009 SMIIDEling Overview 9
Core Research Topic: Related Families of Overlapping DSMLs
• Benefits of DSLs:–Small language decreases
errors.–More abstraction increases
productivity, improves analysis–DSMLs capture expertise of
domain experts• How to get coverage / generality?
–Lots of DSLs–Overlaps in domains gain
coverage by transitivity.–Evolution of DSLs
• Generators / Interpreters / Transforms
–Provide overlap by translation–Often high-to-low level (where
“level” is loosely defined).–Generator logic can be hand-
generated or partially learned.
Geospatial
Geometry
Course of Action
TenneygramsUML classdiagrams
Correspondence, Transformations
Make
Java
Scala
Correspondence, Transformations
pathlocationunit
September 21, 2009 SMIIDEling Overview 10
Core Research Topic: Natural Rationale Capture
• Can natural interaction make rationale capture (nearly) painless?– Why were certain decisions made?– What were assumptions?
• Can captured rationale provide the basis for “parametric design” of software?
September 21, 2009 SMIIDEling Overview 11
Rationale Capture - Rationale
• Every software project involves thousands of design decisions– Most are routine (i.e., obvious to an experienced
programmer); many crucial ones are not
• Writing/modifying/debugging/ software requires constantly re-absorbing those decisions.
• This would be orders of magnitude easier if all the program authors were always there to explain how they made those decisions and why.
• Rationale capture can achieve this effect.
September 21, 2009 SMIIDEling Overview 12
Rationale Capture – Technical Challenges
• Why isn’t rationale captured now?– Cost is too high (writing documentation is a lot of
work)– Task is aversive (writing documentation is unpleasant
work)– Task is disruptive (writing documentation interrupts
design)– The value equation is out of whack:
• The person incurring the cost is (almost) never the one who gains the benefit.
• Cost is now, benefit is in the future.
• Consequences: documentation of small but crucial decisions is deferred.
Can we change the equation by drastically lowering the cost and making it more like talking to another designer?
September 21, 2009 SMIIDEling Overview 13
Core Research Topics: Sketch Understanding, Symmetric Multimodal Interaction
• Sketch understanding– grounding complex abstract diagrams in sufficient
meaning to support the design task; – structure (e.g., data structure) is easy to draw, but a
lot of software is about process, behavior• can we find ways to draw behavior that feel as natural?
• Symmetric multimodal interaction requires an intelligent observer, e.g., knowing what questions to ask and when– How task- and domain-specific is that knowledge?– Is there a body of knowledge about MBSE applicable
across domains? Or is the problem AI-complete?
September 21, 2009 SMIIDEling Overview 14
Central Hypotheses
1. Natural interaction + modestly intelligent observer = drastic increase in agility and lower cost
2. Domain specificity enables intelligence of computer-based observer.
3. Evolvable and overlapping languages enables generality
September 21, 2009 SMIIDEling Overview 15
How far can we reach in terms of operational capability in the scope of a program?
• What to rule out?– “Programmer’s common sense”
• Domain expertise encoded in DSML “smarts”.• Mixed initiative: human in the loop.
• System must be “smart about something” but cannot be smart about everything.
September 21, 2009 SMIIDEling Overview 16
What options for demonstrating the concept and measuring success?
• Prototype and storyboards for demonstrating concept.
September 21, 2009 SMIIDEling Overview 17
Metrics
• Ultimate metric: order of magnitude faster with system – allows missions / capabilities that simply weren’t available before.
• # of lines of code automated versus hand written.
• Some measure of complexity of code generated. In other words, abstraction + automation results in less quantity and lower complexity over course of task.
September 21, 2009 SMIIDEling Overview 18
Program Structure
• Individual pieces, with an integrator
• Or entire systems from each team?
September 21, 2009 SMIIDEling Overview 19
Related Research
• Sketch understanding– MIT (Davis), UWash (Landay), UC Riverside
(Stahovich), Illinois (Forbus)
• Dialogue management– Grosz (Harvard), Sidner (BAE)
• DSMLs– Karsai et al. (Vanderbilt), Metacase (Kelly, Tolvanen),
Gray (Alabama), Sprinkle (Arizona). – See dsmforum.org
• Natural Language• Mixed initiative