1 usc information sciences institute tailor, spring 07 task learning by instruction: benefits and...
TRANSCRIPT
1USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Task Learning by Instruction:Benefits and Challenges
for Intelligent Interactive Systems
Jim Blythe, Prateek Tandon and Mandar TilluUSC Information Sciences Institute
2USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Overview
Complex tasks in intelligent assistants
Overview of task learning by instruction in Tailor
Integration with other task-related capabilities
Challenges and new approaches in specifying tasks
3USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Handling complex tasks in an intelligent assistant
Assistant
UI
Task models
Domain models
Travel
Contracts
Medication Regime
Vacation Activities
4USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Specialized capabilities for complex tasks
Assistant
UI
Task models
Domain models
ApplicationUser invocationAdvice/ PreferencesExplanationMonitoringTrouble-shootingAutonomyOpportunism/ Synergy….
Plan creationLearning/ InstructabilitySharing tasks and goals
5USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Specialized capabilities for complex tasks
Assistant
UI
Task models
Domain models
ApplicationUser invocationAdvice/ PreferencesExplanationMonitoringTrouble-shootingAutonomyOpportunism/ Synergy….
Plan creationLearning/ InstructabilitySharing tasks and goals
6USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Modifying a procedure with Tailor:“Task Learning by Instruction”
User enters text instruction, e.g. “You don’t need authorization when the cost is below $2000.”
Tailor interprets by
(1) recognizing type of modification,
(2) mapping terms to its task and world ontology,
(3) analyzing the proposed modification.
To purchase equipment: 1. find two bids 2. fill out form 10C 3. request authorization from two managers 4. if authorization granted place order with purchasing
To purchase equipment: 1. find two one bids 2. fill out form 10C 3. if it costs > $500 request authorization from two managers 3½. record purchase on web site 4. if authorization granted place order with purchasing
{defProcedure PurchaseEquip cue: [do: (purchase $obj_id)] precond: (and (user $user) (pSelId $obj $obj_id)) body: [seq: [do: (find_bids $obj $bid)] [context: (use_form $form $obj) do: (complete_form $form $obj)] [select: (and (total_price $bid $price) (< $price 2000)) [do: (get_authorization $obj $user)]]]}
7USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Modifying a procedure with Tailor compared with learning from observation/demonstration
Benefits:
Challenges:
Users directly describe the desired change Users describe steps without taking them Users can ignore implementation details
Map user instruction to procedure knowledge using context (e.g. other intentions)
Navigating a set of complex alternatives Follow-up questions to resolve ambiguity and
help fix introduced problems
[Blythe, IUI 05; AAAI 05]
8USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
User describes a new taske.g. “list hotels within +miles of +meeting”
9USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
The first step
“List each hotel name”
10USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
0:1:(Price $sel’n $p+), ..2: (and (HasDisplay $sel’n $d) (Size $d $s)), ..
Aggregating DP approach to find compositions of tasks
ComputerTotalPrice,ShippingCost, RAM, …
HasDisplay
Size
0:$selection (GPA)“the selected laptop”
0:US (constant)1:
<, >, …
Vendor
0:1:(Vendor $sel’n $v+)
Country
2:(and (Vendor $sel’n $v) (Country $v US))
match
DialCode
User instruction: “you don’t need form B when the seller is in the US”
Tailor finds matching expression: (and (vendor $selection $v) (country $v US))
Shown as: “when the country of the vendor of the selected laptop is the US”
laptop
company
0:1:(HasDisplay $sel’n $d+)
displaycountry
number
Variable and description from GPA
Object from user instruction
Synonym fromWordNet: seller
11USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Recent improvements: extend DP with dynamic grouping of partial matches
Group partial expressions by datatype and set of instruction words matched invariant wrt. propagation and matching
Propagate groups, not individual expressions Exponential reduction in propagations, and in iterations
required to find a match
0:$selection (GPA)“the selected laptop”
Vendor
0:1:(Vendor {lapt: Ø})
laptopcompany
0:US (constant)1:
Country
2:(and {cmpny: seller} (Country ? US))
match
country$selection ε {laptop: Ø}
(Vendor {laptop: Ø}) ε {company: seller}
… ε {country: seller, US}
12USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
The last step and resulting procedure
13USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Overview
Task learning in desktop systems
Overview of task learning by instruction
Integration with other task-related capabilities
Challenges and new approaches in specifying tasks
14USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Challenges highlighted by Tailor prototype
Users unsure of the “next step” More context-dependent menus and wizards Fewer options, although there are several valid alternatives
Many alternative input sets for primitive procedures Full list is overwhelming: be selective and summarize Constraints on inputs of primitive procedures
Procedure hard to understand Combining tasks built by users – inappropriate information Hide substeps Hide parameters Allow drill-down
15USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Task-related capabilities: communication and reasoning
Communicating with user
InvocationExplanationAdvice/ preferences
Reasoning
Learning by instruction/ demo Autonomy
Plan creationMonitoringLearning by observationOpportunismCollaboration
16USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Select modification: Add step
Add input
In order to find available hotels near the meeting with input a meeting:
No steps currently defined
Done
Context-related guidance
Add a step
17USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Describe step to add:
In order to find available hotels near the meeting with input a meeting:
get hotels OK
get hotel chains with frequent flyer miles for American
Optionsget hotel chains with frequent flyer miles for American
get hotel chains with frequent flyer miles for United
get hotel chains with frequent flyer miles for Delta
get hotels from onlineReservationz
Hiding steps and parameters
Hide parameters: focus on distinct
actions
18USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Describe step to add:
In order to find available hotels near the meeting with input a meeting, a street and a zip:
list hotel if distance < 2 OK
get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list
OptionsOptions
Heuristic: show external actions (e.g. info access), hide internal book-keeping
Show steps that determine
information access
Highlight new inputs that are automatically
suggested
19USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Describe step to add:
In order to find available hotels near the meeting with input a meeting, a street and a zip:
list hotel if distance < 2 OK
get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list
OptionsOptions
distance between the street and zip, and the hotel street and zip
options that don’t use the hotel street and zip
options that don’t use new input variables
Show parameters when the user asks to see alternatives
Explore only through the inputs, not how
they are used
Show more detail about steps when the user explores the alternatives
20USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Reduce search space using constraints on inputs
Given 3 locations:
YahooDistance(street1, zip1, street2, zip2): 81 alternatives
Given: ‘location’ = tuple(street, zip) YahooDistance(loc1, loc2): 9 alternatives
Given: output is a distance metric: symmetric, d(X,X) = 0 YahooDistance(loc1, loc2): 3 alternatives
21USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Constraints may be reusedfrom task ontology
Action
Information-generating action
Information-generating action:output includes a distance betweentwo inputs A and B
YahooDistance(A:loc, B:loc) -> distance_miles
Symmetric: only consider one orderingconstraints
Identity: d(A,A) = 0, so ignore B = A.
loc = compound(street, zipcode)
22USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Done
get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list
Output is the list of hotels
Select modification:
In order to find available hotels near the meeting given a meeting, a street and zip:
High-level organization of outputs
Add step before this
Add test
Remove this step
Mark step as output
Change step inputs
Show details
Output is created in list form because the constructor is embedded in an iteration
23USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Done
get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list
Output is the list of hotels
Select modification:
In order to find available hotels near the meeting with input a meeting, a street and zip:
Instead of
Result: simpler description and control
24USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Current work
Creating primitive procedures while creating top-level procedure, not before.
Using examples from previously-build procedures to select steps, bindings and follow-ups
Constraints: representation, input and use
25USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07
Summary
Learning complex activities from instruction. Similar issues in modules that communicate about tasks
Many potential matches for an instruction
Describing learned tasks: find the ‘right’ level of initial detail and allow drill-down
Utilize constraints on possible procedure inputs