Contributions & Future Work

• Experiential elicitation for interface customization

• Uses SGQs with real users and non-trivial domain

• Provides repeated experience

• Primed+ as an efficient approximation

• Improves appreciation of sequential value of help

• Learn parametric form for U(N,L,Q)

• Quadratic in L and Q?

• Model general utility function

• Occlusion, bloat, disruption, interruption, etc.

• Understand experiential “affordance”

• User expectations in richer domains

Experiential Conceptual

2 hours 30 minutes

Easy to administer Difficult to explain

(outcome mixture, repeated scenario)

Tired easily Not easily tired

Generally consistent Often inconsistent

Preference Elicitation

• Outcomes, O

• Utility function, u: O → Reals

• u(oi) > u(oj) iff oi is preferred to oj

• u(oi) = u(oj) iff indifferent between oi,oj

• o− is best outcome s.t. u(o−) = 1

• o− is worst outcome s.t. u(o−) = 0

• Strength of preferences

• Standard gamble, SG(pr) = [pr,o−; 1-pr,o−]

Toward Experiential Utility Elicitation for Interface CustomizationBowen Hui & Craig Boutilier

university of toronto

Methodological Comparison

Experiential Queries

• U(N,L,Q) → U(interface configurations)

• Experience via task completions

• Simulate pr with k repeated tasks

• Each query involves 2k tasks

• Discretize pr Є [0,.1,.2,…,1.0]

• Treat options as adaptive vs. static system

Decision-Theoretic

Interface Customization

• Automatic customization of intelligent assistance

• Objectives:

• Minimize user effort

• Maximize ease of interaction

• Explain individual preferences

• Optimize sequential tradeoffs

• Test domain:

• Goals: repetitive highlighting tasks in PowerPoint

• Assistance: toolbar suggestions

• Partially observable Markov decision process (POMDP)

Experiential vs. Conceptual

• Conceptual: imagine task completions

• Experiential: carry out task completions

• Controlled highlighting task in PowerPoint

• Sampled from U(N,L,Q)

• o− = N0,L1,Q4

• o− = N1,L10,Q0

• Elicited until small regions (pr ± 0.05)

Improving Experiential Elicitation

• Objective: reduce time (thus, reduce effort)

• Primed: Training session

• Familiarity with interface and help parameters

• Primed+: Training + 5 experiential queries

Special thanks to all the participants in the various studies of this work.

Thanks to NSERC, OGS, PRECARN/IRIS for their support.

Author Contact: DCS – 10 King’s College Road, University of Toronto, Toronto ON M5S 3G4

Email: bowen@cs.utoronto.ca URL: http://www.cs.utoronto.ca/~bowen/

department of computer science

Query Type Question Range of Responses

SGQ(pr,oi) What is pr s.t. SG(pr) = oi ? pr Є [0,1]

Bound(pr,oi) Given pr, is SG(pr) > oi ? Yes/No

Bound(pr,oi) in Theory

• Constraints allow incremental refinement

U

0 2 4

Quality

U

0 2 4

Quality

U

0 2 4

Quality

pr

SG(pr) > oi

outcomes

pr

SG(pr) < oi

monotonicity

Bound(pr,oi) in Practice

• Extremely informative, but…

• Impossible to answer confidently

• SG difficult to interpret

• Difficult to distinguish differences in pr

• Sequential costs/benefits underestimated

pr%

1-pr%

100%

> ?

Structural Results

• Value of non-perfect help (Q2, even Q0)

• Monotonically non-increasing in L

• Monotonically non-decreasing in Q

• Variations in N (user feature)

• Curvature of partial functions

• Non-additive decomposition

Quantitative Comparison

• H0: experiential μ = conceptual μ

• T2 shows significance (p < 0.01)

• Therefore, reject H0

• Component-wise t-tests with independent means

• Experience enables users to perceive value of

automated help in repeated scenarios

Methodological Comparison

Quantitative Comparison

• H10: primed μ = conceptual μ

• H20: primed+ μ = conceptual μ

• T2 shows significance (H1:p < 0.01; H2:p < 0.05)

• Therefore, reject H10 and H20

• Component-wise t-tests with independent means

• Primed+ approaches Experiential

Primed Primed+ Conceptual

30 minutes 60 minutes 30 minutes

Easy to administer Easy to administer Difficult to explain

(outcome mixture, repeated scenario)

Not easily tired Not easily tired Not easily tired

Often inconsistent Experiential queries

primed future responses

Often inconsistent

N0

,L1

0,Q

0-

N0

,L1

0,Q

2-

N0

,L1

0,Q

4-

N0

,L5

,Q0

-

N0

,L5

,Q2

-

N0

,L5

,Q4

-

N0

,L1

,Q0

-

N0

,L1

,Q2

-

N0

,L1

,Q4

-

N1

,L1

0,Q

0-

N1

,L1

0,Q

2-

N1

,L1

0,Q

4-

N1

,L5

,Q0

-

N1

,L5

,Q2

-

N1

,L5

,Q4

-

N1

,L1

,Q0

-

N1

,L1

,Q2

-

N1

,L1

,Q4

-

2.5 -

2.0 -

1.5 -

1.0 -

0.5 -

0 -

-0.5 -

-1.0 -

-1.5 -

-2.0 -

Experiential vs. Conceptual

Primed vs. Conceptual

Primed+ vs. Conceptual

Value and Costs of Help

• Preferences for assistance:

• Toolbar, t

• Highlighting goal, g

• Complexity of goal

• Quality of toolbar, Q(t|g) = max Q(i|g)

• Quality of icon, Q(i|g)

• Neediness, N(g)

• Length, L(t)

• Suggestion utility, U(N,L,Q)

min partial max

Quality

U

1 5 10

Length

U

low high

Neediness

U