intelligent user interfaces: from machine learning to crowdsourcing

E X C E L L E N C E & E T H I C S I N B U S I N E S S

Jean Vanderdonckt

Louvain School of Management - Université catholique de LouvainPresident of Louvain School of Management Research Institute (ILSM)

Head of Louvain Interaction Lab (LiLab)Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium

[email protected] – http://www.uclouvain.be/jean.vanderdonckt

Intelligent User Interfaces:from Machine Learning to

Crowdsourcing

mailto:[email protected]

http://www.uclouvain.be/jean.vanderdonckt


Place des Doyens, 1 – B-1348 Louvain-la-Neuve, Belgium http://www.lilab.be, http://www.lilab.eu, http://www.lilab.info

Louvain Interaction Laboratory (LILab)

http://www.lilab.be/

http://www.lilab.eu/

http://www.lilab.info/


Who is the speaker?

• Jean VANDERDONCKT– Full Professor Université catholique de Louvain

– Head of Louvain Interaction Lab (1998)

– Scientific coordinator of the UsiXML Consortium

• See more at– Slides: http://www.slideshare.net/jeanvdd

– YouTube: http://www.youtube.com/results?search_query=usixml

– DBLP: http://www.informatik.uni-trier.de/~ley/pers/hd/v/Vanderdonckt:Jean

– Google Scholar: http://scholar.google.com/citations?user=U-FgGrkAAAAJ&hl=fr

– Microsoft Academic: http://academic.research.microsoft.com/Author/495619.aspx

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 3

http://www.slideshare.net/jeanvdd

http://www.youtube.com/results?search_query=usixml

http://www.informatik.uni-trier.de/~ley/pers/hd/v/Vanderdonckt:Jean

http://scholar.google.com/citations?user=U-FgGrkAAAAJ&hl=fr

http://academic.research.microsoft.com/Author/495619.aspx


Agenda

• What is an Intelligent User Interface?

• Model-based design of UI: first generation of IUIs

• Second generation of IUIs

• Third generation of IUIs

• Conclusion



Agenda

• What is an Intelligent User Interface?– 1.1 Definitions over time

– 1.2 IUI coverage

– 1.3 Software architecture

• Model-based design of UI: First generation of IUIs



• Conclusion



1.1 What is an Intelligent User Interface?

• Probably first definition: – An IUI is a user interface containing means aimed at minimizing

the cognitive distance between the user’s mental model and the way a task is presented to this end user

• Other candidates: any UI that– is adaptive (not adaptable)

– provides some interactive support

– accesses functionality and knowledge representation

– supports cooperative problem-solving

– What is « intelligent » is disputable• Focus on adaptation

• How to adapt what according to which user’s parameters

• Focus on knowledge

• What type of knowledge and how to exploit it

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 6[Hancock & Chignell 89]



• Second definition: – An IUI should be knowledge-based and modular, infer and

evaluate the user's goals, and adapt its behavior to users and their tasks

– What is « intelligent » is disputable• What the user wants to do / what the user does not want to do

• « Using information in an appropriate manner »

• Anything that supports a human interacting with a systemin order to carry out a task in a way that is as close aspossible as the human way

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 7[Sullivan & Tyler 91]



• Third definition– Intelligent user interfaces (IUIs) are human-machine interfaces

that aim to improve the efficiency, effectiveness, and naturalness of human-machine interaction by representing, reasoning, and acting on models of the user, domain, task, discourse, and media (e.g., graphics, natural language, gesture)

– IUI is interdisciplinary by nature• Human-Computer Interaction (HCI)

• Artificial Intelligence (AI)

• Ergonomics

• Cognitive psychology

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 8[Maybury & Whalster 90]


1.2 IUI Coverage

• More recent and broader definition: – An IUI is any interface that results from the application of AI

techniques to HCI problems

– Artificial intelligence techniques• Simple techniques: decision tree, decision matrix, optimisation problem,…

• Advanced techniques: Bayesian networks, neuronal networks,…

• More advanced techniques: recommendation algorithms, machine learning, reinforcement learning, relevance feedback

• Engineering Interactive Computing Systems (EICS) is an ACM Community

• Intelligent User Interfaces (IUI) is an ACMCommunity

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 9[Kolski et al. 14]


1.2 IUI Coverage

• IUI = HCI AI & EICS = HCI SE


AI HCI

SE

IUI

EICS


1.3 Software architecture

• A simple IUI software architecture


User interface

Control

Semantic core

User interface

Control

Intelligent semantic core

IntelligentUser interface

Control

Semantic core

Knowledge base

Interactive system Intelligent system System with IUI


Agenda


• Model-based design of UI: first generation of IUIs– 2.1 Model-based design of user interface: two sub-problems

– 2.2 Solving the problems with simple techniques



• Conclusion



2.1 Model-based User Interface Design

• MB-UID is a typical area where « intelligence » is desired sincedesigning a UI is a typically complex knowledge-based problem

• Some particular sub-problems

– Widget selection: how to select the most usable widgets depending on the user’s context (i.e., user, platform, and environment) and the user’s task

– GUI Layout: how to arrange widgets in a way that is usable and meaningful for the user



2.2 Solving the problem with simple techniques

• Widget selection = expert system with rule-based inference

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 14[Vanderdonckt & Bodart 93]



• Widget selection = {rules}

• Pros: simple format, justification

• Cons: no meta-rules, no flexibility




• Widget selection = expert system with rule-based inference




• Widget selection = a decision tree where– each node consists of a design question: e.g., how many possible

values?

– each branch represents one possible design option for each design question: e.g., simple choice vs multiple choice

– design knowledge is encoded as a full decision tree

• Selection could be– Manual

– Automated

– Semi-automated (mixed-initiative)




• Widget selection = decision tree with manual parsing


Selection tree for alphanumeric input/output information

Number of value to choose = 1

Domain definition = unknown

Length <= Lm : profiled uni-linear edit box

Length >Lm : multi-linear edit box

Domain definition = known

Number of possible values [2,3]

Density = low : radio button

Density = high : option box

Number of possible values [4,Nmag]

Density = low : radio button + group box

Density = high : option box

Number of possible values [Nmag,Tm]

Density = low : list box

Density = high : drop-down list box

Number of possible values > Tm

Density = low : scrolling list box

Density = high : drop-down scrolling list box

[Vanderdonckt & Bodart 93]



• Widget selection = decision tree with manual parsing




• Widget selection asa decision matrix




• Widget selection: as a non-sparse decision matrix




• GUI Layout: as a dedicated algorithm


Date of the day :

Name :

Firstname :

Birthdate :

Complete address :Phone number :

Sex :

Civil status :

Organization Code :

Identification number :Affiliation type :

Medicine man :

Service :

Room type :

Regimen :Message :

Male Female

Unmarried Married Widowed Divorced

Single room Two beds room Four beds room

Ok Cancel

[Weisbecker et al. 93]



• GUI Layout: as a dedicated algorithm– to use underlying grid for screen format

– to consistently use this format

– to foster balance, symmetry

– to use math. Relationships (expressed as constraints)

– to use rules for labels, controls

– to address grouping, dissociation

– to preserve visualcontinuity

– to systematicallyderive placement


Objets Interactifs Objets Interactifs

Titre Bouton

Bouton




• GUI Layout: as a dedicated algorithm


Titre

m(H+I/2)

I

h1

l1

l2n1 objets

n2 objets

h2

I/2+I/2+1

H

2L

ltot

Zone message

h1

h1tot

h2tot

hn1tot

objet1

objet2

objetn1

I

I

I

= interligne blanc

Gauche Centré Droit

Alignement

loiai

hoiai

lliH H/2

Normal

Style :

loiai

I/2

H

lli L

hoiai

linvi

Inclure les notes H

lliloiaiL

hoiai

A B C

D

E F G

H

Alignement :

même taille

Orientation :

dessus-dessous

Placement:

bas-droite

Alignement :

même taille

Orientation :

dessus-dessous

Placement :

centre-droite

Alignement :

même taille

Orientation :

dessus-dessous

Placement :

centre-gauche

Alignement :

même taille

Orientation :

gauche-droite

Placement :

distribué-bas

Alignement :

même taille

Orientation :

gauche-droite

Placement :

bas-droite

Alignement :

même taille

Orientation :

gauche-droite

Placement :

bas-centre

Alignement :

même taille

Orientation :

gauche-droite

Placement :

haut-centre

Alignement :

même taille

Orientation :

dessus-dessous

Placement :

centre-droite




• GUI Layout: as a Constraint Satisfaction Problem (CSP)




• GUI layout: as an optimization problem: AIDE (Sears)


Patient

Civil status

Sex

Date of day :

Male

Female

Unmarried

Married

Windowed

Divorced

Name :

Firstname :

Birthdate :

Complete Address :

Phone number :

Organization code :

Identification number :

Affiliation type :

Medecine man :

Single room

Two beds room

Four beds room

Regimen :

Ok

Cancel

Admission

Service :

Room type

Start

Patient

Civil status

Sex

Date of day :

Male

Female

Unmarried

Married

Windowed

Divorced

Name :

Firstname :

Birthdate :

Complete Address :

Phone number :

Organization code :


Affiliation type :

Medecine man :

Single room

Two beds room

Four beds room

Regimen :

Ok

Cancel

Admission

Service :

Room type

Start

Patient

Civil status :

Sex :

Date of day :

Female Male

Unmarried Married

Widowed Divorced

Name :

Firstname :

Birthdate :

Complete Address :

Phone number :

Organization code :


Affiliation type :

Medecine man :

Two beds Four bedsRegimen :

Ok

Cancel

Service :

Room type :

Organization

Medical care

Single

cost

[Sears 93]



• GUI Layout: as state-space search in a binary tree


CIO1

CIO2

Same height CIO1 CIO2

Height(CIO2) < Height(CIO1)CIO2

CIO1

Height(CIO2) > Height(CIO1)

if label

if notCIO2CIO1

unused spaceCIO1

more

CIO2CIO1

CIO2CIO1no

CIO2

right

bottom CIO1

CIO2






OIC1

OIC1 OIC2

OIC1 OIC2 OIC3

OIC1 OIC2 OIC3 OIC4

OIC1 OIC2 OIC3 OIC4 OIC5

OIC1 OIC2 OIC3 OIC4

OIC5

OIC1 OIC2 OIC3

OIC4

OIC1 OIC2 OIC3

OIC4 OIC5

OIC1 OIC2

OIC3

OIC1 OIC2

OIC3

OIC4

OIC1 OIC2

OIC3

OIC4

OIC1 OIC2 OIC3

OIC4

OIC5

OIC1 OIC2

OIC3

OIC4 OIC5

OIC1 OIC2

OIC3

OIC4

OIC5

OIC1 OIC2

OIC3

OIC4 OIC5

OIC1 OIC2

OIC3

OIC4

OIC5

OIC1

OIC2

OIC1

OIC2 OIC3

OIC1

OIC2

OIC3

OIC1

OIC2

OIC3

OIC4

OIC1

OIC2

OIC3

OIC4

OIC1

OIC2

OIC3

OIC4

OIC5

OIC1

OIC2

OIC3

OIC4 OIC5

OIC1

OIC2

OIC3

OIC4OIC5

OIC5

OIC1

OIC2

OIC3

OIC4

OIC1

OIC2 OIC3

OIC4

OIC1

OIC2 OIC3

OIC4

OIC1

OIC2 OIC3

OIC4 OIC5

OIC5

OIC1

OIC2 OIC3

OIC4

OIC5

OIC1

OIC2 OIC3

OIC4

OIC1

OIC2 OIC3

OIC4OIC5

(1)

(1)

(1)

(1)

(1)

(1)

(2)

(1)

(1)

(1)

(1)

(1)

(1)

(1)

(1)

(2)

(2)

(2)

(2)

(2)

(2)

(2)

(2)

(2)

(2)(1)

(2)

(2)

2.2.2.2

2.2.2.1

2.2.1.2

2.2.1.1

2.1.2.2

2.1.2.1

2.1.1.1

2.1.1.2

1.2.2.2

1.2.2.11.1.2.1

1.2.2.2

1.1.2.2

1.1.2.1

1.1.1.1

1.1.1.2




• GUI Layout: as search in a binary tree with history




• As an optimization problem: minimize screen space without any constraint


Les X similaires et les Y similaires Les X différents et les Y différents

Xmax >= Ymax Xmax < Ymax

Xmax >= Ymax Xmax < Ymax Xmax >= Ymax Xmax < Ymax

Xmax >= Ymax Xmax < Ymax

Les X similaires et les Y différents Les X différents et les Y similaires

Orientation verticale :

le plus long X au-dessus

Orientation horizontale :

le plus long Y à gauche







Orientation horizontal :






espacerestant

[Kim & Foley 93]



• As an optimization problem: minimize screen space without any constraint


ADMISSION

Service : (31)

Regimen : (30)

Organization code : (29)

Identification number : (29)

Affiliation type : (26)

Date of day : (24)

single room (25)

tw o beds room (21)

four beds room (21)

Room type

Phone number : (25)

unmarried (16)

married (10)

w indow ed(9)

divorced (12)

Civil Status

Firstname : (30)

Complete address : (41)

Birthdate : (29)

Name : (27)

Patient

female (12)

male (13)

Sex

Medecine man : (17)

MSG-Information

OK Cancel

DON

[Kim & Foley 93]



• GUI Layout cost: as a function of task/user involvement


Coût relatif

(0,0) sans inplication Avec implication

faible à modérée

Avec implication

élevée

Avec implication

très élevée

Degré d'implication de

la tâche interactive

Tullis Streveler & Wasserman

GalitzAxiomes de Perlman

Avrahami

Stratégies anonymesTarlin

Marcus Feiner

KimGraf

Sears

GOMS

Analyse actionnelle Parcours cognitifs

Evaluation

Conception et évaluation


Agenda






3. Second generation

• SURF: Adapting selection rules

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 37[Eisenstein & Puerta 01]



• GUI Layout: prediction based on task conditional probability of past executions

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 38[Mezhoudi 13]



• Based on task model


OrderIndepProbability(action)

= maxs ∈ simulated sequences (a,h) ∈ s

)P(a|h



• Predicting task path


Content ScorePrediction(actions)

= a

actions

OrderIndepProbability(a) ∗ u. b. p. feature weight



• SUPPLE: layout considered as an optimization problem of a cost function

Invited Talk IHMIA'2015 (TeleCom Paris, Paris, March 20th, 2015) 42[Gajos & Weld 04]


Agenda







4. Third generation

• WEASEL: widget selection as a machine learning problem



4. Third generation



Collecting feedback

by crowdsourcing


4. Third generation



Introducing new adaptation


4. Third generation


• Scoring function– Principle: for each design option, display the widget having the

highest score computed by the scoring function

– Recalculate for each usage and input

• A recommandation is characterized by– A profile vector consisting of features (e.g., values, screen estate)

– A similarity function and a prediction function

• Who can play?– The end user, but not all users are equal

– The designer, but not all designers are equal

– The system administrator, but not always appropriate

– Therefore, the scoring function should be parametrizable



4. Third generation

• Ultimate scoring function:S = f(user, task, platform, environment)where– each model is represented by local functions

– user = fu (experience, frequency,

– task = ft (complexity, frequency, criticity)

– platform = fp (screen resolution, screen size, inter. capabilities)

– environment = fe (light, noise, stress)



5. Conclusion

• First generation is over– Incorporating knowledge is infinite and useless

– Little or no designer control

– Almost no user feedback

• Second generation– More flexibility, sophistication

– Better results

– But still limited involvement of users, designers

• Third generation– Mixed-initiative

– But balancing who can contribute to what (by parametercalibration) is key



5. Conclusion

• Third generation– Mixed-initiative

– But balancing who can contribute to what (by parametercalibration) is key


automatization

Degree

of invol-

vement

user

system


Thank you very much for your attention!


intelligent user interfaces: from machine learning to crowdsourcing

Science

crowdsourcinge x c e

intelligent user interfaces

talk ihmia2015 telecom

belgium http

human way

users parametersfocus

users goals

users mental model