rosella gennari- intelligent systems and learning centred design
TRANSCRIPT
LEARNER CENTRED DESIGN
INTELLIGENT SYSTEMS
and
Rosella Gennari
http://www.inf.unibz.it/~gennari
Distinguished Speakers
Oxford Women in CS
Settingtechnology
enhanced learning
Climax
TERENCE case
study
Resolutionreflections
Story outline
Settingtechnology
enhanced learning
Climax
TERENCE case
study
Resolutionreflections
Story outline
TECHNOLOGY ENHANCED LEARNING
Technology Enhanced Learning
(TEL) is the usage of technology
for supporting a learning
experience
Herby we take a narrow view:
intelligent TEL =
Artificial Intelligence (AI)
technology based products
for supporting a learning
experience
TEL 4 LEARNING EXPERIENCE
How can we design
technological products
that supports their users’
learning experience?
LET'S SEE EDUCATORS' VIEWPOINT...
Maria Montessori (1870-1952)
Paraphrasing her words, adequate
tasks that come in a prepared
environment, designed on top of the
learner characteristics can
effectively support the learner’s
learning
was the first Italian woman physician
and educator, best known for
Montessori pedagogy
ousability of technology learning products
opedagogical effectiveness of technology learning products
TEL 4 LEARNING EXPERIENCE
Adequate tasks that come in a prepared
environment designed on top of the learner
characteristics can effectively support learning
HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL
Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
product
final
product
HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL
USABILITY + P. EFFECTIVENESS
Settingtechnology
enhanced learning
Climax
TERENCE case
study
Resolutionreflections
Story outline
TERENCE DESIGN
Based on UCD process diagram (© Tom Wellings)
requirement
specification
designevaluation
plan
models +
prototypes
intermediate
products
final
products
TERENCE was an FP7 TEL project
blending user centred and evidence based design
USABILITY + P. EFFECTIVENESS
THE PROBLEM
‣ TERENCE developed an adaptive learning system (ALS) that, via a learner GUI, recommends poor comprehenders
- its learning material, i.e., books of stories and games
- its learning tasks, i.e., reading and playing
‣ so as to stimulate their reading comprehension
‣More than 10% of primary school children, older than 8, are diagnosed with deep text comprehension problems
‣ They are referred to as poor comprehenders
THE TERENCE WORLD
a d e q u a t e
b o o k o f
s t o r i e s
s i g n i n
a d e q u a t e
s m a r t
g a m e s
r e w a r d
TERENCE INTELLIGENT TEL PRODUCTS
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManager
illustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning
Module
Annotation
Module
Visualisation
Module
game
generation
adaptation to
learners
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
TERENCE DESIGN
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
DATA GATHERING METHODS
‣Data for designing the learning material and tasks were from
- contextual inquiries with
‣ IT & UK diagnosis
‣ as well as IT & USA evidence-based medicine therapy experts
- field studies with educators and primary school children
4510
~ 500
C
H
A
R
A
C
T
E
R
I
S
T
I
C
S
Persona Name: Carol
Age: 8
Classroom: year 4
RC levels: low reading levels
Rural/Urban: urban
Deaf/hearing: deaf
First Language: Italian Sign Language
Cochlear Implantation (if deaf): yes
Degree of hearing loss (if deaf): profound
Motor skills (if deaf): average
Summary of the class
represented by this
persona
A younger deaf girl who is very enthusiastic about using new
technology (such as iPhone and IPad) and who adores her
Nintendo DS. Her reading RC levels are very low, but she reads
together with her parents to learn new words and spelling. She
also likes to do many other things such as drawing, taking care of
her pets and going to the park.
Quote “I really love Mario and Luigi. And I would love to have an iPhone
and an IPad, like my dad.”
Personality Open
Role in classroom Active
Role out of the
classroom
Active
Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry,
Ben 10, Hello Kitty and Mickey Mouse Clubhouse.
Carol sometimes uses the computer, but only to play minilab
games. Her computer is in her bedroom, but her parents don’t
allow her to use it all of the time. She can use it only one hour per
day. Carol’s dad has an iPhone and an IPad, and Carol would
really like to use those as well, but her dad tells her she is a bit
too young. Carol likes watching him with his IPad and iPhone
though.
Carol doesn’t use a mobile phone.
Carol plays games on the computer and on her Nintendo DS.
She plays by herself. She likes the mini-clip games on the
computer, and Mario Kart and brain training games on her DS.
She likes games with non-photorealistic human avatars, and
prefers fantasy avatars to animal avatars.
Socio-Cultural Level of
his/her own family
Medium
School performance Carol has sever reading problems. In her class she is below
average in all activities but drawing, where she feels she can truly
express her intimate feelings.
Homework After school, Carol does her homework together with her mum.
L
I
F
E
S
T
Y
L
E
Outdoors Activities Carol often goes to the park with her mum.
Indoors Activities Games on the DS
Carol reads sometimes. Her mum and dad help her reading in the evening. She likes some of the stories they read together, but mostly, she wants to read because she has to learn new words and spelling.
Carol likes drawing and taking care of her pets.
Her mum often plays with her.
Home activities Carol also likes to help her mum in the kitchen or in the garden.
Sport activities Carol practices no specific sport.
.
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
who is the actor of … ? what does (a main) character do?
when does … happen in relation to a central
event?why does the central event happen?
SMART GAME REQUIREMENTS
What for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
points for each smart
coins for all smartunlocked if read+play
visual feedback
Instructions Questions Motivational Interaction
Choices Choices for learner Fixed event
Solutions Choices that are either correct (c) or wrong (w)
Feedback Interaction Consistency Explanatory Solution
Smart points Proportional to the learner’s ability in the game level
Relaxing points
Constant
Avatar Happy/sad states
Time solution constant interaction constant
Rules States of the system, actions of the learner, constraints
What for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions
Learners should get faster, hence a game has a maximal resolution time
Progress and feedback
Monitor and give learners (1) idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summative
expert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analyticalHMI experts or domain experts
heuristic evaluationformative, summative
expert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
G1. interfaces follow general design guidelines
G2. interfaces support the user’s next step to achieve a task
G3. interfaces provide users with timely feedback
Instructions are not under focus and cannot be easily read
Game question and possible resolutions should be proximally close
Game question and possible resolutions should be proximally close
Evaluation of interfaces
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
problem: 256 stories,
each with ~12 games
Smart game design
how can we automatise the
development of smart games via AI (and, hopefully, be efficient)?
Smart game design
enriched annotations
story
annotations
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManager
illustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning
Module
Annotation
Module
Visualisation
Module
Semi-automated generation
enriched annotations
story
text
text
text
text
annotations
Semi-automated generation
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManager
illustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning
Module
Annotation
Module
Visualisation
Module
text
text
text
textimage
image image image
enriched annotations
story
annotations
Semi-automated generation
games
template visual
text
text
text
textimage
image image image
enriched annotations
story
annotations
Semi-automated generation
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManager
illustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning
Module
Annotation
Module
Visualisation
Module
text
story
text + visual
games
AUTOM. MANUAL AUTOM.
Semi-automated generation
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summative
expert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summative
expert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
LARGE-SCALE STUDY DESIGN
Common design of the intervention with TERENCE:
‣ how: pretest/posttest design, with experimental and control groups
‣ hypothesis: TERENCE improves reading comprehension measured
with standardized text comprehension tests
ControlExperimental
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (2) playing smart
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
‣ Post-test (pedagogical only) for re-assessing text comprehension
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each
-lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
The experimental group is of 344 learners:
‣Avezzano: 270 learners:
- 7-9 years old: 118
- 9-11 years old: 152
‣ Pescina: 74 learners:
- 7-9 years old: 37
- 9-11 years old: 37
‣ They were tested (January-February), stimulated (March-May), and re-tested (may-June)
EXPERIMENTAL GROUP IN IT
Pre-post performances for text comprehension (dependent variable) were as follows:
‣ Pescina:
- pre: 14 poor comprehenders (20.59%)
- post: 6 poor comprehenders (8.82%)
‣ Avezzano:
- pre: 15 poor comprehenders (5.95%)
- post: 2 poor comprehenders (0.79%)
MAIN RESULTS IN IT
Pre
Pescina Avezzano
5,95%
20,59%
‣ Wilcoxon signed-rank test supports that differences are statistically significant
- Pescina: z=-4.904, p<0.0001
- Avezzano: z=-2.266, p=0.0234
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analyticalHMI experts or domain experts
heuristic evaluationformative, summative
expert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXPERT EVALUATION
Experts of pedagogy: 1 coordinator; 9 evaluators
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
How-to:
1. each pair of evaluators read a story, and edited its games
2. the coordinator revised their work
3. a pair of evaluators was blindly assigned revised games, and another the manually created games
Main edit tasks:
(1) creation of missing games (~recall)
(2) revision of games (~precision)
From D4.2 and D4.3 technical annex
overall assessment of generation
text
story
text + visual
games
revision of Automated Reasoning (AR) selection of central events and solutions
revision of Natural Language Processing (NLP) of text
text
text text text
Edit tasks in details
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
From D4.2 and D4.3 technical annex
overall generation
text
story
text + visual
games
AR selection of central events and solutions
revision of NLP text
text
text text text
Analyses of evaluation results
AR selection of central events for games:
>Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one
From D4.2 and D4.3 technical annex
>Implications for AR: none picked up
Automated part evaluation-based re-design
AR selection of plausible solutions:
>Results: out of 140 changes of selection of solutions, the majority was for wrong solutions
- generate a wrong solution from correct one by changing participants, e.g.,
<correct_sentence id="2">The man ran and fell on the ground.
</correct_sentence>
<wrong_sentence id="2wh1">Peter ran and fell on the ground.
</wrong_sentence>
>Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3,
From D4.2 and D4.3 technical annex
Automated part evaluation-based re-design
Overall generation: development times:
>Results for revision time:
- 12’6” per game instance:
↑ 12’8” for time games
↓ 10’6” for who games
>Results for creation time:
- avg. 23” per game instance
text
story
text + visual
games
From D4.2 and D4.3 technical annex
>Implications for AR: the semi-automated development process seems to be promising for optimising development times
Automated part evaluation-based re-design
Game over
1st 2nd 3
Sep. 2011 December 2012 September 2013
Sophie'comes'down'the'steps
He had never been beaten before, since he
only ever raced with kids who were
smaller and slower than him.
He wanted a rematch, so the two boys set
off again. Ben was paddling as fast as he
could, still he didn’t make it to the wall
before Luke. It was completely unfair, he
thought. Luke was so much faster. No
sooner had they climbed out of the water,
than he saw his sister coming down the
steps. She was smiling at Ben and gave
him a playful pat on the shoulder. She also
gave Ben a friendly speech about winners
and losers.
revise selection of
solutions
revise selection of
central event
Requirements+for Description
Dif$iculty*levels Macro*levels*for*learners:
4*entry:*character*games;
4*intermediate:*time*games;
4*top:*causality*games.*Scheduling*of*
reading*and*playing
1st*silent*reading;* 2nd* playing* smart* games;*3rd*playing*
relaxing*games
Constraints*on*
actions
Learners* should* get* faster,* hence* a* game* has* a* maximal*
resolution+time
Progress*and*
feedback
Monitor* and* give* learners* (1)* idea* of* progress,* (2)*
explanatory*feedback,*(3)*recall*their*attention*and*solicit+
them*to*give*a*resolution*(in*time)Representation Production*can*be* impaired* hence*promote* resolution*via*
visual*representation+and+reasoning
Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on
Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event
Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons
Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on
Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level
Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant
Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states
Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant
Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints
data structures
NLP+ AR1 for stories
AR1 for txt games
frameworkAR2 + NLP1 for txt games
AR2 for stories AR3 + NLP2 for txt games
requirements
Settingtechnology
enhanced learning
Climax
TERENCE case
study
Resolutionreflections
Story outline
Till 2007
Cat
egory
Axi
s
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Amsterdam U. and CWI
FBK-irst
Free U. of Bolzano
From 2007
Cat
egory
Axi
s
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Free U. of Bolzano
Possible explanation?
co-designgamification cooperative learning
HOW
WHYengagement design together inclusion
childrendesigners teachersWHO
WHAT
GACOCO
treestree puzzle
Gamification of protocol (tasks, subtasks and types of feedback)
Gamification (competition for cooperation)
MISSIONS
CHALLENGES
REWARDSwell
done!
Acknowledgments to
TERENCE colleagues and schools
Current colleagues and schools
DIARY FOR PRESENT
THE TERENCE BOOK
Settingtechnology
enhanced learning
Climax
TERENCE case
study
Resolutionreflections
Story outline
?