effective methods for educational research the value of prototyping, observing, engaging users diana...
TRANSCRIPT
Effective Methods for Educational Research
The value of prototyping, observing, engaging users
Diana Laurillard
Methods for educational software designAn iterative design-test-analyse-redesign approachThe focus is achievement of learning outcomesAlign outcomes – activities – assessment• Discover requirements – design – design tests• Prototype• Observe• Test• Engage• Evaluate
Prototyping
• Sketch the design idea• Design user tasks– Use materials to represent choices and actions– Allow users to construct the choices and actions
• Data collection– 6 target users (min)– User constructions (materials)– Users’ commentary (audio, observation notes)– Video (usually unnecessary)
A paper prototype
Proposed screen design
Proposed actions
A paper prototype
User defined actions
A paper prototype
Observing
• Individual– Observation notes– ‘Stimulated recall’ (Bloom1953) – video or
playback to act as a memory aide – Exercise: ‘what were you thinking at this point?’– Audio + notes
• Pairs – listen to them talking, audio + notes• Recording of screen activities • Video
Data capture and analysis tools
To capture screen activities
To capture eye movements, logged interactions, and for eye-tracking data analysis
To coordinate and analyse the multiple streams of data
User engagement
• Workshops – with frequent discussions• Focus-group - question then facilitated
discussion• Survey – Closed questions + open comment• Construct elicitation interviews to discover
users’ own categories of description
Learning analytics
Measures that correlate with final grade• Total discussion messages posted (+ve)• Total number of online sessions (+ve)• Total time online (+ve)• Time spent on assignments (+ve)• Time spent on assessments (-ve)
(MacFadyen and Dawson, 2010)
LMS tracking those at risk
Ave grade = 72%Ave grade = 63%Ave grade = 57%Ave grade = 44%
(MacFadyen and Dawson, 2010)
Predicting those at risk
Week of course
Aver
age
hour
s on
line
per w
eek
Learning analytics on outcomesPedagogically-driven analytics• Define the ecology of experiences and
resources in which the software is embedded• Design appropriate assessment of outcomes • Use the software tasks to generate tests• Formulate the pre-post tests that will capture
the differences the software should make• Interpret outcomes for re-design
Working groups
• 1. What kinds of problems do you see with methods of prototyping, observing and engaging?
• 2. What might be the advantages of using such methods?
• 3. How might an educational software developer can draw on evidence-based research?