Testing the prototypes

Testing and data collection from prototypesSelected types of data collection and their featuresInterpreting observation dataA study of a design that used automatic data captureData capture and analysis toolsActivity: Observation 1Activity: Observation 2

1.2.3.4.

Index

5.6.7.

Play slides or use these hyperlinks to navigate to sections.Notes pages show any relevant bibliographic references

1

Diana Laurillard for OLDSMOOC

Testing and collecting data

• Some of the prototyping techniques we’ve looked at provide their own means of collecting learner data on their response to a design – e.g. responses to a ppt visualisation collected in the Notes page in answer to specific questions as in the ‘Storyboarding functionality’ above.

• Ways of collecting data on learner responses depend on the technique being used, and on what you want to find out.

• They can all be a basis for further follow-up with an interview to find out more about learner perceptions and ways of conceptualising the task and what they are trying to learn.

Menu 2

Selected types of data collection Learners’ answers to questions on specific design issues

can collect from several learners at once; elicits their short text-based reflectionsLearners’ outputs on activities based on a design can collect from several learners at once; elicits their perceptions of the task; provides a detailed record for analysisProgram monitoring/recording of learner actionsautomates data collection; may need observation as well to assist interpretation; provides a detailed record for analysisNotes/recording of pairs’ discussionsa researcher can only listen to and record one pair at a time; more natural than one person thinking aloud; useful to record as well; link to program monitoring of learner actionsObservation of learner actionsa researcher can only observe and record one person or group at a time; may need program monitoring or video capture to assist interpretation; provides a detailed record for analysis

Menu

3

Interpreting observation dataInterviews with the student after they have worked through a prototype can be very useful to help with your interpretation of what they were thinking about, or how they interpreted the task or presentation.

Some researchers use a ‘think-aloud’ technique, asking the learner to talk about their reactions as they go. This can be useful, except when the task absorbs the learner’s attention, and then they stop talking.

A good alternative is to use ‘Stimulated recall’ (Bloom1953) • the researcher notes the critical incidents or interesting moments in

the learner’s actions • then returns to that part of the software in the interview and asks

‘what were you thinking at this point?’ • this can often elicit quite detailed accounts of how they were

interpreting the visuals, or the task – or, of course, why they might have been confused or uncertain about what to do.

Menu

4

A study of a design: automatic data capture• Data collection was carried out on 18 students with A-

level Maths or higher• They were observed doing tasks involving interpretations

of prototype mathematical and graphical displays• Each task was presented in either static, dynamic or

interactive forms• Data capture was automatic:

– video and audio of the interaction – real-time capture of work on a tablet– recording of actions on the screen– eye-tracking of eye movements across the screen.

Menu

5

These techniques generate copious data!

Video actions,

gestures, talk

What the learner sees

on screen

Real-time writing on

the Tablet PC

Fixation

Saccade

Allows detailed analysis of user response to interface and interactivityImportant to know what you’re looking for – what’s the hypothesis?

6

Relating talk to action

Participant’s talk: “This is going from minus two…”

When you are involved in different kinds of data capturing techniques, there is some fine-grained observational data that’s not possible to record with just traditional observation. In this figure and the next you can see the eye-movement following the shapes in a particular area, with individual eye movements building up into a coherent picture. This process is evident from the eye tracking data because you can re-construct the saccades.

7

Analysing focus of attention

Interesting that the focus here is much more on the numbers than on the graphs

8

Comparing attention paid to representations

Contrasting the data from different students – it’s important to know what you’re looking for – are these similar because they all focus on numbers, or different because they spend more time (larger circles) on different elements?

9

Data capture and analysis tools

To capture screen activities

To capture eye movements, logged interactions, and for eye-tracking data analysis

To coordinate and analyse the multiple streams of data

Try these if you want to do this kind of data capture and analysis

Menu

10

Typical Activity: Observation 1

Set Goal: To locate a specific documentDescribe Task: Find and open the article XSuggest Tool: A course Moodle site

Observation actions: • take notes of all actions, timings and any talk

• refer back to an interesting moment and ask what they were thinking about at that moment, or why they took that action

• would structured notes help?

Menu

11

Typical Activity: Observation 2

Set Goal: Use an online tool to complete a taskDescribe Task: Add a new page to the wiki, paste in the link to an online resource, and add a short commentSuggest Tool: A course Moodle site

Observation actions: • take notes of all actions, timings and any talk

• refer back to an interesting moment and ask what they were thinking about at that moment, or why they took that action

• would structured notes help?Menu

12