paper prototypes and evaluation - ida > home · 2019-09-23 · – paper prototype •...

PaperPrototypes

andEvaluation

MattiasArvola

�1

Today

• Prototyping – fidelity – paper prototype

• Evaluation – Formative vs summative – With or without users – User testing • What to do and not do

�2

Prototyping

• purpose: evaluate our concept by allowing someone to experience it

• Main goal: not spend time/money/resources developing something that doesn’t work (or has deficiencies).

• But we already checked what the users and stakeholders want?

• Evaluate early and often!

�3

Prototyping

• Fast and cheap, built to be discarded! • Low-fidelity vs High-fidelity • How similar is the system to the finished product?

�4

Leveloffidelitygovernsfeedback

• If the prototype shows this is just the general outline, the basic ideas etc – feedback may concern that

• If the prototype shows attention to detail, colors, layout etc – feedback will focus on those details

�5

�8

�9

Aprototypeisconnectedtoatask

• A prototype illustrates the system in a specific context - the test task.

• Content presented in the prototype needs to suit that task

• Never use placeholders in a prototype – if you do, it’s a wireframe and not a prototype!

• The instructions to the user in the test situation needs to contain enough detail on purpose and context to steer the user to use that data.

• But don’t give instructions for how to perform the task

�10

T-prototype

• The parts relevant to the task have several interaction steps.

• The initial view/first screen/top menu has only one interaction step

�11

toplevelinteraction

deepestlevelofinteraction

Paperprototypingtipsandtricks

• Use paper/cardboard (cut to screen size) for background and things that don’t change.

• Use sticky-notes (cut to appropriate size) for transient objects such as pop-up menus, information boxes etc

• Use transparencies and whiteboard pens to allow users to write in a text field or use mouse-over

• Fold-out menus, scrolling content (careful not to reveal too much)

• Physical products: 3D models of Play-doh, clay, styrofoam, cardboard

• Other materials: glue, tack-it, coloured paper and pens

�12

Prototypeintheproject

• Not too much detail • Use whatever materials you have access to • Lesson provides some material • Phone backgrounds available online

�13

Evaluation

Evaluationatdifferentstages

• Evaluation methods vs system representation

�15

Sketches,

scenarios,

wriJenspec

Paper

prototypes

Computer

prototypes

Working

system

InspecOonmethods

User

tesOng

User

tesOng

User

tesOng

InspecOonmethods

InspecOonmethods

InspecOonmethods

FormativevsSummativeevaluation

• Purpose of evaluation – revise, modify, recreate -> formative – describe, classify, grade -> summative

�16

Methodsforevaluation

• Usability inspection - when no user is available use usability expertise – heuristic evaluation – cognitive walkthrough – action analysis – …

• User testing – put the user in front of a prototype (or system), give

them a task to perform. – Collect information: observe, measure, interview etc

�17

HeuristicsusedinHeuristicEvaluation

ex: Nielsen’s 10 heuristics for usability: n Visibility of system status n Match between system and real world n User control and freedom n Concistency and standards n Error prevention n Recognition rather than recall n Flexibility and efficiency of use n Esthetic and minimalist design n Help users recognise, diagnose and recover from errors n Help and documentation

�18

Usertesting

• Put the user in front of your prototype (or running system)

• Give tasks • Animate the prototype • Collect data: Observe, think-aloud, interview… • Analyse the data

• A type of role-play – help your subject stay in role

�19

Usertestingpaperprototype-persons

�20

Think-aloudprotocol

• The user is asked to think aloud while performing the task

• Purpose: tracking reasoning • BUT: complex task->silence. Prompting is sensitive! – What are you thinking? Keep talking. – NOT: can’t you find the search button?

• Test leader may NOT provide assistance!

�21

Usertestingprocedure

• Determine tasks to focus on • Create prototype for the task/s – prototype only the parts needed now

• Determine what data to collect how • Pilot test and modify • Specify, contact and brief subjects • Run test and collect data • Analyse and present results

�22

Determiningtasks

• Goal of evaluation - design goals • Representative tasks. Often based on scenario. • Instructions - give context, goal, describe WHAT, not HOW:

– Find a suitable hotel using the hotel search button – You want a hotel with at least 3 stars, a heated pool and not

too far from the golf course (1km). – Find and start the version of the song ‘Give Peace a Chance’

as it was performed by Paul McCartney on the album ‘Good Evening New York City’?”

• Write out instructions one task per page • Provide for training if necessary • Don’t plan too long sessions (1h).

�23

Createtheprototype

• Focus on the parts necessary for the tasks • also for potential misunderstandings/mistakes • T-prototype • Provide content adapted to the task. Provide detail in

the instructions (on purpose, not actions). Never placeholders!

�24

Testset-up

�25

72

Observingbehaviour-dependsongoal

• Performance – what is the user doing? indications of confusion vs

certainty, task progress – measure task success, time-on-task, no of

interactions • Problems – mistakes, misunderstanding, non-action – task completion, no of problems (type/severity)

• Using think-aloud, interviews, questionnaires etc but also eye-tracking, logging etc

�26

Whattomeasurehow?

• Efficiency – time-on-task – number of interactions/clicks

• Effectiveness – task success, errors … – number of problems – Physiological measures as indicators of stress

(perspiration, heart rate …)

�27

Measuringsatisfaction

• opinion • Interview/Questionnaire • Likert scales, ex:This design is beautiful Stongly agree o o o o o Str0ngly disagree

• Note: even/odd, 5-7 items • User a standardized questionnaire such as SUS or

PSSUQ

�28

Observationsvsinterpretations

• The subject did not use the search button • because (?) the subject

did not find the search button

• because (?) the search button icon was not recognisable

• Use observation protocol to avoid too much writing

• Taking notes on index cards

�29

Thingstolookforwhenobserving

• Problem indicators: – things that prevent or delay task completion – things that lead the user astray – things that cause confusion – choosing the wrong function – misunderstanding content – misunderstanding navigation

�30

Analysingandpresentingyourresults

• Review notes soon after each trial - fill in things you missed noting at the time

• Interpret your data • Calculate statistics if appropriate (note significance) • Draw conclusions (if formative: identify problems) • If appropriate: suggest solutions/redesign and

prioritize • In reports: present observations and interpretation

not just the conclusions.

�31

Expectancymeasures-helpwithpriority

• Before task: Ask subjects the EXPECTED difficulty • After task: Ask subjects how difficult it really was

(immediately after the task) • Using scale from very easy to very difficult • Plot the results in scatter plot

�32

Expectancy-diagram

�33

0

1,75

3,5

5,25

7

0 1,75 3,5 5,25 7

Expected easy

Expected difficult

Was easy

Was difficult

Don’ttouchit!

Mustbefixed! Bigopportunity!

Promoteit!

Presentingnumericaldata

• Diagrams • Significance • using

confidence intervals

�34

Confidenceinterval

• 95% confidence interval means that with 95% probability, the result from all possible users would be within that interval around the result from your subjects.

�35

All possible usersYour subjects

Practicalhintsforyourprojects

• Lesson wed w 41 (9/10) – finish your prototype and pilot test. Paper, board,

glue, sticky-notes, scissors etc will be available. • Focus your prototype on the tasks you have in

scenarios/storyboards/flows • Practice being the computer, delay throws the user

out of the role. • Note: only 1 week between lesson and presentation

(book your subjects in good time) • Tell your participants the design is being tested, not

them. • Don’t explain your design. Don’t defend your design.

�36

Thefinalpresentation

• Start with brief background: your design task, personas and design goals.

• Focus presentation on your design solution (your concept), usability test results and sustainability discussion. Remember the presentation needs to be in english.

• 20-25 minutes presentation, 5-10 minutes questions, feedback.

• Audience: listen and comment on the other groups!

�37

Thefinalreport(totheclient)

• No template because needs to be adapted to the client. • Should at least contain: – A description of the concept (your prototype and

enough description for the client to understand how it works. Scenarios.

– Results of the user test (including test tasks). – Reflection on sustainability of your design

• If you did not show the other design concepts you generated, include them.

• CC to examiner

�38

www.liu.se

Questions?