questionnaires contain closed questions (attitude scales) and open questions pre- and post...
TRANSCRIPT
Questionnaires
contain closed questions (attitude scales) and open questions
pre- and post questionnaires obtain ratings on an issue before and after an design change
can be used to standardise attitude measurement of single subjects following direct observation
can be used to survey large user groups
Types of rating scales
Can you use the following edit commands?yes no don't know
duplicate
paste
A simple checklist
Multipoint checklist
Rate the usefulness of the duplicate command on the following scale?
very of no useful use
Likert Scale
statement of opinion to which the subject expresses their level of agreement
Computers can simplify complex problems
very much agree slightly neutral slightly disagree strongly
agree agree disagree disagree
Caution!
The help facility in system A is much better than the help facility in system B
very much agree slightly neutral slightly disagree stronglyagree agree disagree
disagree
what does 'strongly disagree' mean?
Semantic differential Scaleuses a series of bi-polar adjectives and obtains
ratings which respect to each
Rate the Beauxarts drawing package on the following dimensions
extremely quite slightly neutral slightly quite extremely easy
difficultclear
confusingfun dreary
Rank Order
Place the following commands in order of usefulness (rank the most useful as 1, the least useful as 4)
paste duplicate group clear
Do and Don'ts with Questionnaire evaluation do be clear about the information you want
to obtain don't risk subjects becoming demotivated don't be lazy do provide specific task reference for
questions don’t assume that responses will be positive do pilot the questionnaire first
9
Planning and logistics of questionnaire design
Quantitative or qualitative? Legal requirements: the Data Protection Act Confidentiality and anonymity Sample size Volunteer respondents Identifying subject areas Determining appropriate length Typical time scale Main components of questionnaires
10
Content of items Avoiding response set Components of attitudes Common types of faulty items
leading questions context effects double barelled questions vague and ambiguous terminology hidden assumptions social desirability
11
Leading questions and context effects
Would you agree that the governments policies on health are unfair?
Item wordings should not contain value judgements
How many pints of beer did you drink last night? Think how the context of the study would
affect the response, say in a survey of young peoples life styles survey of health behaviour and heart disease
12
Double barreled questions
Do you believe the training programme was a good one and effective in teaching you new skills?
avoid questions that involve multiple premises
13
Vague and ambiguous terminology
How often do you clean your teeth? Frequently often infrequently never
what does ‘frequently’ mean? Give quantifiers to ensure all respondents
understand the same thing by the response categories
14
Hidden assumptions, social desirability
When did you last borrow a video tape? Avoid hidden assumptions - what are
these?
Do you ever give to charity? May lead to a positive response as
otherwise something negative about the respondent is being conveyed
15
User diaries Used with early releases of complete systems People use system as part of their normal work and
keep a log of the tasks they have used the system for and whether or not they were successful in using the system
Has the advantage of using real tasks not contrived standard tasks
Requires that the system is capable of supporting enough tasks to be useful to the person doing the evaluation
Requires input from the evaluator to maintain the person’s motivation to keep using the diary
Observation and monitoring usage
User trials direct and indirect observation verbal protocolls Collecting user opinions
User diaries over period of extended use Surveys Software logging
17
User trials - duration Aimed at observing people who are typically of the
intended user group using the interface (often a prototype)
People are usually volunteers – this limits the time available for each trial – serious constraint on what can be done
How much of your time would you give to help someone test a piece of software?
Assume a total trial length of 30 – 45 minutes – this has to include introduction, demonstration, data collection and de-brief
If subjects are paid then longer trials are possible
User Trials: structured tasks One approach is to give subjects a series of
standard tasks to complete using a prototype observe subject completing tasks under
standardised conditions data collection aimed at ensuring that qualitative
descriptions of problems during task completion are captured
Intention is to see whether different people encounter similar problems when using the interface
what problems are likely to arise in data recording?
Standard tasks in user trials structure tasks into incremental difficulty
(easy ones first) have a clear policy on subject becoming
stuck and providing help have a reason for including each task (avoid
unnecessary duplication) ensure (all) functional areas of interface
usage are covered ensure tasks of sufficient complexity are
included
20
Example of standard tasks ‘Find the time of the latest train service leaving
Leicester that I can take next Tuesday to arrive in Dundee before 8.00 pm’
‘Find the cost of a return ticket for 2 adults and 2 children for the journey from Leicester to Bristol with no discounts such as saver or supersaver’
‘Find how many copies of Preece ‘Human-Computer Interaction’ the library currently holds’
Note: each task has a definite end point – the user can provide the answer to the question, which is either correct or incorrect
21
Unstructured user trials Another approach to user trials is to ask the user
to browse through information – more appropriate for web-sites or multimedia presentations
Browsing behaviour is directed by the user’s interest rather than being asked to retrieve a specific piece of information
No guarantee that subject visits all parts of the application or site – how much of the site they visit is often useful information in itself
Requires that subject is actually interested in the application or site
Indirect observation - video enables post-session debriefing 'talk-through'
(post-event protocolls) enables quantitative data to be extracted -
e.g. part task timings serves as a diary and visual record of
problems usually very time consuming to analyse usability laboratories – facilities to administer
standard tasks, record data and analyse these
Verbal protocols
means of enhancing direct observations user articulates what they are thinking during
task completion (think-aloud protocols) but…
doing this can alter normal behaviour subject likely to stop when undertaking complex
cognitive activities user may rationalise behaviour in post-event
protocols get subjects working in pairs - co-discovery can
overcome some of these problems.