[ixd] week 08. data gathering
TRANSCRIPT
Lecture 7
Data Gathering
Interaction Design / IID 2017 Spring Class hours : Wednesday 2:00 pm – 5:50 pm Lecture room : International Campus Veritas Hall B308 26th April
DATA GATHERING
Chapter 7
Lecture #7 IID_Interaction Design 2
Objectives
• Discuss how to plan and run a successful data gathering program.
• Enable you to plan and run an interview.
• Enable you to design a simple questionnaire.
• Enable you to plan and carry out an observation.
Lecture #7 IID_Interaction Design 3
Five key issues
• Setting goals
– Decide how to analyze data once collected
• Identifying participants
– Decide who to gather data from
• Relationship with participants
– Clear and professional
– Informed consent when appropriate
• Triangulation
– Look at data from more than one perspective
– Collect more than one type of data, eg qualitative from experiments and qualitative from
interviews
• Pilot studies
– Small trial of main study
Lecture #7 IID_Interaction Design 4
Data recording
• Notes, audio, video, photographs can be used individually or in
combination:
– Notes plus photographs
– Audio plus photographs
– Video
• Different challenges and advantages with each combination
Lecture #7 IID_Interaction Design 5
Interviews
• Unstructured - are not directed by a script. Rich but not replicable.
• Structured - are tightly scripted, often like a questionnaire. Replicable but
may lack richness.
• Semi-structured - guided by a script but interesting issues can be
explored in more depth. Can provide a good balance between richness
and replicability.
• Focus groups – a group interview
Lecture #7 IID_Interaction Design 6
Interview questions
• Two types:
– ‘closed questions’ have a predetermined answer format, e.g.. ‘yes’ or ‘no’
– ‘open questions’ do not have a predetermined format
• Closed questions are easier to analyze
• Avoid:
– Long questions
– Compound sentences - split them into two
– Jargon and language that the interviewee may not understand
– Leading questions that make assumptions e.g.. why do you like …?
– Unconscious biases e.g.. gender stereotypes
Lecture #7 IID_Interaction Design 7
Running the interview
• Introduction – introduce yourself, explain the goals of the interview, reassure about
the ethical issues, ask to record, present the informed consent form.
• Warm-up – make first questions easy and non-threatening.
• Main body – present questions in a logical order
• A cool-off period – include a few easy questions to defuse tension at the end
• Closure – thank interviewee, signal the end,
eg. switch recorder off.
Lecture #7 IID_Interaction Design 8
Enriching the interview process
• Props - devices for prompting interviewee, e.g. use a prototype,
scenario
Lecture #7 IID_Interaction Design 9
Questionnaires
• Questions can be closed or open
• Closed questions are easier to analyze, and may be distributed and
analyzed by computer
• Can be administered to large populations
• Disseminated by paper, email and the web
• Sampling can be a problem when the size of a population is unknown
as is common online evaluation
Lecture #7 IID_Interaction Design 10
Questionnaire design
• The impact of a question can be influenced by question order.
• You may need different versions of the questionnaire for different populations.
• Provide clear instructions on how to complete the questionnaire.
• Strike a balance between using white space and keeping the questionnaire compact.
• Avoid very long questionnaires
• Decide on whether phrases will all be positive, all negative or mixed.
Lecture #7 IID_Interaction Design 11
Question and response format
• ‘Yes’ and ‘No’ checkboxes
• Checkboxes that offer many options
• Rating scales
– Likert scales
– semantic scales
– 3, 5, 7 or more points
• Open-ended responses
Lecture #7 IID_Interaction Design 12
Encouraging a good response
• Make sure purpose of study is clear
• Promise anonymity
• Ensure questionnaire is well designed
• Offer a short version for those who do not have time to complete a
long questionnaire
• If mailed, include a stamped addressed envelope
• Follow-up with emails, phone calls, letters
• Provide an incentive
• 40% response rate is good, 20% is often acceptable
Lecture #7 IID_Interaction Design 13
Advantages of online questionnaires
• Relatively easy and quick to distribute
• Responses are usually received quickly
• No copying and postage costs
• Data can be collected in database for analysis
• Time required for data analysis is reduced
• Errors can be corrected easily
Lecture #7 IID_Interaction Design 14
Example of an online questionnaire
Lecture #7 IID_Interaction Design 15
Figure 7.8 An excerpt from a web-based questionnaire showing check boxes, radio buttons, and pull-down menus
Problems with online questionnaires
• Sampling is problematic if population size is unknown
• Preventing individuals from responding more than once can be a
problem
• Individuals have also been known to change questions in email
questionnaires
Lecture #7 IID_Interaction Design 16
Observation
• Direct observation in the field
– Structuring frameworks
– Degree of participation (insider or outsider)
– Ethnography
• Direct observation in controlled environments
• Indirect observation: tracking users’ activities
– Diaries
– Interaction logging
– Video and photographs collected remotely by drones or other equipment
Lecture #7 IID_Interaction Design 17
Observation
Lecture #7 IID_Interaction Design 18
Figure 7.9 Mars Exploration Rover
Structuring frameworks to guide Observation
• Three easy-to-remember parts:
– The person: Who?
– The place: Where?
– The thing: What?
Lecture #7 IID_Interaction Design 19
Structuring frameworks to guide Observation
• A more detailed framework (Robson, 2014):
– Space: What is the physical space like and how is it laid out?
– Actors: What are the names and relevant details of the people involved?
– Activities: What are the actors doing and why?
– Objects: What physical objects are present, such as furniture
– Acts: What are specific individual actions?
– Events: Is what you observe part of a special event?
– Time: What is the sequence of events?
– Goals: What are the actors trying to accomplish?
– Feelings: What is the mood of the group and of individuals?
Lecture #7 IID_Interaction Design 20
Planning and conducting observation in the field
• Decide on how involved you will be: passive observer to active
participant
• How to gain acceptance
• How to handle sensitive topics, eg. culture, private spaces, etc.
• How to collect the data:
– What data to collect
– What equipment to use
– When to stop observing
Lecture #7 IID_Interaction Design 21
Ethnography (1)
• Ethnography is a philosophy with a set of techniques that include
participant observation and interviews
• Debate about differences between participant observation and
ethnography
• Ethnographers immerse themselves in the culture that they study
• A researcher’s degree of participation can vary along a scale from
‘outside’ to ‘inside’
• Analyzing video and data logs can be time-consuming
• Collections of comments, incidents, and artifacts are made
Lecture #7 IID_Interaction Design 22
Ethnography (2)
• Co-operation of people being observed is required
• Informants are useful
• Data analysis is continuous
• Interpretivist technique
• Questions get refined as understanding grows
• Reports usually contain examples
Lecture #7 IID_Interaction Design 23
Ethnography (3)
Lecture #7 IID_Interaction Design 24
Figure 7.10 (a) The situation before MERboard; (b) A scientist using MERboard to present information Source: J. Trimble, R. Wales and R. Gossweiler (2002): “NASA position paper for the CSCW 2002 workshop on Public, Community and Situated Displays: Merboard”.
Online Ethnography
• Virtual, Online, Netnography
• Online and offline activity
• Interaction online differs from face-to-face
• Virtual worlds have a persistence that physical worlds do not have
• Ethical considerations and presentation of results are different
Lecture #7 IID_Interaction Design 25
Observations and materials that might be collected (Crabtree, 2007)
• Activity or job descriptions.
• Rules and procedures that govern particular activities.
• Descriptions of activities observed.
• Recordings of the talk taking place between parties.
• Informal interviews with participants explaining the detail of observed activities.
• Diagrams of the physical layout, including the position of artifacts.
• Other information collected when observing activities:
– Photographs of artifacts (documents, diagrams, forms, computers, etc.)
– Videos of artifacts.
– Descriptions of artifacts.
– Workflow diagrams showing the sequential order of tasks.
– Process maps showing connections between activities.
Lecture #7 IID_Interaction Design 26
Observation in a controlled environment
• Direct observation
– Think aloud techniques
• Indirect observation – tracking users’ activities
– Diaries
– Interaction logs
– Web analytics
• Video, audio, photos, notes are used to capture data in both types of
observations
Lecture #7 IID_Interaction Design 27
Web analytics
• A system of tools and techniques for optimizing web usage by:
– Measuring,
– Collecting,
– Analyzing, and
– Reporting web data
• Typically focus on the number of web visitors and page views.
Lecture #7 IID_Interaction Design 28
A section of Google analytics dashboard for id-book.com
Lecture #7 IID_Interaction Design 29
Figure 7.14 Segments of the Google Analytics dashboard for id-book.com in September 2014 (a) audience overview, (b) screen resolution of mobile devices used to view the website
A section of Google analytics dashboard for id-book.com
Lecture #7 IID_Interaction Design 30
Figure 7.14 Segments of the Google Analytics dashboard for id-book.com in September 2014 (a) audience overview, (b) screen resolution of mobile devices used to view the website
Choosing and combining techniques
• Depends on the:
– Focus of the study
– Participants involved
– Nature of the technique(s)
– Resources available
– Time available
Lecture #7 IID_Interaction Design 31
Summary
• Data gathering sessions should have clear goals.
• An informed consent may be needed.
• Five key issues of data gathering are: goals, choosing participants, triangulation,
participant relationship, pilot.
• Data may be recorded using handwritten notes, audio or video recording, a camera,
or any combination of these.
• Interviews may be structured, semi-structured or unstructured
• Focus groups are group interviews
• Questionnaires may be on paper, online or telephone
• Observation may be direct or indirect, in the field or in controlled settings.
• Techniques can be combined depending on the study focus, participants, nature of
technique, available resources and time.
Lecture #7 IID_Interaction Design 32
DESIGNING FOR THE DIGITAL AGE: HOW TO CREATE HUMAN-CENTERED PRODUCTS AND SERVICES CHAPTER 9. OTHER SOURCES OF INFORMATION AND INSPIRATION
Workshop #1
Lecture #7 IID_Interaction Design 33
When You Have Less Time
• When possible, try to squeeze in at least one or two days of user
interviews.
– It's true that you risk getting unusual interview participants who could
skew your thinking, but this is rare, and the risk is limited as long as you
compare what you see to what you're hearing from stakeholders.
– Focus on the kinds of users who are most critical to the product's success
or the ones you suspect are the least understood.
– If you're not allowed to recruit users on your own, use the friends-and-
family method
Lecture #7 IID_Interaction Design 34
When You Have Less Time
• A larger interview sample allows for some quantitative analysis of the
data, but is unlikely to result in better design.
• Any existing focus group or survey data may also be useful.
– Don't limit your requests for such information to the product team;
corporate marketing groups frequently conduct research that isn't
disseminated.
– Product managers or professional services staff may also have trip notes
from customer site visits that are worth perusing.
– If possible, review these before the stakeholder meeting so you can use
the data to move things forward.
Lecture #7 IID_Interaction Design 35
When You Have More Time
• A larger interview sample allows for some quantitative analysis of the data, which
can be persuasive or comforting to many people.
– However, a larger sample adds considerable analysis time and is unlikely to result in
better design, except to the extent that it lets you catch an edge case you otherwise
might have missed.
• A long session should include an initial interview using the techniques described
in Chapter 7, followed by some mixture of quiet observation time with opportunities
to ask more questions about what you observe.
– If activities cannot be interrupted, use a small video camera; replay interesting
segments for the interviewee later so you can ask questions about specifics. Consider
whether it's better to have the whole team observing or to have just one person follow
the informant with a video camera; surgeons are accustomed to working with large
audiences, but most office workers are not.
Lecture #7 IID_Interaction Design 36
Supplemental Research Methods
• Public-space observation
Lecture #7 IID_Interaction Design 37
Figure 9.1. If you were designing wayfinding or an information kiosk for an airport, it would be informative to hang out at the information desk for a while.
Exercise
• Would you be comfortable conducting research in the following ways?
Why or why not?
1. You're designing a café it acceptable to buy coffee in a competitive café,
then sit at a table and watch what people do there, how long they stay,
and what seems to encourage them to spend money?
2. You're designing a children's game. Is it acceptable to sit near an
elementary school and watch how children play? How about wandering
over and asking questions of some of the children? What about looking at
their pages on MySpace or another social networking site?
Lecture #7 IID_Interaction Design 38
Exercise
3. You're attempting to improve the patient experience at a hospital. Is it acceptable to sit
in a waiting room observing how people behave there?
4. You're designing a patient Web site for a new diabetes drug. Is it acceptable to sit in an
endocrinologist's waiting room and chat with people about their conditions, not
claiming to have diabetes but also not identifying yourself as a researcher?
• Provided you're comfortable doing so, conduct some informal observations in a
public space for the LocalGuide or RoomFinder. What behaviors do you see in this
context that you did not see (or don't think you would have seen) in an interview?
What did you see that provided an interesting insight?
Lecture #7 IID_Interaction Design 39
Supplemental Research Methods
• Mystery shopper
– A variation on public-space observation is being a "mystery shopper," someone
who tries out the customer experience without being identified as a
researcher.
– Anonymous usage is an important research tool because the story you get
during explicit research may be sanitized; this is why good restaurant
reviewers don't tell the staff who they are.
– This technique is a powerful tool for convincing executives of the need for
change; many become passionate advocates for improvement after seeing their own products or services through a customer's eyes. I'm convinced
that more products and services would be better if more executives, as one of
my clients put it, "ate their own dog food."
Lecture #7 IID_Interaction Design 40
Exercise
• If you work for a company that sells products or provides customer
support, try using the service as an ordinary customer.
1. Alternatively, try using a service you're not familiar with but might
ordinarily use anyway (such as a physical store, online service, or
telephone customer support center).
2. What makes you feel good about the service and the company providing
it?
3. What gives you a negative opinion?
Lecture #7 IID_Interaction Design 41
Supplemental Research Methods
• Diaries
1. In interviews, it can be difficult to get a sense of behavior over time
because you have to rely on the participant's memory of past activities or
circumstances, and artifacts can only do so much to prompt that.
2. One way to widen your view of someone's activities without shadowing
them 24/7 is to ask them to keep a diary.
3. This can be somewhat structured, much like a survey taken several times,
or can be free-form entry guided by a few questions.
Lecture #7 IID_Interaction Design 42
Supplemental Research Methods
4. A diary can take almost any form: written responses to a periodic e-mail
reminder, a handwritten notebook, a narrated video, or photos with
written commentary.
5. Keep in mind that a diary has limitations: Self-reporting error is likely. If
possible, sit down with each participant and conduct a follow-up
interview using the diary as a basis for questions.
Lecture #7 IID_Interaction Design 43
Supplemental Research Methods
• Surveys
– Numbers helpful in disabusing stakeholders of pet beliefs.
– My opinion alone wouldn't have been persuasive, but quantitative data
helped stakeholders see that they'd be limiting the experience of their
most important audience.
Lecture #7 IID_Interaction Design 44
Figure 9.2. An example of a survey with a Likert scale.
Supplemental Research Methods
• Surveys (continues)
– FINDING EXISTING SURVEY DATA
– DEVELOPING YOUR OWN SURVEY
• Step 1: Identify your audience and goals
• Step 2: Craft questions and instructions
• Step 3: Determine your sample size
• Step 4: Decide how to recruit participants
Lecture #7 IID_Interaction Design 45
Supplemental Research Methods
• Step 1: Identify your audience and goals
– First, work with your teammates (and probably the stakeholders) to
determine what you're trying to learn.
– Is this a single survey to understand demographics and attitudes to inform
interview planning?
– Are you trying to assess how many people fit the behavior patterns you've
already observed?
– Or are you trying to assess the impact of your design with a before and
after comparison?
Lecture #7 IID_Interaction Design 46
Supplemental Research Methods
• Step 2: Craft questions and instructions
– Ask only one question at a time. For example, "If you own a camera, who is
the manufacturer?" is really asking two questions. It would be better to ask
whether the participant owns a camera, then ask who the manufacturer is
only if the first answer is yes.
– Be specific. "Do you ever use the Web?" won't really tell you much, and a
respondent who's used it once may not know how to answer. "How often do
you use the Web? Daily / Two or more times a week / Two or more times a
month / Less than twice a month / Never" would be better. Offer quantity or
frequency choices with specific numbers, since terms like "often" and "seldom"
are relative.
Lecture #7 IID_Interaction Design 47
Supplemental Research Methods
– Make the options for any single-answer question mutually exclusive. For example, if
you're asking about income, don't have a $50,000-$60,000 category and a $60,000-$70,000
category, because the person whose salary is $60,000 won't know what to pick; $50,000-
$59,999 is better.
– Make lists as complete as possible. If someone who drives a minivan has to choose
either "car" or "truck," she will either abandon the survey in frustration or be forced to
choose an answer that doesn't really fit, thereby introducing error.
– Allow for participants who can't provide definitive answers. Include options such as
"other," "not applicable," or "I don't know" when possible, so you don't get incorrect
answers or cause people to quit in frustration.
Lecture #7 IID_Interaction Design 48
Supplemental Research Methods
– Avoid negative construction for multiple choice questions. For
example, "which of the following do you not use" is likely to be read as
"Which of the following do you use." If you absolutely can't avoid it,
visually or verbally emphasize the negative word.
– Limit the options in a Likert scale to five. A scale of 1 to 5 lets people
differentiate between good and great, but a scale of 1 to 10 causes
confusion. Having an odd number of choices allows for a neutral answer,
which generally provides a more accurate picture of attitudes. Make sure
the high/low or positive/negative values are always at the same end of the
scale.
Lecture #7 IID_Interaction Design 49
Supplemental Research Methods
• Use both positive and negative phrasing within your sample. When asking if
people strongly agree or strongly disagree whether something is good, help
minimize bias due to a desire to please by using positive phrasing with half the
group and negative phrasing with the other half. For example, if you present a
Likert scale with a statement like, "Product X is affordable," you will get an
artificially high level of agreement. Balance this by phrasing it as "Product X is
expensive" for the other half of the group.
• Vary list sequences. If possible, vary the sequence of items in lists; this helps
minimize any bias due to people picking things at the top of the list and skipping the
rest. Don't randomize the elements in any list that has a natural progression to it,
such as income or age ranges.
Lecture #7 IID_Interaction Design 50
Supplemental Research Methods
• Step 3: Determine your sample size
– https://www.surveymonkey.com/mp/sample-size-calculator/
Lecture #7 IID_Interaction Design 51
Size of population 5% Error 90% Confidence
5% Error 95% Confidence
5% Error 99% Confidence
3% Error 99% Confidence
1,000 214 278 399 648
10,000 265 370 622 1556
100,000 272 383 659 1810
1,000,000 272 384 663 1840
10,000,000 272 384 664 1843
Table 9.1. Sample size examples.
Supplemental Research Methods
• Step 4: Decide how to recruit participants
– Once you've identified how many people you need to recruit, you have to
decide where to find them and how to invite them to participate.
– Many companies recruit users online through invitations on discussion lists,
pop-up invitations on their own Web sites, or e-mail invitations to existing
customers.
– Inviting only your existing customers limits you in two ways. First, it won't tell
you anything about the people you're not reaching. Second, it tends to bias the
results toward favorable responses, since the least happy people have
probably switched to a competitor.
– Inviting only ex-customers provides an unfavorable bias, since they're likely to
be unhappy.
Lecture #7 IID_Interaction Design 52
Supplemental Research Methods
– A mix of current customers, former customers, and those who have never
been customers is the best bet for most studies; proportion depends on the
goal of the survey.
– Inviting people using e-mail, discussion groups, and Web sites is fine if you're
targeting people who spend much time online, but could be missing an
important segment.
– Telephone, advertising, or direct mail may work best in some circumstances.
However, opt-in invitations, such as permanent links on a Web site, may
create a bias toward extreme views, since people who are very happy or very
unhappy are the most likely to seek out those opportunities.
– Opt-out invitations, such as pop-ups and phone calls, minimize this bias.
Lecture #7 IID_Interaction Design 53
Supplemental Research Methods
• Step 5: Decide when and for how long to conduct the survey
– The timing and duration of your survey can have a tremendous impact on your
sample.
– Certain audiences are either more or less available at particular times of year,
days of the week, or times of day.
– If you conduct a telephone survey or in-store study during a weekday, your
results will be skewed toward people who don't work outside the home.
– Even online, some people may be less likely to spend time on a survey during
the work week or during a busy season. Make sure your data collection
window spans enough time to gather data from people with varied habits.
Lecture #7 IID_Interaction Design 54
Exercise
• Imagine that the stakeholders for the LocalGuide (see Chapter 6) are
trying to determine how many of a possible ten million business
travelers are likely to use it.
– What questions would help you understand this?
– Develop a survey.
– How large does the sample need to be, assuming a high degree of
confidence is critical?
– How would you recruit?
Lecture #7 IID_Interaction Design 55
Supplemental Research Methods
• Web analytics and customer support data
– When during the day people are visiting your site or particular pages
– How often people visit (if you use identity cookies)
– How long people stay on the site or particular pages
– What percentage of users makes purchases (if applicable)
– Which types of customers are worth the most revenue
– What percentage doesn't purchase the items in their shopping carts
– Where frequent page errors occur
– What sites are referring people to yours
Lecture #7 IID_Interaction Design 56
Supplemental Research Methods
– What terms people entered into a search engine if it's the referring page
– What people search for on your site
– What operating systems and browsers people are using
– The most typical paths through the site (though cached pages can throw
this off)
Lecture #7 IID_Interaction Design 57
Figure 9.3. Web analytics can help you identify where there might be a problem, but won't explain why the problem occurs.
Supplemental Research Methods
• Focus groups
– A focus group is a facilitated, usually 60- to 90-minute meeting with anywhere from five
to a dozen members of a target market.
– The best use for focus groups is when you have a new product idea and know very little
about the people you think might buy and use it.
– If you do plan to conduct a focus group, begin by defining what you want to learn.
• Industry trends?
• General work processes and relationships among roles?
– Consider whether it's more interesting to get the range of views within a particular set of
people or to see how views differ across roles or perspectives. It's generally a good idea
to conduct two to four similar groups; as with interviews, this helps you see if one group
could be an outlier.
Lecture #7 IID_Interaction Design 58
Supplemental Research Methods
• Card sorting
– Card sorting is best suited to relatively small Web sites with users who
understand most of the content. Sorting will help you see that some users
organize mentally based on one criterion while others start from a
different entry point altogether.
Lecture #7 IID_Interaction Design 59
Figure 9.4. In card sorting, participants arrange and sometimes edit category names to fit their mental models.
Supplemental Research Methods
• Competitive products and services
– Companies that spend more time analyzing their competitors than
understanding their customers are likely to be followers rather than
market leaders. That said, a good designer should spend enough time on
the competitors to understand their vulnerabilities and any opportunities
for differentiation.
• Literature and media
– Fiction is also a way for designers to access the zeitgeist that affects how
people respond to new technology products and new interaction
paradigms.
Lecture #7 IID_Interaction Design 60
Research Design
• Participants Sampling
– Random Sampling, Stratified Sampling, Volunteer Sampling, Opportunity
Sampling
• Research Questions
– IV >> DV
• Measurements
– Performance, Satisfaction.
• Research Methods
– Interviews, Survey, Diary, Observation, Ethnography
• Stimuli Development
– Iterative Development
Lecture #7 IID_Interaction Design 61
Homework
Lecture #7 IID_Interaction Design 62
[Team] Research Design
[Team] Data Gathering
[Team] Data Analysis, Interpretation, and Presentation
1 2 3
Steps - Participants
Sampling - Research Questions - Measurements - Research Methods - Stimuli
Development
Steps - Set a goal - Recruit participants - Conduct data gathering by
- Interviews - Questionnaires - Observations
Outcomes - Raw data
- Transcripts - Voice recordings - Video Recordings
- Coding - Open coding - Axial coding - Selective Coding
- Models and possible quantitative Analysis
- Presentation
Submission Due : 11: 59 pm Sun. 30th April