![Page 1: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/1.jpg)
“Yours is better!”Participant Response Bias in HCI
Nicola Dell1
Vidya Vaidyanathan2
Indrani Medhi3
Edward Cutrell3
William Thies3
1University of Washington2San Jose State University3Microsoft Research India
![Page 2: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/2.jpg)
HCI is expanding and diversifying
2
![Page 3: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/3.jpg)
HCI is expanding and diversifying
There are large differences between the people being studied and those studying them 3
![Page 4: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/4.jpg)
Demand characteristics are aspects of a study that convey the investigator’s
hypothesis to participants
4
![Page 5: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/5.jpg)
Demand characteristics may result in participant response bias
5
![Page 6: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/6.jpg)
Demand characteristics may result in participant response bias
6
![Page 7: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/7.jpg)
Demand characteristics may result in participant response bias
Response bias can influence the outcome of studies with human participants
7
![Page 8: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/8.jpg)
What about demand characteristics in HCI?
8
![Page 9: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/9.jpg)
How do social and demographic factors influence demand
characteristics?
9
![Page 10: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/10.jpg)
Let’s design an experiment to explore demand characteristics in HCI
10
![Page 11: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/11.jpg)
Let’s design an experiment to explore demand characteristics in HCI
11
Completely identical video clips
![Page 12: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/12.jpg)
Observe participants interacting with technological devices
12
![Page 13: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/13.jpg)
Interview Script“Thank you for participating in my experiment. I am a computer scientist and I'm trying to improve video players on mobile phones. I want you to watch a short video on these two phones and tell me which one looks better, or if they look the same. The same video will play on both phones, but this phone uses my new player [indicate phone]. Please tell me your honest opinion and please concentrate because I will play each video only once. Do you have any questions? Ok, watch this one first. This one uses my new player [play video]. Now watch this one [play video]. Which one do you think looks better or do they look the same? Why? Thanks very much!”
13
![Page 14: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/14.jpg)
Two different interviewers
Foreign interviewer and local interviewer
14
![Page 15: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/15.jpg)
Participants: Auto rickshaw drivers
250 male participants aged 19 to 72 (M=37, SD=11.2)15
![Page 16: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/16.jpg)
Conditions tested with the identical video clip
Rickshaw Drivers
Foreign Interviewer (+ translator) 50 participants
LocalInterviewer 50 participants
16
![Page 17: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/17.jpg)
Participant Responses
17
0
10
20
30
40
50
35
7 8Num
ber
of p
eopl
e
Foreign Interviewer
Preferred interviewer’s videoPreferred other videoThought videos looked the same
![Page 18: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/18.jpg)
Participant Responses
18
0
10
20
30
40
50
19
9
22
0
10
20
30
40
50
35
7 8Num
ber
of p
eopl
e
Foreign Interviewer Local Interviewer
Num
ber
of p
eopl
e
Preferred interviewer’s videoPreferred other videoThought videos looked the same
![Page 19: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/19.jpg)
Demand characteristics can result in substantial participant response bias
19
![Page 20: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/20.jpg)
Demand characteristics can result in substantial participant response bias
Social and demographic factors influence the amount of response bias
20
![Page 21: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/21.jpg)
21
What happens if the two clips are different?
![Page 22: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/22.jpg)
What happens if the two clips are different?
21
![Page 23: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/23.jpg)
What happens if the two clips are different?
21
![Page 24: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/24.jpg)
What happens if the two clips are different?
21Interviewer always associated to the low-quality clip
![Page 25: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/25.jpg)
Conditions tested with the degraded video clip
Rickshaw Drivers
Without association 50 participants
Foreign Interviewer (+ translator) 50 participants
LocalInterviewer 50 participants
22
![Page 26: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/26.jpg)
0
10
20
30
40
50
4
38
8
Preferred low-quality videoPreferred high-quality videoThought videos looked the same
Participant Responses
23
Num
ber
of p
eopl
e
Without association
![Page 27: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/27.jpg)
Preferred low-quality videoPreferred high-quality videoThought videos looked the same
Participant Responses
23
Num
ber
of p
eopl
e
0
10
20
30
40
50
27
19
4
Foreign Interviewer
Num
ber
of p
eopl
e
0
10
20
30
40
50
4
38
8
Without association0
10
20
30
40
50
18
29
3
![Page 28: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/28.jpg)
The characteristics of the interviewer can determine the outcome of the study
24
![Page 29: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/29.jpg)
Explaining participant responses
25
![Page 30: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/30.jpg)
Explaining participant responses
“The quality of the background color and figures is too light in that player while the quality of the color
and graphics is better in [your] one. Will [your] new player be introduced in the market?”
26
![Page 31: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/31.jpg)
Explaining participant responses
“I prefer this one (associated) because the other one wasn't clear at all. This one was fully clear. I
can definitely show you the difference if we watch them together. I've given you my honest opinion, so please don't be cross with me if it wasn't the
right one.”
27
![Page 32: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/32.jpg)
Detailed and insightful participant comments can’t be taken at face value
28
![Page 33: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/33.jpg)
What about other participant groups?
200 male university students aged 19 to 41 (M=25, SD=3.8)
29
![Page 34: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/34.jpg)
Responses with university students
30
• Identical videos: 2:1 response bias
![Page 35: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/35.jpg)
Responses with university students
30
• Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference
![Page 36: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/36.jpg)
Responses with university students
30
• Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference
• “I feel that in the newer version which you have coded, whenever there was a significant color contrast between two parts of an image, your version was somewhat smoother and less pixelated.”
![Page 37: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/37.jpg)
There is still substantial response bias in studies with peers
32
![Page 38: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/38.jpg)
Limitations
33
![Page 39: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/39.jpg)
Limitations
• Presence of a translator for some interviews
34
![Page 40: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/40.jpg)
Limitations
• Presence of a translator for some interviews• The effects of social status and ethnicity will
vary from culture to culture
35
![Page 41: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/41.jpg)
Limitations
• Presence of a translator for some interviews• The effects of social status and ethnicity will
vary from culture to culture • Many other aspects of demand characteristics
36
![Page 42: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/42.jpg)
What can we do to minimize demand characteristics in HCI?
37
![Page 43: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/43.jpg)
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies
38
![Page 44: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/44.jpg)
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution
39
![Page 45: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/45.jpg)
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution • Minimize the differences in social status
between investigators and participants
40
![Page 46: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/46.jpg)
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution • Minimize the differences in social status
between investigators and participants• Use triangulation to validate data collected
41
![Page 47: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/47.jpg)
Demand characteristics influence the outcome of studies with human participants
42
![Page 48: “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of](https://reader035.vdocuments.us/reader035/viewer/2022081519/56649d9f5503460f94a89815/html5/thumbnails/48.jpg)
“Yours is better!”Participant Response Bias in HCI
Nicola Dell1
Vidya Vaidyanathan2
Indrani Medhi3
Edward Cutrell3
William Thies3
1University of Washington2San Jose State University3Microsoft Research India
[email protected] http://cs.washington.edu/homes/nixdell