The Interviewer Fallacy:Evidence from 10 years of MBA interviews

Uri Simonsohn Francesca GinoHBS

Photo not necessary

Motivation

• How is a journal editor like a venture capitalist?• Continuous flow of judgments

“random” “daily” subsets.• Research question: Impact of subsetting?

Narrow bracketing+Belief in law of small numbers interviewer fallacy

Definition. Reluctance to create subsets of judgments that differ too much from expected distribution.

Paper in one slide

• Data: 1-5 Rating of MBA interviewees – Handful per day.

• corr[avg(so far), this interview]<0• Ruled out alternatives:

– Contrast effects– Non-random sequence

Data Description

• A business school gave us data• 10 years: N=9,323, k=31

***INTERRUPT THIS TALK TO COMMENT ON ANOTHER PROJECT***

False-Positive (PsychScience2011): “list all your variables”Naysayers: “love to, have too many”Authors of False-Positive: “really?”Uri: “watch me.”

Note: another 22 variables are listed in this page

Other side of that single sheet of paper

Note:The .pdf weighs 13Kb.The Wharton logo from slide 1:

11kb

A hardliner may say: Only reason to choose not to post is to hide information from readers.

Back to this talkData Description

• A business school gave us data• 10 years: N=9,323, k=31*

– Interviews per day M=4.5, SD=1.9– Cluster SE [repeated measures]

• Info on:– Applicant (e.g, GMAT scores, experience, race, gender)– Interviewer identity– Interview: time, date– Ratings (1-5 likert)

• 5 subscores: communication, leader, etc.• Overall score (M=2.9, SD=0.9)

• Would like to analyze like gambler fallacy– HHHHpr(T)↑

• Problem– Non-binary data– Covariates– Different interviewers

Instead:

Scorek,i = OLS(average score so fari , covariates)

k: Interviewee, 1 to N that day.i : Interviewer

Prediction:

<0

(1) (2) (3) (4)

Dependent variable:

Specification BaselineInterviewee

controlsInterviewcontrols

Score (1-5) of written

application

Average interview score -0.116*** -0.110*** -0.105*** -0.088**Given by same interviewer to previous interviewees that day (1-5) (0.038) (0.035) (0.036) (0.035)

GMAT score of applicant (/100) 0.244*** 0.250*** 0.079**(0.036) (0.035) (0.032)

Months of job experience of applicant (/100) 0.324*** 0.319*** 0.254***(0.057) (0.055) (0.055)

Number of interviews by same interviewer that dayTotal -0.000 0.001

(0.012) (0.012)

So far -0.018 -0.010(0.013) (0.014)

Score given by reader of application 0.340***(0.044)

Other controlsMonth*year dummies (k=12*9) Yes Yes Yes Yes

Interviewer dummies (k= 21) Yes Yes Yes Yes

Interviewee gender, race (k=9), age & age-squared No Yes Yes Yes

Interview's time (k=12) & location (k=4) No No Yes Yes

Interview Score(1-5)

Effect Size

• Average interview 1 point higher,• Equivalent to losing:

– 40 GMAT points, or – 30 months of experience.

Alternative Explanations

• Contrast effects

• Non-random sequencing of interviews

Contrast vs. Interviewer Fallacy

Two divergent predictions:1) Same effect on the interview subscores? Explanation Prediction

Contrast: yes, and strongerInt.Fallacy: no, or at least weaker.

Data:- Every one of five subscores: n.s.- Average a-la Robyn Dawes: n.s.- Biggest point estimate, ¼ as big- one is >0

Contrast vs. Interviewer Fallacy

Two divergent predictions:2) Effect as end of day approaches. Explanation Prediction

Contrast: weaker (arguably)Int.Fallacy: stronger (absolutely)

Data:Estimate same regressions for:• last interview of day• 1 interview left• 2 interviews left

Effect of previous interviews as day’s end approaches

Alternative Explanations

• Contrast effects

• Non-random sequencing of interviews

• If better candidates follow bad ones or vice-versa spurious finding.• Can we predict objective quality with average-interview-score-so-far?• Test:

GMAT=OLS(avg.score)Job Experience = OLS(avg.score)

Same table + 2 new columns(1) (2) (3) (4) (5) (6)

Dependent variable:GMAT

(250-800)Experience (in months)

Specification BaselineInterviewee

controlsInterviewcontrols

Score (1-5) of written

application

Same as (3)

Same as (3)

Average interview score -0.116*** -0.110*** -0.105*** -0.088** 0.085 0.251Given by same interviewer to previous interviewees that day (1-5) (0.038) (0.035) (0.036) (0.035) (2.063) (0.959)

GMAT score of applicant (/100) 0.244*** 0.250*** 0.079** -- 1.140**(0.036) (0.035) (0.032) -- (0.495)

Months of job experience of applicant (/100) 0.324*** 0.319*** 0.254*** 10.363** --(0.057) (0.055) (0.055) (4.541) --

Number of interviews by same interviewer that dayTotal -0.000 0.001 0.845 0.504*

(0.012) (0.012) (0.676) (0.290)

So far -0.018 -0.010 -0.461 0.008(0.013) (0.014) (1.275) (0.360)

Score given by reader of application 0.340*** 24.190***(0.044) (1.719)

Other controlsMonth*year dummies (k=12*9) Yes Yes Yes Yes Yes Yes

Interviewer dummies (k= 21) Yes Yes Yes Yes Yes Yes

Interviewee gender, race (k=9), age & age-squared No Yes Yes Yes Yes Yes

Interview's time (k=12) & location (k=4) No No Yes Yes Yes Yes

PLACEBOSInterview Score

(1-5)

Possible Mechanisms

1) Gambler fallacy + confirmation bias

2) Mental Accounting

3) Accountability

A note on the internal validity of non-lab data

• In the lab: hard to study interviewer fallacy• Participants could be learning about

– Scale use– Distribution of underlying stimuli quality

• Some psychological questions are better studied outside the lab.

• This seems likes one of them.