new approaches to surveying organizations aea january 5 th 2010

1

New Approaches to Surveying Organizations

AEA January 5th 2010

Nick Bloom (Stanford) & John Van Reenen (LSE)

2

Why survey organizations?

Large variation across firms & countries in productivity, which differences in organizations and management may help to explain

But little data on management and organizations

So we have been developing survey techniques to collect more data

3

1. Reducing Respondent (e.g. manager) bias

2. Reducing Interviewer bias

3. Obtaining interviews

4. Evaluating survey responses

Outline

4

Blind surveys - where the responder does not know the purpose of the survey - can reduce bias

• Psychology literature finds respondents like to give answers they believe interviewers want to hear

• For example, Schwartz (1999) asked subjects their views on newspapers stories on mass murderers

– One group was given paper headed “Institute of personality research”. Other group was given paper entitled “Institute of social research”

– First group focused on personality and the second group on social factors behind mass murders

5

Open questions & objective scales reduce bias

Open/Closed questions:• Open questions have no fixed set of responses: “Tell me how

you monitor production”

• Closed questions have a fixed set of responses: “Do you monitor production daily” [Yes/No] – potentially leading

Objective/Subjective scales:• Objective scales are absolute: “How frequently is production

monitored?” [Daily, Weekly, Monthly, Yearly]

• Subjective: “How frequently is production monitored?”, [Very frequently, frequently, average, rarely, very rarely]

6

Asking for examples also helps reduce bias

• In some areas theory may differ from practice, and asking for examples can help to elicit the truth

• One example is practices around firing underperformers:

– Asking “If your firm had a persistently poor performer what would you do?” may not induce an accurate response

– We found following up with a request for an example was informative: “Can you give me a recent example”

7

In Bloom and Van Reenen (2007) used MBA students to survey managers in manufacturing firms about management practices.

Blind:

MBAs told managers they were students wanting to interview them about manufacturing practices. No mention of scoring.

We also did not tell our MBA interviewers in advance about the firm’s performance to further reduce bias (so double blind)

Example of a blind survey, with open questions, subjective scoring and requests for examples

8

Score (1): Measures tracked do not indicate directly if overall business objectives are being met. Certain processes aren’t tracked at all

(3): Most key performance indicators are tracked formally. Tracking is overseen by senior management

(5): Performance is continuously tracked and communicated, both formally and informally, to all staff using a range of visual management tools

Open questions:Questions to score production monitoring (dimension 5/18):• What kind of KPIs would you use for performance tracking?• Who gets to see these and how frequently?• If I were to walk through your factory what KPIs could I see?

Objective scoring:

9

Finally, impossible to eliminate all bias, but you can collect interview characteristics to control for this

• Different seniorities and tenures may respond in different ways (we find senior managers with low tenure are most positive)

→ So collect detailed information on respondents

• Khanemann et al. (2004) report that happiness varies systematically over the day (we find responses are more positive earlier in the day and later in the week)

→ Collect information on interview time and length

• Different interviewers may score tougher or softer (we find significant interviewer fixed-effects)

→ Collect information on interviewers

10

1. Reducing Respondent Bias

2. Reducing Interviewer Bias



Outline

11

Calibrate early, calibrate often

• The downside of open questions with objective scales is the scoring grid requires more interpretation

• So hiring a good team (e.g. MBA types) and calibrating the whole team to the same scoring system is essential

• To do this we:

•Run all interviews from a common location (video)

•Have initial one week training and calibration

•Have ongoing weekly re-calibration sessions

•Rotate interviewers across countries

12





Outline

13

Team organization and incentives matter

• Initially we paid interviewers flat rate salaries, with direct encouragement to run more interviews

• But scheduling and running interviews is hard work so we moved to piece-rate pay with (flat rate pay) supervisors

• Discovered (much like Lazear 2000) that piece-rate pay led to massive improvements in interview productivity

• Also find that team bonuses is helpful in addition to piece-rate pay – generates good internal monitoring by peers

14

Correct interview positioning is essentials

• Most switchboards in the US refuse to connect surveys or market research

• So when approaching firms we simply asked the switchboard “Can I speak to the plant manager please”, and if pushed say, “This is Nick Bloom, I am calling from the LSE in London”.

• If pushed further, state we were doing a “piece of work on manufacturing…” - never using the word “survey” or “research”

15

Different countries require different approaches

• After being connected to managers in Europe and Asia we would use our endorsement letters, which took time to obtain but have been very valuable

• In the US managers don’t care about Government but are generally willing to talk anyway, but getting them is hard

•So our MBAs discovered an effective trick..….

• In all cases during the interview we stated (and upheld):

•Confidential – no information released

•No financials – only management questions

16





Outline

17

Important to have a data validation procedure

• Using double-blind interviews, with open question, objective scoring, calibration and bias controls is good practice

• But, we found some people were still sceptical

• So we used two types of survey validation procedure

18

12

34

5m

anag

eme

nt_1

1 2 3 4 5management_2

Internal validation: re-rater evaluation

1st interview

2nd i

nte

rvie

w

Re-interviewed 222 firms with different interviewers & managers

Firm average scores (over 18 question)

Firm-level correlation of 0.627

19

Iternal validation: different survey instruments

We have also been using different management survey instruments on the same firms as another validation test

• Grous (2009) and Bloom et al. (2009) ran field visits and found strong consistency of field and double-blind telephone data

• EBRD (2009) surveyed 418 of our Bloom and Van Reenen (2007) firms using another survey instrument, with a good match

20

External validation: compare to other data, for example firm performance data

Performance measure

iiihikilimi uxhklMNGy '

ln(capital)

ln(materials)

organization ormanagement data ln(labor)

other controls

• Note – not a causal estimation, but only an association to check survey data is correlated with external data

21

(1) (2) (3) (4) (5)

Dependentvariable

Productivity(% increase)

Profits (ROCE)

5yr Salesgrowth

Share Price (Tobin Q)

Exit

Estimation OLS OLS OLS OLS Probit

Firm sample All All All Quoted All

Management 28.7*** 2.018*** 0.047*** 0.250*** -0.262**

Firms 3469 1994 1883 374 3161

Includes controls for country, with results robust to controls for industry, year, firm-size, firm-age, skills etc.

Significance levels: *** 1%, ** 5%, * 10%.

External validation: compare to other data, for example firm performance data

22

My favourite quotes:

[Male manager speaking to an Australian female interviewer]

Production Manager: “Your accent is really cute and I love the way you talk. Do you fancy meeting up near the factory?”

Interviewer “Sorry, but I’m washing my hair every night for the next month….”

The traditional British Chat-Up

23

Production Manager: “Are you a Brahmin?’

Interviewer “Yes, why do you ask?”

Production manager “And are you married?”

Interviewer “No?”

Production manager “Excellent, excellent, my son is looking for a bride and I think you could be perfect. I must contact your parents to discuss this”

The traditional Indian Chat-Up

MY FAVOURITE QUOTES:

24


Interviewer: “How many production sites do you have abroad?

Manager in Indiana, US: “Well…we have one in Texas…”

Americans on geography

Production Manager: “We’re owned by the Mafia”

Interviewer: “I think that’s the “Other” category……..although I guess I could put you down as an “Italian multinational” ?”

The difficulties of defining ownership in Europe

25


The bizarre

Interviewer: “[long silence]……hello, hello….are you still there….hello”

Production Manager: “…….I’m sorry, I just got distracted by a submarine surfacing in front of my window”

The unbelievable

[Male manager speaking to a female interviewer]

Production Manager: “I would like you to call me “Daddy” when we talk”

[End of interview…]

26

Conclusion

• Using blind surveys, with open questions, subjective scoring grids and calibrated interviews can reduce survey bias

• Given the lack of data on firm organization and management the returns to collecting this type of data is very high

• We are currently working with Lucia Foster and Ron Jarmin (Census Bureau) and Erik Brynjolfsson (MIT) to add an organizations and management module to the ASM

• Aim to create a high quality, large sample, public panel database on organizational and management practices

Future research

27

-6-4

-20

24

labp

1 2 3 4 5management

FINALLY, OUR LARGE SAMPLE ILLUSTRATES THE RISKS OF RELYING ONLY ON CASE STUDIES

Case studies provide invaluable firm-level detail. But the self-selection of these firms combined with variation in management practices means these can be misleading(e.g. Enron, was a case-study favorite). So case studies and large data sets combined are best

Management score

Log

of S

ales

/em

ploy

ee (

$’00

0)

new approaches to surveying organizations aea january 5 th 2010

Documents