the data consultant as archaeologist

The Data Consultant as Archaeologist

Digging for Meaning in World War II Era U.S. Public Opinion Surveys

The Project

A historian of the twentieth century United States wants to incorporate public opinion survey data into research on the growth of the national state in the eras of the Great Depression and Second World War

The Sources

• Largely unmined U.S. public opinion polls from the 1930s and ’40s– The pioneering age of scientific survey

research– Survey takers included Gallup (AIPO), NORC,

OPOR (Hadley Cantril), Roper

The Documentation

• Quality varied, but at its best far short of modern metadata standards– Typically scans of questionnaires with

handwritten notes– Some typed or handwritten code lists for

response options not directly listed on the questionnaire; missing for some variables

– Other random bits of documentation, such as frequencies, interviewer instructions

Gallup example

Code List for Question with Many Possible Responses

Case 1: Economic Class “N”?

Roper/Fortune Survey, January 1939:

(Unknown variable)

Roper/Fortune Survey, January 1940:

Confirming N=“Negro”

• Professor’s “informed guess” was that N represented “Negro”, i.e., black, respondents

• Strategy for confirming was to compare code Ns with remainder of sample (codes A-D, presumably white), on documented variables known to differ sharply by race in 1939

Test 1: OccupationOccupation White? Black? Total

Professional 6% 4% 5%

Proprietor-Farm 9% 1% 9%

Proprietor-Other 11% 3% 10%

Housekeeper 37% 25% 36%

Salaried-Minor 13% 4% 12%

Salaried-Executive 5% 1% 5%

Wages-Factory 2% 6% 3%

Wages-Farm 3% 16% 4%

Wages-Other 8% 28% 10%

Retired 2% 1% 2%

Unemployed 3% 8% 4%

Student 1% 1% 1%

Number of cases 4,666 510 5,176

Test 2: Feelings Toward Roosevelt

Feelings toward Roosevelt White? Black? Total

Roosevelt, or a man like Roosevelt, as President, is essential for the good of the country.

15% 46% 18%

Roosevelt may have made mistakes, but the good he has done definitely outweighs the bad.

42% 35% 41%

Roosevelt may have done many things that needed doing, but he has made so many mistakes that his usefulness is now over.

24% 8% 23%

It is almost a calamity for this country that we must have two more years of Roosevelt.

13% 2% 12%

Don't know 5% 9% 6%


Test 3: Voter Turnout

Voted in 1936 White? Black? Total

Yes 80% 36% 76%

No 20% 64% 24%


Case 2: What Church?

• 1941 OPOR survey• Sample: Residents of Pittsburgh,

Pennsylvania• One question of interest: What church do

you belong to, i.e, What is your religion?• Answers had been coded

alphanumerically, but what church corresponded to which code was on documentation that had been lost

The documentation

The data

-> tab q13a (Are you a member of a church?) | Freq. Percent Cum. ------------+----------------------------------- 1 | 1,055 89.18 89.18 2 | 125 10.57 99.75 4 | 2 0.17 99.92 5 | 1 0.08 100.00 ------------+----------------------------------- Total | 1,183 100.00

-> tab q13b (If "yes" on a, ask: Which church?) | Freq. Percent Cum.------------+----------------------------------- | 155 12.88 12.88 & | 8 0.67 13.55 0 | 141 11.72 25.27 1 | 669 55.61 80.88 2 | 20 1.66 82.54 3 | 29 2.41 84.95 4 | 87 7.23 92.19 5 | 27 2.24 94.43 6 | 49 4.07 98.50 7 | 9 0.75 99.25 8 | 1 0.08 99.33 9 | 8 0.67 100.00------------+----------------------------------- Total | 1,203 100.00

Church member? Which church? Yes No Total 13 124 137

& 8 0 8 0 141 0 141 1 667 0 667 2 20 0 20 3 28 0 28 4 86 0 86 5 26 0 26 6 48 1 49 7 9 0 9 8 1 0 1 9 8 0 8

Total 1,055 125 1,180

-> Which church? = 1

Father born in Number %

U.S. 153 23%

Italy 123 19%

Poland 112 17%

Ireland 66 10%

Germany 39 6%

Austria 38 6%

Lithuania 27 4%

Czechoslovakia 27 4%

Hungary 25 4%

Syria 13 2%

England 9 1%

France 9 1%

Which church? 2nd generation native born

32%

& 38%

0 30%

1 20%

2 70%

3 76%

4 10%

5 52%

6 71%

7 78%

8 0%

9 25%

Total 27%


Father born in Number %

Russia 36 43%

Poland 13 15%

U.S. 9 11%

Rumania 9 11%

Austria 5 6%

Hungary 5 6%

Lithuania 5 6%

Czechoslovakia 1 1%

Latvia 1 1%

Total 84

-> Which church? = 5 Father born in Number % U.S. 16 64% Germany 8 32% England 1 4% Total 25


Father born in Number %U.S. 2 25Hungary 4 50Germany 1 12.5Czechoslovakia 1 12.5Total 8

-> Which church? = 0 Father born in Number % U.S. 49 35% Austria 33 24% Germany 9 6% Yugoslavia 8 6% Greece 7 5% Poland 6 4% England 6 4% Czechoslovakia 6 4% Russia 5 4% Hungary 5 4% Ireland 2 1%

Final Check

Vote in 1940

Religion Democrat Republican

Protestant 59% 41%

Catholic 80% 20%

Jewish 99% 1%

Other 76% 24%

Not affiliated 85% 15%

Total 79% 21%

Confirmation!

• Codebook for 1943 OPOR poll

Couple of other examples

• Both from earliest Gallup surveys– Case 3: Used presence or absence of a city

identification code on undocumented “city” variable to divide “Urban” category on “Rural-Urban” variable into “Big city” and “Small city”

– Case 4: Used farm residence, gender, and age to split useless “Other and none” category on occupation variable into farmer, housewife, student, and retired

Avoiding Unintelligent Appearance Bias (Roper, 1946)

the data consultant as archaeologist

Documents