![Page 1: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/1.jpg)
Quick and Painless Introduction to Survey Methodology
R. Michael Alvarez
PS 120
![Page 2: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/2.jpg)
Testing Theories or Models
• Experimental data: expensive, and has validity problems
• Quasi-experimental data: aggregate election statistics, other data. Suffers from various problems.
• Survey data: data about individual voters
![Page 3: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/3.jpg)
Fundamentals of Surveying
• Population: all elements of interest, usually in a geographic area
• Sample: subset of population• Sample frame: list of sample (addresses,
phone numbers, email addresses, etc)
![Page 4: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/4.jpg)
Basic Typology of Surveys
• Probability designs: population elements have a known (at least in theory) probability of selection into the sample.
• Nonprobability designs: population elements have an unknown probability of selection into the sample All of the statistical tools we use to study
survey data are based on probability designs!
![Page 5: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/5.jpg)
Literary Digest 1936: What Went Wrong?
0
10
20
30
40
50
60
70
Roosevelt Vote
Election
Literary DigestGallup Poll
Crossley PollFortune
![Page 6: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/6.jpg)
Literary Digest Methodology
• Sent out 10 million straw ballots, using a list drawn from auto registration lists and telephone books.
• 2.3 million were returned, about 25% response. Flawed sample (overrepresented rich and
Republicans) Low response rate
![Page 7: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/7.jpg)
Literary Digest Fiasco Reforms Polling
• Underlying flaws of Literary Digest straw polls revealed --- not using a scientific sampling procedure
• Others, especially Gallup, Roper and Crossley began to work to find better ways of generating samples …
• The Literary Digest soon folded!
![Page 8: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/8.jpg)
Problems Continue, 1948
0
10
20
30
40
50
60
Truman Dewey Thurmond Wallace
Crossley
GallupRoper
Election
![Page 9: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/9.jpg)
New Sampling Techniques Were Flawed!
• Before 1948, they used “quota sampling”• Each interviewer is assigned a fixed quota
of subjects to interview from certain demographic categories … gender, age, education, residential location.
• Once they met their quota, the interviewer could select anyone they desired until they conducted all their required interviews
![Page 10: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/10.jpg)
Quota Sampling
• It’s not necessarily a stupid idea, as long as the underlying data (Census data?) used to construct the parameters of the sample are okay.
• But, what can happen is that interviewers end up working to talk with people who are easy to contact. In 1948 that tended to be people in nice neighborhoods, with fixed addresses and phones (ie, Republicans).
![Page 11: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/11.jpg)
Random Sampling
• In the 1950’s, most scientific surveys shifted to the use of random sampling
• For example, Gallup in 1956 moves to the use of random selection methods and seems to generate more accurate presidential election forecasts thereafter
![Page 12: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/12.jpg)
Gallup’s Track Record
-8
-6
-4
-2
0
2
4
6
19361940 194419481952 19561960 196419681972 19761980 198419881992
Difference
![Page 13: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/13.jpg)
Basic Introduction to Sampling
• Concept: The population (or universe or target population).
• The population is the entire set of units to which a survey will be applied. Individual members of the population are called units or elements.
![Page 14: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/14.jpg)
More on sampling ...
• Next, we need a list of population units from which we can draw a sample.
• This list is called the SAMPLE FRAME• The basic property of a sample frame is that
every unit in the population has some known chance of being selected into the sample by whatever method is used to select units
![Page 15: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/15.jpg)
Then ...
• Probability sample: units are selected using a method that insures that each unit has a known, nonzero probability of being included.
• Nonprobability sample: units are selected and inclusion probabilities are unknown (quota sampling …)
![Page 16: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/16.jpg)
Simple Random Sampling
• All elements of population have equal probability of being sampled Cluster sampling: population is divided into
clusters or groups, and clusters are sampled. Why? Cost and simplicity.
Stratified sampling: population is divided into subpopulations, or strata, and sampling occurs within strata. Why? Strata might be of interest or require different methods of analysis.
![Page 17: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/17.jpg)
Sampling Error
• Best way to think of survey error is in the context of proportions (percent saying “yes” or “no”).
• Standard error of a proportion in SRS: se(p) = sqrt[ ( p(1-p) )/( n - 1 )]
![Page 18: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/18.jpg)
An Example of Survey Error
+- 2 Standard Deviation Example
0
2
4
6
8
10
12
Sample size
Standard deviation of P
P=.5
P=.7
P=.9
P=.5 10.05 7.089 5.783 5.006 4.477 4.086 3.782 3.538 3.335 3.164 3.016 2.888 2.775 2.674 2.583 2.501 2.426 2.358 2.295 2.237
P=.7 9.211 6.497 5.3 4.588 4.103 3.745 3.467 3.242 3.057 2.9 2.765 2.647 2.543 2.45 2.367 2.292 2.224 2.161 2.103 2.05
P=.9 6.03 4.253 3.47 3.004 2.686 2.452 2.269 2.123 2.001 1.898 1.81 1.733 1.665 1.604 1.55 1.5 1.456 1.415 1.377 1.342
100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000
![Page 19: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/19.jpg)
An Example of Nonresponse Error? March 2001 CSLP RDD
![Page 20: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/20.jpg)
NES Response and Refusal Rates
0
10
20
30
40
50
60
70
80
90
195219601964196819721976198019841988199219962000
ResponseRefusal
Response rate: interviews net of refusals and respondentswho cannot provide an interview (e.g., language, etc)
![Page 21: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/21.jpg)
Misreporting: Voting in Recent Federal Elections
0102030405060708090
1964 1968 1972 1976 1980 1984 1988 1992 1996 2000 2004
Official Turnout CPS TurnoutNES Turnout McDonald-Popkin
Note: Percentage of voting age population
![Page 22: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/22.jpg)
Item Nonresponse
• Don’t know is necessary in any survey, so that people can tell you if they don’t have an opinion
• Due to uncertainty, vague questions, or respondent unwillingness to answer some questions
![Page 23: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/23.jpg)
Should Gov’t Provide More Services?
050
100150200250300350400450500
Fewer More
No Opinion
Number
1996 NES
![Page 24: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/24.jpg)
Certainty of Responses?
05
10
15
2025
30
3540
45
50
1 2 3 4 5 6 7
Not
PrettyCertain
Senator Position on Abortion Scale, Alvarez and Franklin 1993
![Page 25: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/25.jpg)
Question Wording and Order?
• Would you say that traffic contributes more or less to air pollution than industry? (45% traffic primary contributor to 32% industry).
• Would you say that industry contributes more or less to air pollution than traffic? (57% industry primary contributor to 24% traffic) Wanke et al. 1995
![Page 26: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/26.jpg)
Types of Surveys
• Self-administered questionnaires (mail, web) cheap but:
low response rates uncertainty about who completes questionnaire
![Page 27: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/27.jpg)
Types of Surveys
• Telephone: RDD/CATI quick, random? Uncertainty about respondent, difficult to ask
complex questions, must be short
![Page 28: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/28.jpg)
Types of Surveys
• Face-to-face (on doorstep, exit polls) highly accurate, high response rates very expensive to implement interviewer biases are problematic
![Page 29: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/29.jpg)
Internet surveying --- the future?
• Cheap to implement• Quick in the field, quick with analysis• Can implement complex designs, for
example, use multimedia
![Page 30: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/30.jpg)
Basic types of Internet surveys
• Probability designs• Nonprobability designs• Mixtures of probability and nonprobability
![Page 31: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/31.jpg)
Probability-based Internet surveys
• Intercept-based surveys of visitors to particular web sites
• known email lists (students, etc)
![Page 32: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/32.jpg)
Nonprobability Internet surveys
• Entertainment surveys• Self-selected surveys• Volunteer survey panels
![Page 33: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/33.jpg)
Surveys are not perfect!
• Sampling error (difference between sample and pop.)
• Coverage error (deviation between sample and frame)
• Systematic sampling error; error in frame
• Nonresponse (unit) bias
• Nonresponse (item) bias
• Question wording or ordering effects
• Interviewer error; coding mistakes
![Page 34: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/34.jpg)
How do I evaluate survey results?
• Sample size• Sampling
methodology (probability or non-probability)
• Estimated sampling error
• Survey response rate
• Questionnaire design and question wording
• Item response rates• Intuition: do the
results make sense?
![Page 35: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/35.jpg)
Caltech’s National
Public Relations Initiatives
March 11, 2003
![Page 36: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/36.jpg)
Brief recap of survey methodology
• Survey conducted by ICR• Wednesday, February 12-Sunday February
15• Omnibus survey• N=1010• Tabulation presents weighted results,
weighted to map to American adult population
![Page 37: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/37.jpg)
Questions
1 Considering what you might have seen or heard about the California Institute of Technology, also known as Caltech, in Pasadena, California, which of the following best describes your opinion of Caltech’s reputation. Would you say Caltech’s reputation is excellent, good, fair, or poor?
![Page 38: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/38.jpg)
Questions
2 How did you hear about Caltech? (not asked to those unable to answer 1).
3 What do you think Caltech is best known for (not asked to those unable to answer 1).
![Page 39: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/39.jpg)
Questions
4 Now, as I read each of the following topics, please tell me, generally speaking, whether or not you are interested in the topic: voting, the brain, climate changes, astronomy, earthquakes, nano-technology, detecting gravity waves
![Page 40: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/40.jpg)
Questions
5 And considering those topics in which you said you had an interest, how do you usually get news and information about these topics? (asked to only those who were interested in at least one topic)
![Page 41: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/41.jpg)
National Awareness of Caltech
0
10
20
30
40
50
60
Level of Awareness of Caltech Among General Public
AwareUnaware
![Page 42: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/42.jpg)
0
10
20
30
40
50
60
70
80
High Income College West 55-64
AwareUnaware
Caltech Awareness Successes
![Page 43: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/43.jpg)
Caltech Awareness Challenges
0
10
20
30
40
50
60
70
North East > 64
AwareUnaware
![Page 44: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/44.jpg)
Caltech’s Reputation
0102030405060708090
Caltech's Reputation as Judged by ThoseAware of the Institute
Excellent/Good
Fair/Poor
![Page 45: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/45.jpg)
Media Relations Focus
• National and northeast TV - visit them, pitch them, invite them to campus.
• Households with children• Senior-oriented media
![Page 46: Quick and Painless Introduction to Survey Methodology](https://reader035.vdocuments.us/reader035/viewer/2022062807/5681505a550346895dbe5cba/html5/thumbnails/46.jpg)
Evaluate the Caltech Awareness Survey
• Technical evaluation• Substantive evaluation• Policy evaluation