Measurements, Mistakes and Misunderstandingsin Sample Surveys

Lecture 1

Goals Understand the difficulty of precisely

defining and measuring something. Realize bias can enter surveys many

ways, even with a good sample. Compare open and closed questions. Understand why random sampling

works

Simple Measures...

...don’t exist!

Measurement Variability

Variable measurements include unpredictable errors or discrepancies that aren’t easily explained.

Natural variability is the result of the fact that individuals and other things are different.

Reasons for variable measures

Measurement error Natural variability between

individuals Natural variability over time in a

single individualStatistics are tools to help us work

with measurements that vary

Some Pitfalls in Studies

Deliberate bias

Deliberate Bias?

If you found a wallet with $20, would you:

“Keep it?”

“Do the honest thing and return it?”

Deliberate Bias?

If you found a wallet with $20, would you:

“Keep it?” (23% would keep it) “Do the honest thing and return

it?” (13% would keep it)

Unintentional Bias?

“Do you use drugs?” “Are you religious?”

Desire to Please?

People routinely say they have voted when they actually haven’t, that they don’t smoke when they do, and that they aren’t prejudiced.

One study six months after an election: 96% of actual voters said they voted. 40% of non-voters said they voted.

Asking the uninformed?

Washington Post poll : “Some people say the 1975 Public Affairs Act should be repealed. Do you agree or disagree that it should be repealed?”

24% said yes 19% said no rest had no opinion

Asking the uninformed?

Later Washington Post poll: “President Clinton says the 1975 Public Affairs Act should be repealed. Do you agree or disagree that it should be repealed?”

36% of Democrats agreed 16% of Republicans agreed rest had no opinion

Unnecessary Complexity?

“Do you support our soldiers in Iraq so that terrorists won’t strike the U.S. again?”

Unnecessary Complexity?

Roper 1993: “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?”

22% said it was possible and 12% more said they didn’t know

Unnecessary Complexity?

Gallup 1994: “Do you doubt that the Holocaust actually happened?”

Only 9% said they had doubts Only about 2% said the Holocaust

“definitely” or “probably” didn’t happen

Question Order

“About how many times a month do you normally go out on a date?”

“How happy are you with life in general?”

Closed vs. Open questions

Closed: Respondent given list of choices

Open: Respondent answers in own words

Problem with closed format: Choices may not reflect respondent’s ideas

Problem with open format: Difficult to summarize

How to Get A Good Sample

Common Research Methods Randomized experiments: Measure

deliberate manipulation of the environment

Observational studies: Measure the differences that occur naturally

Meta-analyses: Quantitative review of multiple studies

Case Study: Descriptive in-depth examination of one or a few individuals

Common Research Methods

Experiments Observational Studies Meta-Analyses Case Studies

Sample Surveys

Why Sample? A properly-selected random sample will

mirror the characteristics of the population from which it was taken.

The accuracy (margin of sampling error) depends on size of the sample -- not the size of the population!

A sample survey is much cheaper and faster than a census.

Sometimes impossible to do a census. Sample can be more accurate.

Margin of Error

95% of the time, a random sample’s characteristics will differ from the population’s by no more than about

where N= sample size(Yes, memorize this formula.)

n1

Margin of Error Example

Imagine that 70% of a random sample of 400 UNL students say professors should wear propeller beanies while lecturing. About what percentage of all students think that?

Calculating Margin of Error

Square root of 400 = 20 1/20 = 0.05 = 5 percentage points 70% plus or minus 5 points 70-5 = 65 and 70+5 = 75 Therefore, the “real” percentage

almost certainly is within +/- 5 percentage points, or between 65 and 75 percent.

Margin of Error Example

Imagine Estádio da Luz filled with 25,000 men and 25,000 women. Pick at random:

100 people -- Get 40.0%-60.0% men 200 people -- Get 43.0%-57.0% men 400 people -- Get 45.0%-55.0% men 900 people -- Get 46.7%-53.3% men 1,600 people -- Get 47.5%-52.5% men

Two Important Concepts about Error Margin

The larger the sample, the smaller the margin of sampling error.

The size of the population being surveyed doesn’t matter.*

*Unless the sample is a significant fraction of the population.

Sampling realities

Bigger sample means more cost (money and/or time)

Diminishing return on error margin improvement as sample increases.

Sample needs only to be large enough to give a reasonable answer.

Sampling error affects subsamples, too.

Simple Random Sampling

Everyone should have the same chance of being sampled.

All that’s needed is: A list of the population Random numbers (or some other

physical method of selection, like drawing names from a hat.)

Systematic Sampling

Pick a random starting place in a list, then survey every nth person on the list...

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Random Digit Dialing

Generate telephone numbers by randomly selecting a known exchange, then four randomly-chosen digits

602+496+5+7+9+8

Advantages of RDD?Disadvantages of RDD?

Difficulties in Sampling

The major difficulties in doing a survey are getting a good sample and getting them to respond. Reasons include:

Using the wrong “universe”. Not reaching the individuals selected. Getting low response or getting a

volunteer response.

Disasters in Sampling

Some kinds of samples are so bad they make the so-called “survey” worthless:

Volunteer (self-selected) samples Mail-in surveys Phone-in surveys

Convenience samples, such as so-called “man in the street” surveys

Literary Digest Poll In 1936, the Literary Digest magazine

sent out 10 million surveys and from the responses predicted Alf Landon would beat FDR by a 60 to 40 margin.

George Gallup surveyed 50,000 people: He correctly predicted FDR would win And correctly predicted how wrong the

Literary Digest poll would be

Perguntas?