institute for public relations summit on measurement class measurement 201

Copyright © 2010 by The Institute for Public Relations1

Measurement 201:Collecting quantitative information

Institute for Public Relations Summit on Measurement

David Geddes, Ph.D.

evolve24, a Maritz Research company

Saint Louis, Missouri

[email protected]

October 6, 2010


Today’s objectives

• Smart consumer

• Quantitative survey methods

1. Telephone

2. Web

3. Mail and multi-modal

4. Face-to-face

• Sampling

• Case studies

• Questions … as they come

2


Great resources


First steps

What are your objectives?

What do you want to explore, discover, test, or document?

Who are the right people to talk with?

What are appropriate data collection methods?

What is the value of the information?


Telephone surveys

• Types

– Random digit dial (RDD)

– List from sample provider or panel

– Company or client list (customers, employees, industry analysts, donors, partners, etc.)

• Uses and advantages

• Limitations and weaknesses

5


Telephone surveys: Current issues

Cell phone only households

– Same or different?

– What to do?

– Ethical and legislative issues

– A trend to follow


Surveys in general: Current issues

• Non-responsebias

Behavior Risk Factors Survey Response Rate

20

30

40

50

60

70

80

90

100

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Year

Resp

on

se R

ate

Median All

States

Pennsylvania

Minimum

Maximum

-.74%

-1.5%


Are more intensive methods helpful? The Keeter et al study

• Standard Survey: 36% response rate

– Calling done over five days

– Selected respondent from people at home at time of call (no random selection)

– Five call-backs, one call-back to refusals

• Rigorous Survey: 60.6% response rate

– Eight-week calling period

– Random selection of respondent from list

– Pre-notification letters with $2 incentive

– Multiple attempts (including letters to refusals)

– Multiple call-backs

Source: Scott Keeter et al, Consequences of Reducing Nonresponse in a National Telephone Survey, Public Opinion Quarterly 64:125-148 (2000)


Other design issues

• Oversampling

– Example: Attitudes to location-based apps

– Example: National survey on water conservation

• Weighting

– Example: Ethnicity


Telephone omnibus polls

• National random sample of 1,000 households

• Offered by all major research firms

• Fast

• Low cost

– Cost per question

– Demographics included

– Costing parameters

• Deliverables

• Applications

• Limitations


Internet surveys

• Advantages and uses

• Limitations– http://www.aapor.org/Content/NavigationMenu/Home/Left/A

APOROnlinePanelsTFReportFinalRevised.pdf


Mail surveys and multi-modal surveys

• Don Dillman et al, Internet, Mail, and Mixed-Mode

Surveys: The Tailored Design Method, (Wiley, 2008)

• TDM method:

1. Respondent-friendly questionnaire

2. Personalized correspondence

3. Token financial incentive ($1 or $2 prepaid)

4. Up to five phone contacts

5. Mail survey with stamped return envelopes

6. Phone again

• Other ways to improve response rates


Sampling

– Examine different sampling techniques

– Strengths and shortcomings

– Cases and examples

– Tools to make decisions in practice

– Not a comprehensive textbook treatment


Why care about sampling?

• Formal definition of our targets

– Example: Caregivers of Type II diabetes patients

• Generalize or project?

– Example: Poll of Kansas City voters about rental

car tax

• Understand and minimize sampling error

– How far off might our result be if we interviewed

another group of individuals?

• Make tradeoffs

– Budget, time, other factors given objectives


What is sampling?

• Probability sampling

– “Sampling is the science of systematically

drawing a valid group of objects from a

population reliably.” (Stacks, p. 150)

• Non-probability sampling (informal

definition)

– Process of systematically drawing a group of

objects from a population sufficient to meet

information needs. (Adapted from Stacks)


Some definitions

• Universe

– General concept of who or what will be sampled

• Population

– People or units to be sampled, formally defined

and described

• Sampling frame

– List of all people to be surveyed

– Example: List of all 90,000 registered

veterinarians under age 65


Some definitions

• Sample

– Actual people chosen for inclusion in the research

– Example: Selection of 10,000 veterinarians from

the list

• Completed sample

– People who actually responded to the survey

– Example: 3,000 veterinarians completed the

survey


Types of error

• Sampling error

– Issue: Potential error or uncertainty as a result of not

sampling from all members of sampling frame

– How far off would we be if we interviewed a different

500 people?

• Coverage error

– Issue: The sampling frame does not contain all

members of a population or contains a biased list

– Example: People without landlines in a telephone poll

– Example: People with invalid e-mail addresses in

membership


Some definitions

• Measurement error

– Error when respondents misunderstand or

incorrectly respond to questions

• Nonresponse error

– Respondents unlike nonrespondents


Problems and errors in sampling

• Understanding and reducing coverage

error

– Does the sampling frame (list) contain

everyone in the population?

– Does the list contain people who are not in

the sampling frame?

– How is the list maintained and updated?

– Does the list contain other information that

can be used to improve sampling?


Three approaches

1.Census

2.Probability sample

3.Nonprobability sample


Census sampling

• Interview or measure all members of a

population

– Example: Wal-Mart annual employee survey

• No error due to sampling

– Other types of error

• Rare in practice

• Is it worth the effort?


Probability sampling

• Every individual in a population has an

equal chance of being chosen

– In theory

– In practice

• Allows generalization or projection to the

population

• Known sampling error parameters

• What other sources of error?

• How much to invest, given objectives?


Probability sampling

Key types of probability sampling

• Simple random sampling

• Systematic sampling

• Stratified random sampling

• Cluster sampling


Nonprobability sampling

• Interview or measure without access to every

individual in a population

– Examples

• Situations where it is difficult to fully specify

the population or sampling frame

– Examples

• Cannot generalize

– How far off might our result be if we interviewed

another group of individuals?

• Key: Understand limitations … justify choice



• Convenience sampling

– Selecting based on availability

– Example: Hospital survey of nurses leaving a shift

• Quota sampling

– Selecting based on availability but weight based

on predetermined characteristics

– Example: Mall intercept sampling



• Purposive sampling

– Selecting participants based on knowledge of the

population and focus or objectives of the research

– Example: Survey of most influential journalists

covering the air transport industry

• Volunteer sampling

– Select based on agreement to participate

• Snowball sampling

– Selecting participants based on recommendations

of other participants


Sample size in probability sampling

• Key questions:

– How much might our results differ had we

interviewed another 100 American voters?

– How much more would we learn, given our

objectives, had we interviewed another 100

customers?

– More technically, how much sampling and

measurement error can we tolerate?

• To reduce sampling error and measurement

error, you must increase sample size


Sample size

• “Normal” curve

– Mean, standard deviation,

we can calculate confidence

intervals

– See an interactive demo at http://geographyfieldwork.com/StandardDeviation1.htm

– Sample size calculators on Web

• Maritz Stats (download)

• National Statistical Service

http://www.nss.gov.au/nss/home.NSF/pages/

Sample+Size+Calculator+Description?OpenD

ocument

68%

95%

99%

http://geographyfieldwork.com/StandardDeviation1.htm


Sample size

– Sample size of 385 is

necessary for a

confidence level of plus

or minus 5 percentage

points at the 95%

confidence level.

– Is this the biggest

source of error?


Case: National omnibus poll

• National random digit dialing completing surveys

with 1,000 adults

• Conducted Friday through Sunday

• Balanced post-survey to census figures for age,

gender, HHI, ethnicity (results only differ slightly)

• Evaluation

• Universe and population

• Sampling frame

• Sample and completed sample

• Sources of error or bias

• Final assessment – when is this appropriate?


Case: Online survey

– National online panel survey with 1,000 adults

– Balanced post-survey to census figures for age,

gender, HHI, ethnicity (results only differ slightly)

– Evaluation


• Sampling frame



• Final assessment


Case: Employee survey at a multi-division corporation

– Four divisions

– Management vs. non-management

– Results by age, gender, tenure at company

– Which survey methods?

– Develop a sampling plan:


• Sampling frame



• Final assessment


Case: Veterinarian survey

– American Veterinary Medicine Association

• 90,000 veterinarians under age 65

• 50,000 valid email addresses

• Goal: Low-cost survey

– Evaluation

• Which methods?


• Sampling frame



• Can we work around the limits?

• Final assessment and recommendation


Case: Journalist survey

– Client: Financial services company

– Respondents: List of 1,000 journalists who cover

personal finance, the economy, and lifestyle.

– Which survey methods?

– Evaluation


• Sampling frame



• Final assessment and recommendation


Some resources

Public relations research

• Don W. Stacks and David Michaelson. 2010. A Practitioner's Guide to Public Relations Research, Measurement and Evaluation. Businessexpert Press.

• Don W. Stacks. 2002. Primer of Public Relations Research.New York: Guilford Press.

Market research (leading business school texts)

• Gilbert A. Churchill and Dawn Iacobucci. 2004. Marketing Research: Methodological Foundations. Mason, OH: South-Western Cengage Learning.

• Naresh K. Malhotra. 2007. Marketing Research: An Applied Orientation. Upper Saddle River, NJ: Prentice Hall.