ramscartalk

8/7/2019 ramscartalk

1/54

Surveys, questions and other cheap

ways of getting data

How to do stuff

1/14/02

Michael Ramscar


2/54

Plan

Surveys

What is correlational research? Sampling a population

Ways of collecting survey data

How to formulate Good questions Scales

Soothing strategies


3/54

Correlational Research

The nature of correlation

A numerical relationship between two variables

Predictions:

Will score A covary with score B?

Can be either negative or positive

Presence of correlation allows predictionPrediction can be tested on samples


4/54

Does drinking beer make you appear cooler, funnier

and sexier? How do you test this as a hypothesis?



5/54

2 ways:

Experimentally

Randomly assign people to groups

Get one to drink no beer; other drink 3 beers/day

Measure the change in their coolness in relation to pre-test

Conduct a survey

Ask people to estimate how much beer they drink Ask people to rate them for coolness, sexiness and funniness

Find the relationship between these two variables



6/54

Surveys: Pros and Cons

Advantages of survey research

Can sample a large number of individuals

Do not have to worry that the situation is artificial

Surveys are often easy to administer


7/54

Disadvantages of survey research

Danger of biased surveys

Sample may not be representative

Questions may bias responses

People may not answer truthfully/competently

Asking questions may not be the right methodologyCannot make statements about causality

Surveys: Pros and Cons


8/54

Correlation and Causality

Correlation

A numerical relationship between two variables

One value need not cause the other

The third variable problem - spurious relationships

Does drinking beer make you cool?

Perhaps uncool people are just too intimidated to go into bars

A Mantra: Correlation does not equal causation!


9/54

Sampling

Population

Sampling Frame

Sample

The hard part of conducting a survey is getting a representative sample


10/54

Potential sampling problems

Flawed surveys often reflect poor samples

Sampling bias can enter in two places

At selection of sampling frame

Use of phone directories

List of subscribers to a magazine

At selection of a sample

Use of an email survey

Emails made to students on Friday and Saturday nights


11/54

Suppose you were interested in student attitudes to

beer on campus

Sampling frame might be the student email directory

You could randomly sample names from this

What is good and bad about this frame?

Potential sampling problems


12/54

Types of Samples Nonprobability samples

No guarantee that elements of pop can beincluded

Will (almost never) be reflective of the population

May be helpful for studying ideal cases May be helpful for studying specialized populations

Accidental samples

Whoever happens to be around...

Surveys in the street Newspaper call-in polls

Problems with accidental samples

Built in bias

Generalizability?


13/54

More non-probability samples

Purposive sampling

Studying a particular population

May seek out special individuals

Survey of types of employee

May provide insight into what makes good vs. bad

Generally assume that non-probability samples are

biased True for many mail-in surveys


14/54

Subject Selection Effects Any study in which subjects self-select themselves

for participation are potentially problematic

Some types of selection are worse than others

Those willing to participate for money

Those interested in a particular topic

Those with a particular viewpoint Those with a particular skill

Problem with passive net surveys?


15/54

What to do? Probability samples

In a probability sample, there is a better chance that the

sample will reflect the underlying population

Types of probability samples

Simple random sampling

Select N individuals from sampling frame

Stratified random sampling

Useful if there are key subgroups in data

Suppose your students were half male and half female

Divide sampling frame into males and females

Select N/2 males and N/2 females

Ensures that this aspect of the sample matches the population


16/54

Survey research designs

Cross-sectional designs

Try to get a view of a population at one time

Time


17/54

Example

Suppose we were interested in the number of left-

handers in the population

We could conduct a survey

What if we found that left-handers make up about 10%

of the population up to age 60, and then there is a

continuous decrease in percentage

Might conclude that left-handers die earlier

Other factors?


18/54

Successive Independent Samples

ime

Can be used to assess changes in attitude over time

Must be sure that samples are comparable

Want results to be due to changes in time, not changes in

population


19/54

Longitudinal Studies

i e

Follow a single group of individuals over ti e

Requires a lot of ti e and effort

Subject ortality a proble

Drop out -- so eti es literal ortality

Most often people drop out for so e reason

Must ensure that there is not a syste atic reason for the

ortality

Left handers again?


20/54

Summary So Far... Research is only as good as the sample you draw

Must ensure the most representative sample

possible

Survey and correlational research cannot determine

causal relationships

Next - how do we go about it?


21/54

Conducting Surveys

There are many ways to conduct surveys

Through the mail

In person via an interview

Over the phone

Via the internet

some of all of the above can apply to this

Each of these methods has advantages and

disadvantages


22/54

Mail Surveys Mail surveys are used frequently

Minimal effort needed to collect data

Good for personal / embarrassing topics

Respondents replies are anonymous

May cut down on the amount of socially desirable responding

Problems with mail surveys

Respondents cannot ask for clarification

A big problem is questions are often poorly worded

No follow-up questions can be asked

No control over the order in which a survey is completed

Big potential for response bias


23/54

Self-selection in mail surveys

Respondents in mail surveys always self-select

Respondent must decide to do survey and mail it

Controlling response bias in mail surveys

Try to ensure a high return rate

Over 50% is pretty good

Response bias is less severe with return rates over 50%

Make survey attractive Include a pre-paid reply envelope

Personal introductions

Raffle a prize

But


24/54

Personal Interviews

Survey can be conducted in person

Advantages

High response rate

C

an control order of responding Respondent can ask for clarification

Interviewer can ask follow-up questions


25/54

Personal Interviews

Survey can be conducted in person

Disadvantages

There is a potential for interviewer bias

May suggest the desired response (nonverbally)

People may give socially desirable responses

They may not want to express their true beliefs to the interviewer

Difficult to ask about embarrassing topics

No anonymity for the respondent


26/54

Telephone Surveys

Doing surveys over the phone has some advantages

More anonymity than for personal interviews

Can control order of responding

Respondent can ask for clarification

Interviewer can ask follow-up questions


27/54

Disadvantages of telephone surveys

People may not want to be bothered over the phone

Particularly now that telemarketers often use surveys as

a sales pitch (think also email spam)

No control over surroundings

Respondent may be doing a number of things at once

Still the potential for interviewer bias

People may provide socially acceptable responses

They may not want to express their true beliefs to the

interviewer


28/54

Constructing Surveys

Kinds of questions

Open-ended

Range of allowable responses not determined in advance

Respondents decide how much information to give

Open-ended question can be difficult to score


29/54

Constructing Surveys

Kinds of questions

Closed questions

A fixed set of responses is provided

Respondent must pick one

Easy to score

May miss responses that respondent wants to give

Limited in the scope of the answer that can be given


30/54

More on questions

Mixed questions A fixed set of responses is provided

An open-ended choice is given to cover any alternativesthat may have been missed

Still has a limited scope

What meanings can bank have:

1. A place to keep money

2. The edge of a river

3. A large snail 4. To lean an aeroplane into a turn etc...

7 Other. _________________


31/54

Response Scales

For closed-ended questions, a list of responses is

needed

For some questions, the list is a set of options

What meanings can bank have: 1. A place to keep money

2. The edge of a river

3. A large snail etc.

Many questions ask for a degree of preference Yes/No questions

Easy response

No information about strength of preference


32/54

Response Scales

Likert Scales

Labeled points on scale

Strongly agree, Agree, Disagree, Strongly disagree

Variations on these scales Scale from 1-5 with ends labeled

How many values should be put on a scale?

People are reasonably good with 7 2 options

With more options, people sub-divide the space

Ratings are not necessarily more accurate on a 1-100 scale thanthey are on a 1-9 scale

False feeling of accuracy


33/54

How many points? How many points

The number of points you use depends on how

much your subjects will be able to discriminate,and the size of the effect you want to measure.

If the effect you're trying to measure is verysmall, then you'll need more points on the scale

If you don't think people can meaningfullydiscriminate more than 3 levels of a variable, thenyou should only have 3 points - having morepoints would just add noise to your data.


34/54


35/54

When might you not use a 7-

point scale? 1. Measuring a small effect in the middle of

the scale:

How good at maths are women generally?1------2------3------4------5------6------7------8------9------10

not at all.............................................................................extremely


36/54


point scale? 2. Getting more info out of your subjects

than they think they know

Is this picture old or new?

1------2------3------4

I'm sure it's old.........................I'm sure it's new


37/54


point scale? 3. Finding out that people aren't sure

Is this picture old or new?

1------2------3

I'm sure it's old.................I'm sure it's new


38/54


point scale? 4. Another example of forcing a binary

decision

Please rate whether each item below is

masculine or feminine:

violin ..........M or F

hammer.......M or Fchair............M or F

salt..............M or F


39/54

Writing Good Questions

A survey (or test) lives and dies with the questions

Unclear questions can confuse respondents

Biased questions can skew results

Vocabulary in questions should be clear and simple

Never use two syllables where one will do


40/54

Writing Good Questions

Questions should be clear and specific

Is something wrong with these sentences?

How do you feel about the grammar of these sentences?

Please indicate if one of the following sentences is

grammatical or not, and indicate why.


41/54

More on questions Edit questions for readability

Use short questions

Some words are ambiguous meaning thatpeople attribute multiple meanings to them.

Given this, do you think or not that more than

one meaning could be given to the following

words?


42/54

More on questions Do not assume that participants share your

enthusiasm

Get to the point Did I mention use short questions

Help us understand memory - do this.

Q-day!


43/54

Asking questions Participants may not always help you even

when they want to

Show me how you stop your children from

doing naughty things


44/54

Asking questions Participants may not always help you even

when they want to

How to get round this?


45/54

Asking questions - stratagems Participants may not always help you even

when they want to

I bet you couldnt stop little Jimmy from doing

something he really wanted to do


46/54


when they want to

Children especially may be too eager to

please / to nervous to perform


47/54


when they want to

Children especially may be too eager to

please / to nervous to perform

Deflect the task

Could you help this really dumb cookie

monster to


48/54


when they want to

Do fat people eat more when they are

stressed?


49/54

Asking questions - stratagems

Participants may not always help you even

when they want to

What happens when people have folk

psychological theories?

Or when theories are famous?


50/54

Things to avoid

Leading questions

Questions that suggest the right answer

People often recognise ambiguities -- or multiple meanings --in sentences. Re-write the following sentences in a way that

uses different words, but preserves their meanings?


51/54

Loaded questions

Questions that are emotionally charged

Competent readers can often spot ambiguities when reading.Does this sentence have more than one possible reading?

Things to avoid


52/54

Double-barreled questions

Packing too much into one question

Say which aspects of the the following sentence are ambiguousand why you think they are.

Things to avoid


53/54

Other things

Include conditional information before the key idea

in the question alter responses?

If you were reading the following sentence, would you

consider alternative meanings for it?

Would you consider alternative meanings if you were

reading the following sentence?


54/54

Consider varying the polarity of the questions

Some questions should be phrased negatively Other questions should be phrased positively

Controls for a positivity or negativity bias

Some people just like to say No (or Yes).

Other things

ramscartalk

Documents