Outline of Today’s Discussion1. Developing Questionnaires

2. In-Class Exercise: Developing Questionnaires

Part 1

Developing Questionnaires

Developing Questionnaires1. Questionnaire - A written set of items that are

asked of every respondent in the study.

2. The items can be closed-ended or open-ended.

3. The questions may be self-administered, or interviewer-administered.

Developing Questionnaires1. Caution is needed to achieve accuracy and

precision when defining variables.

2. Ex: “Race” and “Ethnicity”.

“Hispanic” – A Spanish-speaking ethnic group.

“Latino” - people originating from North or South America, outside of the U.S. or Canada.

A person having European Spanish heritage is Hispanic and Caucasian (not Latino).

Developing Questionnaires

Rensis Likert (1903-1981)

B.A. University of Michigan 1926Ph.D. Columbia University 1932

Doctoral thesis addressed the now-famous Likert ScaleFounder: U of M’s Institute for Social Research

Director: 1946-1970

Developing Questionnaires1. Central Tendency Bias - Respondents may

avoid the extreme response categories.

2. To ensure the participants demonstrate “polarity” (a positive or negative tendency) remove the neutral response option:

Strongly Agree Agree Disagree Strongly Disagree

Issues for Likert-Like Scales

An even number of response categories can force “polarity”.On the other hand, if you want your participants to have the

option to express neutrality, use an odd number of response categories.

Developing Questionnaires1. Social Desirability Bias - Respondents may

portray themselves or their group in a favorable way.

2. Anecdote: Krueger & Clement on experimentally manipulated in-group out-group differences.

- Social Desirability- Out-group homogeneity effect

Issues for Likert-Like Scales

Developing Questionnaires1. Test-Retest Method - The same questionnaire is

administered to the same, large sample at two separate times.

2. Reliability is indexed by the extent to which the relative rankings of respondents remains stable.

3. Note: the overall scores on the second administration of the test may be higher or lower than the first. The critical factor is the relative rankings.

4. What statistic might we use for this?

Reliability

Developing Questionnaires1. An alternative would be the “split-half method”…

2. Let’s assume you want to measure depression, and you have a 100-item questionnaire on the topic.

3. The 100 items can be “split in half” (first half versus second half, or odd versus even items) and the two halves could be separately scored for each respondent.

4. If the test were reliable, people who score high on one half should score high on the other half.

Reliability

Developing Questionnaires1. Establishing validity is, typically, more complex than

establishing reliability.

2. How did Binet establish the validity of his IQ tests?

3. Binet’s efforts exemplified convergent validity - independent measurements of a given construct are correlated. (They independently “converge” on whatever it is they are measuring.)

4. Subtle Point: We can use correlations to inform us about validity, not just reliability!

Validity

Developing Questionnaires1. Ideally, we would also like to establish discriminant

validity - The ability of a questionnaire designed to measure one concept to yield answers that are NOT correlated with those of another questionnaire that measures a similar (but different) concept.

2. Discriminant Validity Example: “Life satisfaction”, versus “Positive Affect”. (See PubMed Abstract)

http://www.ncbi.nlm.nih.gov/pubmed/8831165

3. Potential Pop Quiz Question: Generate your own example of discriminant validity.

Validity

Developing QuestionnairesTips for good questionnaire items

-The vocabulary should be simple

-The item should be clear & specific

-No leading, loaded, or double-barreled items

-The item should be brief (<20 words)

Developing QuestionnairesTips for good questionnaire items

-Present all conditional information prior to the key ideaGood: If money were not an issue, what would you study in college?

Not As Good: What would you study in college, if money were not an issue?

-Detect response bias with oppositely worded itemsI often feel anxious: 1 2 3 4 5 I rarely feel anxious: 1 2 3 4 5

-Test for readability…

Developing Questionnaires

Pre-testing is critical!(say it with me)

Developing Questionnaires1. The Order of Questions Matters: Ex. “Do you

think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not more children?”

60.7% agreed when this question preceded another.48.1% agreed when this question followed another.

Schuman, Presser and Ludwig (1981).

2. What are “filter questions”, and what is their value?

Order Matters!

Developing Questionnaires1. Reported versus Actual Behavior…not as

congruent as we would like!

2. People self-report that they would help someone in need, regardless of by-standers. Behaviorally, the “by-stander” effect replicates reliably. (Latante & Darley, 1970)

3. This suggests a multi-method approach to a phenomenon may be best –surveys would be one of several methods.

About Interpretation

Developing QuestionnairesInterpreting Correlations:

Why Correlation Does Not Imply Causation

1. Spurious correlations – “Third Variables”

2. The direction of causation is ambiguous!

About Interpretation

Developing QuestionnairesAbout Interpretation

• At best, correlational analyses can only identify POTENTIAL causes.

• Path Analysis – a statistical technique that can help clarify the interpretation of correlations.

– Mediator: A variable that is used to “explain” the correlation between two variables.

– Moderator: A variable that affects the strength and/or direction of the correlation between two variables.

(e.g., interaction effects)

Developing QuestionnairesPath Analysis

Path A Path B

Path C

Mediator

D.V.Or

Criterion

I.V.Or

Predictor

Developing QuestionnairesPath Analysis

Path A Path B

Path C

S.E.S

College GPAAcademic

PerformanceSAT Scores

Developing QuestionnairesPath Analysis

Path A Path B

Path C

Chaos

PsychologicalDistress

Poverty

Developing QuestionnairesAbout Interpretation

• Assume that Poverty (predictor) correlates significantly with Psychological Distress (criterion).

• Is that correlation mediated (explained) by “Chaos”?

• Yes, if the data pass this “Mediational Test”:– Predictor & mediator are significantly correlated.– Criterion & mediator are significantly correlated.– The predictor & criterion correlation is significantly

reduced after controlling for the mediator.» We’ll describe the role of so-called “moderator” variables later,

if you take the stat course (PSYC 370).

Part 2

In-class ExerciseOn

Developing Surveys

AcknowledgmentsImages used in this educational presentation were obtained from Wikimedia Commons, in accordance with regulations regarding copyright, use, and dissemination.

http://commons.wikimedia.org/wiki/Main_Page