Measurement and Questionnaire Design

Operationalizing

• From concepts to constructs to variables to measurable variables

• A measurable variable has been operationalized, has an operational definition

• Some of the greatest research creativity can be found here

Scale Types

• Nominal – assigns names, try to determine equality among categories

• Ordinal - ranks phenomena

• Interval – ranks with assumed equal distances between responses, categories

• Ratio – the properties of an interval scale with an “absolute” (meaningful) zero

Reliability

• Necessary but not sufficient condition for validity

• Can indicate– Stability (across people, times, instruments)– Internal consistency

• T = X – ewhere T is a true score, X the observed score and e is error

Stability & Consistency

• Check stability, consistency in a number of ways– Parallel forms to the same individual

– Test – retest variation (same form)

– Split-half with halves scored separately (but there are 92,378 ways of dividing a 20 item test in half)

– Item – total correlations

– Cronbach’s takes the mean of all possible split-half correlations

To Increase Reliability Estimates

• Increase the number of items

• Use a sound theoretical base to construct items

• Items from a more heterogeneous group will be more reliable

• Use non-speeded tests if ability to finish rapidly isn’t assessed

Validity

• Are we measuring what we intend to measure?

• Reliability a precondition

• Many measures of validity are correlational

Types of Validity

• Face validity – commonsensical (but not necessarily obvious)

• Criterion (convergent) validity – do alternate measures provide similar results– Predictive– Concurrent

• Construct validity – the extent to which an operationalization measures the concept it is supposed to measure

Attitudes

• Three components– Affective (eg., intention to buy)– Cognitive (eg., intention to buy)– Behavioral (eg., purchase behavior)

Attitude Scales

• Having two options – Yes/no– Agree/disagree

• More than two options– Likert (or Likert-type)– Semantic differential, bipolar adjectives– Graphic rating scale

Attitude Scales

• Balanced versus unbalanced response set

• Forced versus non-forced response set

Questionnaire Design

• An art

• How will it be administered?

• Will you use open versus closed items?– Closed items should have exhaustive, mutually

exclusive responses whenever possible

• As a rule, continuous measures are better than categorical

Questions

• Consider least knowledgeable respondent for wording

• Avoid loaded, leading questions

• Avoid double-barreled questions (look for every instance of the word “and”)

• Exorcise ambiguity

• Consider aided recall when possible

Question Sequence

• Easy questions first

• Sensitive questions middle or last

• Branch if necessary (pivot, filter)

• General before specific questions

• Avoid anchoring effects

• Consider reverse scoring

Layout

• Create interest (with a header, cover, pre-contact)

• Offer a results summary

• Use color if budget allows

• Use white space liberally

• Keep coding requirements in mind

Pilot

• Consult experts before question construction, design if unfamiliar with material

• Show experts draft survey• Accurately estimate true time required to

complete• Permit questions during pilot administration• Do the pilot in person