Designing Case Forms with Validity in Mind

Louis A. Morris, Ph.D.

Barnett International

December 3, 2002

“Experience never errs; it is only your judgment that errs in promising itself results [that] are not caused by your experiments”

“Experience never errs; it is only your judgment that errs in promising itself results [that] are not caused by your experiments”

Leonardo da Vinci

What is Bias?

• Systematic Variation from the Truth – Opposite of Fair, Not Random Variation– Selection Bias

• effect of the treatment is confounded with pre-existing differences in the treated and control groups

– Confirmation bias • one tends to notice what confirms one's beliefs and to ignore, or

undervalue what contradicts one's beliefs

– Demand Characteristics, Observer Bias, Social Desirability Effects, Response Artifacts, Volunteering Effects, Evaluation Apprehension, Hawthorne Effects, Sensitization Effects, ….

Question-Asking Bias

Objectives

• When do we need to be concerned about question-asking bias?

• Why is it important to understand how people interpret:– Words, Questions, Questionnaires?

• What are the psychometric properties of questionnaire scales that determine measurement acceptance?– Validity, Reliability, Sensitivity, Responsiveness, and

Minimally Important Difference

Potential For Bias Impacts• Drug Approvals & Labeling

– Judgmental Responses (Symptoms, Mood/Emotion, Subjective Evaluations)

– Patient Reported Outcomes• Quality of Life, Satisfaction, Productivity, Drug-Specific Outcomes

(e.g., bothered by facial hair, ease of use of inhaler)

• Advertising & Promotional Claims– Patient/Consumer Studies– DTC promotion drug benefits not just clinical effects

• OTC Drugs Switch Approvals– Comprehension– Actual Use and Effects

Questions – Answers - Information

Patient - Reported Clinician - Reported

For example,Functional status

SymptomsHRQL….SatisfactionEvaluation Criteria

Perceptions, LinkagesGlobal Impression

Well-beingTreatment adherence

For example,

Global impressions Observation & tests

of function

Caregiver - Reported

For example,

DependencyFunctional status

Physiological

For example,

FEV1

HbA1cTumor size

Outcomes Assessment Sources and Examples

QALYs

BotherDiscomfort

Symptoms

Satisfaction

HRQL

ADL

Cost

Productivity

“Outcomes” claims classification

PROs

Meyer, Burke, 1999

Can Biased Questions Affect Drug Approval?

• clear statement of the objectives• the proposed or actual methods of analysis• valid comparison with a control • protocol for the study and report of results

should describe the study design precisely • subjects … adequate assurance that they

have the disease or condition being studied • method of assigning patients to treatment

and control groups minimizes bias and is intended to assure comparability of the groups

(21 CFR 314.126)

Adequate and Well Controlled Studies

Adequate and Well Controlled Studies (2)

• Adequate measures are taken to minimize bias on the part of the subjects, observers, and analysis of the data

• methods of assessment of subjects' response are well-defined and reliable.

• report of the study should describe the results and the analytic methods

Sources of Question-Asking Bias

• Individual Words– Word Interpretation

• Individual Question– Question Phrasing: Leading, Framing, – Question Asking “Environment”

• Placement in Questionnaire, Broader Atmosphere

• Series of Questions– Scale Validity– Other Psychometric qualities

Remember the comic strip Dagwood?his wifehis bosshis pet

Question Interpretation

• I got in late last night….– Are you tired?

Yes No

• Last night I was reading about how chemotherapy causes anemia which has profound physical effects on cancer patients….– Are you tired?

Yes No

Context Dependency

• How tall is the Statue of Liberty?

• How important is Freedom?

• Some concepts are vivid and invariant– Context-independent

• Some concepts vary in interpretation based on context– Context-dependent

What provides Context?

• Respondent look for “cues”– Question itself

• How fast were the cars going when they– Bumped?, Collided?, Crashed?, Smashed?

– Previous Questions – Presumed Intent

• Question-asking as a Linguistic Conversation

• This is an experiment (Demand Characteristics)

Classify each of these as a vegetable or an animal:CarrotBeagleDaisy

Bias Also Due To:

• Memory – Recency Effects (Daisy Question)

• Judgment – Decision Making– Decision Heuristics (framing, anchoring and

adjustment, etc.)

Would take a drug where 15% of the people have a serious side effect?

Would take a drug where 85% of the people do not have a serious side effect?

What is wrong with this Question?

• Given all the painful heartburn you suffered before this drug was available, how well did it work compared to the other treatments available?

– Focuses on painful heartburn and suffering (leading)– Multipart – what is question? (cognitive responses)

• Even more of a problem for interviewer based and telephone surveys

– Does not specify what other treatments (stimulus under evaluation)

– Assumes it worked (social desirability effects)• Postulate the negative (how well did it work or not work)

Types of Questions/Data (1)

• Nominal or Categorical– Are you tired?

• Ordinal– Rate how tired you feel

• Not at all, A little bit, Somewhat, Extremely

• Four point, Likert Scale

What type of questions do we ask? depends on type of data desired

Types of Questions/Data (2)

• Interval/Ratio – How tired do you feel (select one):

• I have no physical sensations of tiredness

• My eyes are closing a little bit

• My head is slumping

• My whole body is limp

– Thurstone Scale

Are some scale forms less bias than other forms?

Different Forms, Different Biases

– How tired do you feel (select one):• I am not at all tired

• My eyes are closing a little bit

• I cannot concentrate

• I want to get out of this lecture

• Vague, multiple interpretations• Physical Issues and Cognitive Issues• Confuses Tiredness with Boredom

Designing Questionnaires• If initial questions influence others:• Recall – Open-Ended relies on memory

– Unaided• What did the leaflet say or suggest about the drug?

– Aided• What did the leaflet say or suggest were the side effects of the

drug?

• Recognition• Was sleepiness mentioned or not mentioned as a side effect of the drug?

• Demographics at end– Invariant interpretation

The Funnel Design

Question-Asking as an Artifact

• Do we randomize or vary the order of questions?

– Yes, Comprehension Test – Side Effect Knowledge• Avoid order bias, seek robust results, test knowledge

– No, Clinical Trial – Assess Tiredness with Scale• Want invariant interpretation of terms, want “bias” to be the same

across administrations

• Three stages of an artifact: (McGuire)– Ignorance, Coping, Exploitation

Scale Psychometrics

• Validity

• Reliability

• Sensitivity

• Responsiveness

• Minimally Important Difference

Scale - Tiredness

• Measures a complex concept– Fair sampling of items

Cognitive--------Physical

Amount

How do we know we are measuring what we want to

measure?• Validity - process not a characteristic -

Understanding what is measured– Face Validity - examine items – Content Validity - coverage– Construct Validity - any theory?– Concurrent Validity - positive correlation– Discriminative Validity - negative correlation

FACT-Fatigue Subscale– I feel fatigued – face validity– I feel weak– I feel listless (washed out)– I feel tired– I have trouble starting things because I am tired – (cognitive)– I have trouble finishing things because I am tired– I have energy – the opposite, negate yea saying, halo effect– I need sleep during the day - outcome– I am too tired to eat– I need help doing my usual activities – (social)– [Plus others]

Correlate with Red Blood Cell Levels - Anemia

How do we know we are measuring a concept

consistently?• Reliability

– Within the same scale• inter-item

– Over time• test-retest

How do we know that our measures can pick up differences

that actually exits?• Sensitivity

– Type of scale items• I am not tired at all

• I am so tired I must go to sleep right now

• I am having trouble concentrating

• I am feeling a little bit sleepy

2 items

How do we know that our measure corresponds to changes

in the variable in question?

• Responsiveness– Correlate changes in direct measure of clinical

outcome (Red Blood Cell Count) with changes on scale (tiredness).

How do we know that an observed effect is clinically

meaningful?

• Minimally Important Difference– Smallest scale difference judged to be

meaningful (e.g., where a change in therapy would be warranted)

– Effect Size

Developing a Scale? Boredom Scale

• Conceptual - what is boredom? – Other scales? – literature, experts– Boredom effects, validation methods

• Operational - Item Generation/Reduction/Validation– modular, adaptation from general scales– focus groups – initial question design (scale measure type)– ratings: How important are each of these items?– Factors Analysis - how many dimensions?– Validation studies - psychometrics– Clinical Impact (substantiation studies)

Selecting a Scale

• Practical Aspects - will people fill it out?– Number, complexity of items– “Involvement” with scale

• Bias - – Leading questions, socially desirability, yea-

saying, etc.

• Validity in my population?: pilot

Conclusions• Science is a process

– Use observation to obtain the truth• Question-Asking Bias – not responding to question content• Validation – not determining the truth

• Bias as an Artifact– Cannot rule it out, Known/Unknown– Seek to understand and ask a fair set of questions

• Always tradeoffs– Understand measurement goals to determine best

approach