part 2 of 3 by: danielle davidov, phd & steve davis, msw, mpa introduction to research:...
TRANSCRIPT
Part 2 of 3 By: Danielle
Davidov, PhD &Steve Davis,
MSW, MPA
INTRODUCTION TO RESEARCH:
MEASUREMENT
1) Threats to research studies
2) Steps in the research design process
3) Identifying and defining variables
4) Validity and reliability of measurement
OUTLINE
Starts After the research question has been developed and refined
The who, what, when, where, and how of research
It comprises the Materials and Methods and Limitations sections of publications
RESEARCH DESIGN
The goal of research design is to provide the most valid and correct answer to the question i.e., we want to make sure we are “doing it right”
This is done by minimizing the threats to the soundness of your study’s conclusion(s): CHANCE
BIAS
CONFOUNDING
WHY IS DESIGN IMPORTANT?
The threat that the study’s findings are merely the result of random processes (chance) i.e., the findings are a “fluke”We can’t do much to control random error
Also referred to as: Type 1 Error Random Error Unsystematic Error
STUDY THREATS: CHANCE
The threat that the study’s results are due to an unfair preference given to one group or a set of outcomes in a studyWe can try to control bias in our study design and subject recruitment
Also referred to as: Systematic Error
STUDY THREATS: BIAS
The threat that the association or relationship observed in the study is influenced by or related to another variableWe can control for this in our study design, subject recruitment, and data analysis techniques
STUDY THREATS: CONFOUNDING
We try to minimize the three main threats during all stages of design process, which are:
1) Identifying and Defining Variables*2) Selecting Measurement Methods*3) Selecting (Sample) Subjects4) Selecting a Research Design5) Establishing an Analysis Plan
*We will be talking about steps 1 and 2 in this presentation
STEPS IN RESEARCH DESIGN
What do you want to measure? (Identify) Ex) Patient satisfaction levels with ultrasound vs. history and physical
exam only
How do you want to operationalize “patient satisfaction?” (Define) Ex) Answers of “Good”, “Very Good”, or “Excellent” on a survey
given to patients about the care they received in the emergency department
IDENTIFY & DEFINE VARIABLES
Classifying Variables:
Independent Variable The variable that has an effect on or influences the
dependent variable. This is the FACTOR/INTERVENTION
i.e.) History and Physical Exam or Ultrasound + H & P
Dependent Variable The variable that is affected by, or dependent upon,
the independent variable. This is the OUTCOME i.e.) Patient Satisfaction
IDENTIFICATION & DEFINITION OF VARIABLES
Classifying Variables (continued)
Confounding Variables –a variable that is related to both the independent and the dependent variable CONFOUNDER or CONTROL variable Common confounders/controls in medical research:
Age Gender Race Severity of Illness
IDENTIFICATION & DEFINITION OF VARIABLES
Controlling for Confounding Variables Not adequately controlling for confounding variables can
have disastrous consequences on your research Identify and define as many as possible
From previous literature From clinical observations From theory
IDENTIFICATION & DEFINITION OF VARIABLES
What if we didn’t consider these important variables when examining the relationship between the Independent and Dependent variables???
Operationalizing variables The process of defining variables in a measurable way.
IDENTIFICATION & DEFINITION OF VARIABLES
Levels of Measurement(NOIR)
Nominal
Ordinal
Interval
Ratio
IDENTIFICATION & DEFINITION OF VARIABLES
Lower
Higher
Characteristic data that cannot be rank ordered
This data is “categorical” – made up of “categories”, not “levels” or “increments” Ex) Ice cream flavors—vanilla is not “better” or “more”
than chocolate Examples: Gender, Race, Student, Marital Status, State or Country of Residence, Insurance Status, Discharge Status, etc. Yes/No Responses are Nominal This type of data is usually “descriptive”
Used to describe a population or sample
NOMINAL LEVEL DATA
Data that can be rank ordered but that do not have measurable distances between each level of rank
Likert Scales - Strongly Disagree to Strongly Agree Class rank - Freshman, Sophomore, Junior, Senior; PGY-I, PGY-II,
PGY-III Degree of illness: None, Mild, Moderate, Severe
Senior is a higher rank than Freshman, but there is no way to quantify how much higher Senior is vs. “Freshman” or how much “more” illness those with a severe illness have compared to those with a mild illness
ORDINAL LEVEL DATA
Data that can be ordered and that have a measurable distance between each level The Interval Scale - Distances between positions are
equal, but "0" is an arbitrary assignment. For example, with temperature, each degree is equally distant from another, but "0" does not mean that there is no temperature. It is simply a reference point on the scale.
The Ratio Scale - All positions are equally distant and "0" means that the value is truly "0". If you have "0" money, you have none. But if you have $200, you have twice as much as a friend who has $100.
Examples of Interval/Ratio Data: Age Height Weight Many Clinical Serum Levels Blood Pressure
INTERVAL/RATIO DATA
Defining variables at higher levels of measurement allows the use of statistical tests that have more Power Power = the probability of finding a true relationship of
difference if it genuinely exists
It is usually better to collect data at higher levels of measurement and then collapse into categories later Ex) Age
What is your age? ____ (best) vs.
What is your age? vs. 18 – 25 26 – 35 36 – 45 etc.
vs.
Under 40 & over 40
LEVELS OF MEASUREMENT AND POWER
Once you have defined and operationalized your research question’s variables, you must decide how to measure them and/or what measurement tool you will use.
There are two forms of error that we must minimize when selecting measurement methods and/or tools: Random error (CHANCE) Nonrandom error (BIAS AND CONFOUNDING)
SELECTING A MEASUREMENT METHOD
To minimize random error we choose a tool or method that is RELIABLE Reliability – The extent to which a measurement method or
tool produces the same results over several measurements AKA precision
Threats to Reliability Observer error: different measurements from the same or
different observers (i.e., blood pressure readings) Instrument error: different measurements from the
instrument itself due to extraneous environmental factors Subject error: different measurements from the natural
biological variability among humans
RELIABILITY
How to assess Reliability: Repeat measurements on the same subject.
Give a survey at two different time points Take blood pressure at two different time points
Use more than one observer. Assess inter-rater agreement
Have two different people take blood pressure
How to maximize Reliability: Standardize the measurement methods
Choose surveys and instruments that have been proven to be reliable
Train observers
Refine & update instruments
Repetition Averaging the measures can cancel out error.
ASSESSING & MAXIMIZING RELIABILITY
To minimize nonrandom error we choose a measurement method and/or tool that is VALID Validity – The extent to which a measurement method and/or
tool measures what is sets out to measure AKA Accuracy
Threats to Validity Observer bias: conscious or unconscious distortion in the
perception and/or reporting of the measurement Subject bias: bias-distortion of self-reported measurements due to
subjects beliefs and biases Hawthorne Eff ects and Social Desirability
Instrument bias: consistently biased or inaccurate measurements due to such things as worn parts or mechanical malfunction
Lack of a clear gold standard: No “best” instrument out there Abstract/behavioral variables: These things are diffi cult to
measure Patient satisfaction, pain, quality of life, intelligence
VALIDITY
Strategies for maximizing Validity Blinding
Ex) Do not allow physician who is taking blood pressure readings to know which subjects are receiving blood pressure medication
Deception Ex) Do not allow subjects to know which “group” they are in Give placebos
Instrument Calibration Make sure instruments are working properly
Use standardized/validated surveys and assessment tools Find these from literature searches Usually better to use “pre-made” surveys or instruments than
creating one from scratch
MAXIMIZING VALIDITY
Identify and define your variables at the VERY beginning of your study Don’t forget your control or confounding variables!
Using higher levels of measurement is better!
Choose instruments and data collection tools that are: RELIABLE – produce the same results over time (precise) VALID – produce results that represent “the truth”
(accurate)
IN SUMMARY
Go through “Introduction to Research Part 3: Sampling and Design”
NEXT STEPS
Hulley SB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing Clinical Research. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2001:37-49
Spector PE. Research Designs. Newbury Park, CA: SAGE Publications, Inc.; 1981. ISBN: 0-8039-1709-0
http://www.research-assessment-adviser.com/levels-of-measurement.html
REFERENCES