validity and reliability chapters 8. validity and reliability validity is an important consideration...
TRANSCRIPT
![Page 1: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/1.jpg)
Validity and Reliability
Chapters 8
![Page 2: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/2.jpg)
Validity and Reliability
• Validity is an important consideration in the choice of an instrument to be used in a research investigation– It should measure what it is supposed to measure– Researchers want instruments that will allow them to make
warranted conclusions about the characteristics of the subjects they study
• Reliability is another important consideration, since researchers want consistent results from instrumentation– Consistency gives researchers confidence that the results
actually represent the achievement of the individuals involved
![Page 3: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/3.jpg)
Reliability
• Test-retest reliability
• Inter-rater reliability
• Parallel forms reliability
• Internal consistency (a.K.A. Cronbach’s alpha)
![Page 4: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/4.jpg)
Validity
• Face– Does it appear to measure what it purports to
measure?
• Content– Do the items cover the domain?
• Construct– Does it measure the unobservable attribute
that it purports to measure?
![Page 5: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/5.jpg)
Validity
• Criterion– Predictive – Concurrent
• Consequential
![Page 6: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/6.jpg)
Types of validity (cont.)
The construct
The instrument
Here the instrument samples some and only of the construct
![Page 7: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/7.jpg)
Types of validity
The instrument
The construct
Here the instrument samples all and more of the construct
![Page 8: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/8.jpg)
The construct
The instrument
Here the instrument fails to sample ANY of the construct
![Page 9: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/9.jpg)
The construct
The instrument
Here the instrument samples some but not all of the construct
![Page 10: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/10.jpg)
Perfection!
The construct and the instrument!
![Page 11: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/11.jpg)
Reliability and Validity
![Page 12: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/12.jpg)
Experimental Research Designs
![Page 13: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/13.jpg)
The (Never-Ending) Search for Causation
• Establishing causation among variables :
» Produces increased understanding of those variables
» Results in the ability to manipulate conditions in order
to produce desired changes
![Page 14: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/14.jpg)
Experimental Research
• Can demonstrate cause-and-effect very convincingly
• Very stringent research design requirements
• Experimental design requires:
» Random assignment to groups (experimental and
control)
» Independent treatment variable that can be applied to
the experimental group
» Dependent variable that can be measured in all groups
![Page 15: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/15.jpg)
Quasi-Experimental Research
• Used in place of experimental research when random
assignment to groups is not feasible
• Otherwise, very similar to true experimental research
![Page 16: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/16.jpg)
Causal-Comparative Research
• Explores the possibility of cause-and-effect relationships
when experimental and quasi-experimental approaches are
not feasible
• Used when manipulation of the independent variable is not
ethical or is not possible
![Page 17: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/17.jpg)
Fundamentals of Experimental and Quasi-Experimental Research
• Cause and effect:
» Incorporates a temporal element—the cause is a
condition that exists prior to the effect; effect is a
condition that occurs after the cause
» There exists a “logical connection”—cause-and-effect
is demonstrated when manipulation of the independent
variable results in differences in the dependent variable
(as evidenced by comparing the experimental group to
the control group)
![Page 18: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/18.jpg)
What Aids Our Causal Arguments?
• Theory– "causes certainly are connected to effects; but this is because our
theories connect them, not because the world is held together by cosmic glue. The world may be glued together by imponderables, but that is irrelevant for understanding causal explanation." Hanson, 1958.
• Temporal Elements
• Design– "No causation without manipulation" Rubin &
Holland
![Page 19: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/19.jpg)
Inferring Causality
Sir Bradford Hill
• Strength of association• Consistency• Specificity• Temporal order• Dose-Response (biological gradient)• Plausibility• Experimental evidence• Analogy
![Page 20: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/20.jpg)
Fundamentals of Experimental and Quasi-Experimental Research
• Random selection and random assignment :
» Distinguish between “selection” and “assignment”
» Random selection helps to assure population validity
» If you incorporate random assignment
Experimental research
» If you do not use random assignmentQuasi-experimental research
![Page 21: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/21.jpg)
Fundamentals of Experimental and Quasi-Experimental Research (cont’d.)
• When to use experimental research design :
» If you strongly suspect a cause-and-effect relationship
exists between two conditions, and
» The independent variable can be introduced to
participants and can be manipulated, and
» The resulting dependent variable can be measured for
all participants
![Page 22: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/22.jpg)
Internal and External Validity
• “Validity of research” refers to the degree to which the
conclusions are accurate and generalizable
• Both experimental and quasi-experimental research are
subject to threats to validity
• If threats are not controlled for, they may introduce error
into the study, which will lead to misleading conclusions
![Page 23: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/23.jpg)
Internal and External Validity
• “Validity of research” refers to the degree to which the
conclusions are accurate and generalizable
• Both experimental and quasi-experimental research are
subject to threats to validity
• If threats are not controlled for, they may introduce error
into the study, which will lead to misleading conclusions
![Page 24: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/24.jpg)
Threats to External Validity
• External validity—extent to which the results can be
generalized to other groups or settings
» Population validity—degree of similarity among
sample used, population from which it came, and target
population
» Ecological validity—physical or emotional situation or
setting that may have been unique to the experiment
» If the treatment effects can be obtained only under a limited
set of conditions or only by the original researcher the findings
have low ecological validity.
![Page 25: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/25.jpg)
Threats to Internal Validity
• Internal validity—extent to which differences on the
dependent variable are a direct result of the manipulation
of the independent variable» History—when factors other than treatment can exert influence
over the results; problematic over time
» Maturation—when changes occur in dependent variable that may be due to natural developmental changes; problematic over time
» Testing—also known as “pretest sensitization”; pretest may give clues to treatment or posttest and may result in improved posttest scores
» Instrumentation – Nature of outcome measure has changed.
![Page 26: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/26.jpg)
Threats to Internal Validity (cont’d.)
» Regression – Tendency of extreme scores to be nearer to the mean at retest
» Implementation-A group treated in an unintentional differential manner.
» Attitude-Hawthorne effect, compensatory rivalry.
» Differential selection of participants—participants are not selected/assigned randomly
» Attrition (mortality)—loss of participants
» Experimental treatment diffusion – Control conditions receive experimental treatment.
![Page 27: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/27.jpg)
Experimental and Quasi-Experimental Research Designs
• Commonly used experimental design notation :
» X1 = treatment group
» X2 = control/comparison group
» O = observation (pretest, posttest, etc.)
» R = random assignment
![Page 28: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/28.jpg)
Common Experimental Designs
• Single-group pretest-treatment-posttest design:
O X O
» Technically, a pre-experimental design (only one
group; therefore, no random assignment exists)
» Overall, a weak design
»Why?
![Page 29: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/29.jpg)
Common Experimental Designs (cont’d.)
• Two-group treatment-posttest-only design:
R X1 O
R X2 O
» Here, we have random assignment to experimental,
control groups
» A better design, but still weak—cannot be sure that
groups were equivalent to begin with
![Page 30: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/30.jpg)
Common Experimental Designs (cont’d.)
• Two-group pretest-treatment-posttest design:
R O X1 O
R O X2 O
» A substantially improved design—previously
identified errors have been reduced
![Page 31: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/31.jpg)
Common Experimental Designs (cont’d.)
• Solomon four-group design:
R O X1 O
R O X2 O
R X1 O
R X2 O
» A much improved design—how??
» One serious drawback—requires twice as many
participants
![Page 32: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/32.jpg)
Common Experimental Designs (cont’d.)
• Factorial designs:
R O X1 O
R O X2 O
R O X1 O
R O X2 O
» Incorporates two or more factors
» Enables researcher to detect differential differences (effects apparent only on certain combinations of levels of independent variables)
![Page 33: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/33.jpg)
Common Experimental Designs (cont’d.)
• Single-participant measurement-treatment-measurement designs:
O O O | X O X O | O O O
» Purpose is to monitor effects on one subject
» Results can be generalized only with great caution
![Page 34: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/34.jpg)
Common Quasi-Experimental Designs
• Posttest-only design with nonequivalent groups:
X1 O
X2 O
» Uses two groups from same population
» Questions must be addressed regarding equivalency of groups prior to introduction of treatment
![Page 35: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/35.jpg)
Common Quasi-Experimental Designs (cont’d.)
• Pretest-posttest design with nonequivalent groups:
O X1 O
O X2 O
» A stronger design—pretest may be used to establish
group equivalency
![Page 36: Validity and Reliability Chapters 8. Validity and Reliability Validity is an important consideration in the choice of an instrument to be used in a research](https://reader035.vdocuments.us/reader035/viewer/2022062309/56649f065503460f94c1c80a/html5/thumbnails/36.jpg)
Similarities Between Experimental and Quasi-Experimental Research
• Cause-and-effect relationship is hypothesized
• Participants are randomly assigned (experimental) or
nonrandomly assigned (quasi-experimental)
• Application of an experimental treatment by researcher
• Following the treatment, all participants are measured on
the dependent variable
• Data are usually quantitative and analyzed by looking for
significant differences on the dependent variable