validity and reliability illustration of types of evidence of validity (figure 8.1)
TRANSCRIPT
Validity and Reliability
Validity and Reliability
Illustration of Types of Evidence of Validity (Figure 8.1)
Reliability and Validity (Figure 8.2)
Reliability of Measurement (Figure 8.3)
Methods of Checking Validity and Reliability (Table 8.2)
Validity (“Truthfulness”)
Method ProcedureContent-related evidence Expert judgmentCriterion-related evidence Relate to another measure of the same variableConstruct-related evidence Assess evidence on predictions made from theory
Reliability (“Consistency”)
Method Content Time Interval ProcedureTest-retest Identical Varies Give identical instrument twiceEquivalent forms Different None Give two forms of instrumentEquivalent forms/retest Different Varies Give two forms of instrument, with
time interval betweenInternal consistency Different None Divide instrument into halves and
score each or use KRObserver agreement Identical None Compare scores obtained by two or
more observers
Reliability Worksheet (Figure 8.5)
Educational Research
Chapter 13Experimental Research
Gay and Airasian
Experimental Research
13
Experimental Research
Key Characteristics of Experimental Designs Procedures are designed that
address potential threats to validity Internal External Construct Statistical Conclusion
Statistical comparisons of different groups are conducted
Selecting Participants and Assigning Them to Treatments Decide on the experimental unit of
analysis to be treated individual group or groups organization
Randomly assign individuals to groups control for extraneous characteristics that might influence the outcome
Topics Discussed in this Chapter Defining characteristics The experimental process Manipulation and control Threats to validity
Internal validity External validity
Group designs Single subject designs
Defining Characteristics Research designed to investigate cause and
effect relationships through the direct manipulation of an independent variable and control of extraneous variables Independent variable – the variable being
manipulated Dependent variable – the variable in which the
effect of the manipulation of the independent variable are observed
Researcher manipulation and control – choice of treatments, choice of a research design, use of specific procedures, etc.
Experimental Process Selection and definition of problem Selection of participants & instruments
Selection and assignment of participants Selection of research plan
Comparison of two approaches Comparison of new versus existing approaches Comparison of different amounts of single approach
Execution of research plan Sufficient exposure to treatment Operationally different treatments
Analysis of data Formulation of conclusions
Manipulation and Control Manipulation
The researcher’s decisions related to what will make up the independent variable
Active variables versus assigned variables Control
The researcher’s efforts to remove the influence of any extraneous variables that might have an effect on the dependent variable
The goal is to be assured the only differences between groups is that related to the independent variable
Internal Validity
9
Experimental Validity
Internal validity – the degree to which the results are attributable to the independent variable and not some other rival explanation
External/ecological validity – the extent to which the results of a study are generalizable
Relative importance of internal and external validity
Internal Validity
Threats to Internal Validity History Maturation Testing Instrumentation Statistical regression Differential selection of participants Mortality Selection-maturation interaction, etc.
Mortality Threat to Internal Validity (Figure 9.1)
Location Might Make a Difference (Figure 9.2)
Instrument Decay (Figure 9.3)
A Data Collector Characteristics Threat (Figure 9.4)
A Testing Threat to Internal Validity (Figure 9.5)
A History Threat to Internal Validity (Figure 9.6)
Could Maturation be at Work Here? (Figure 9.7)
The Attitude of Subjects Can Make a Difference (Figure 9.8)
Regression Rears Its Head (Figure 9.9)
Illustration of Threats to Internal Validity (Figure 9.10)
Techniques for Controlling Threats to Internal Validity (Table 9.1) Technique
Obtain More Obtain More ChooseStandardize Information Information Appropriate
Threat Conditions on Subjects on Details Design
Subject characteristics X X
Mortality X X
Location X X
Instrumentation X X
Testing X
History X X
Maturation X X
Subject attitude X X X
Regression X X
Implementation X X X
Threats to External Validity Pre-test treatment interaction Multiple treatment interference Selection treatment interaction Specificity of variables Treatment diffusion Experimenter effects Reactive arrangements
Artificial environment Hawthorne effect John Henry effect Placebo effect Novelty effect
Controlling for Extraneous Variables
Randomization Selection Assignment
Matching Identifying pairs of subjects “matched” on
specific characteristics of interest Randomly assigning subjects from each pair
to different groups Difficulty with subjects for whom no match
exists
A Randomized Posttest-Only Control Group Design, Using Matched Subjects (Figure 13.7)
Matching Process Based on Gender
ExperimentalGroup
ControlGroup
JohnJimJamesJoshJacksonJaneJohannaJulieJeanJeb
Controlling for Extraneous Variables Comparing homogeneous groups
Restricting subjects to those with similar characteristics
Problems related to restriction of generalization
Build the variable into the design (e.g., factorial design)
Using subjects as their own controls Multiple treatments across time Problem with carry-over effect
Analysis of covariance (ANCOVA)
Group Designs
Two major classes of designs Single-variable designs – one independent
variable Factorial designs – two or more independent
variables Three types of designs
Pre-experimental designs Experimental designs Quasi-experimental designs
Pre-Experimental Designs
Types One-shot case study One-group pretest-posttest design Static group comparison
Threats to internal validity – see Figure 13.1
Example of a One-Shot Case Study Design (Figure 13.1)
Example of a Randomized Posttest-Only Control Group Design (Figure 13.4)
Example of a Randomized Solomon Four-Group Design (Figure 13.6)
Example of a One-Group Pretest-Posttest Design (Figure 13.2)
Example of a Static-Group Comparison Design (Figure 13.3)
Example of a Randomized Posttest-Only Control Group Design (Figure 13.4)
True Experimental Designs
Types Pretest-posttest control group design Posttest only control group design Solomon four-group comparison
Threats to internal validity – see Figure 13.2
Example of a Randomized Pretest-Posttest Control Group Design (Figure 13.5)
Quasi-Experimental Designs
Types Non-equivalent control group design Time series design Counterbalanced design
Threats to internal validity – see Figure 13.2
Results (Means) from a Study Using a Counterbalanced Design(Figure 13.8)
Possible Outcome Patterns in a Time-Series Design(Figure 13.9)
Using a Factorial Design to Study Effects of Method and Class Size on Achievement (Figure 13.10)
Illustration of Interaction and No Interaction in a 2 by 2 Factorial Design (Figure 13.11)
Example of a 4 by 2 Factorial Design (Figure 13.13)
Effectiveness of Experimental Designs in Controlling Threats to Internal Validity (Table 13.1)
Subject Instru- Data Collec-Charac- Morta- Loca- ment tor Charac- Data Col- Matur- Atti- Regres- Implemen-
Design teristics lity tion Decay teristics lector Bias Testing History ation tudinal sion tation
One-shot casestudy – – – (NA) – – (NA) – – – – –
One group pre-posttest – ? – – – – – – – – – –
Static groupcomparison – – – + – – + ? + – – –
Randomized post-test-only controlgroup ++ + – + – – ++ + ++ – ++ –
Randomized pre-post-test controlgroup ++ + – + – – + + ++ – ++ –
Solomon four-group ++ ++ – + – – ++ + ++ – ++ –
Randomizedposttest onlycontrol group with matchedsubjects ++ + – + – – ++ + ++ – ++ –
Matching-onlypre-posttestcontrol group + + – + – – + + + – + –
Counterbalanced ++ ++ – + – – – ++ ++ ++ ++ –
Time-series ++ – + _ – – – – + – ++ –
Factorial withrandomization ++ ++ – ++ – – + + ++ – ++ –
Factorial without randomization ? ? – ++ – – + + + – ? –
KEY: (++) = strong control, threat unlikely to occur; (+) = some control, threat may possibly occur; (–) = weak control, threat likely to occur; (?) = can’t determine; (NA) = threat does not apply
Single-Subject Research Designs that can be applied when the sample
size is one Study behavior change in an individual as the result
of some treatment Subject serves as his or her own control
Rationale Sophistication of specific designs allows for the
control of internal validity threats Research is focused on therapeutic impact in clinical
settings, not contribution to a research base Group comparison designs are sometimes opposed
or unethical Group comparison designs are not possible
Single-Subject Research Concerns
External validity Low generalizability due to the nature of the design The effect of the baseline condition on the subsequent
effects of the treatment Threats can be lessened through replication
Internal validity Possible to control for most threats Repeated and reliable measures
Baseline stability Number of data points
Single-variable rule Specification or the nature and conditions of the
treatment
Single-Subject Research Designs
A B A Withdrawal A B A B Multiple Baseline Alternating Treatments
Data analysis and interpretation Based on visual inspection and analysis of a
graphic presentation of the results Criterion of effectiveness is clinical
significance, not statistical significance Debate about the use of statistical
procedures
Single-Subject Research Replication
Establishes the generalizability of findings Three types
Direct – same researcher with the same or different participants in a specific setting
Systematic – follows direct replication but with different researchers, behaviors, or settings
Clinical – development of treatment packages composed of tow or more interventions which have been found to be effective individually