sotl: research design nancy gourash bliwise, phd step faculty development workshop emory university...
TRANSCRIPT
SoTL: Research Design
Nancy Gourash Bliwise, PhDSTEP Faculty Development Workshop
Emory UniversityJune, 2014
Research Design in SoTL
Commitment to Outcomes Research
Learning Objectives
Identify the basic elements of four common quasi-experimental designs used in outcomes research
Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs
Be able to identify ethical issues in educational outcomes research and resources for IRB review
Outcome Research is …
Important
Rewarding
Necessary
And often…
Difficult
Time-consuming
Small effect sizes
Type of Research
Descriptive/Qualitative
Question: What Is? descriptive, generative naturalistic,
observational, constructivist
Samples: small, targeted Assessment: interviews,
observations inductive formative, “thick
description”, expansive
Inferential/Quantitative
Question: What Works? hypothesis testing,
confirmatory empirical, statistical,
comparative Samples: large,
representative Assessment: scores, rates
deductive summative, precise,
reliable
Formal Design
Focus of SoTL Actions as educators Student learning and performance
Question guides choice of design
Source Material: Cook & Campbell (1979) – Quasi-Experimentation:
Design & Analysis Issues for Field Settings Trochim (2006) – Research Methods Knowledge Base
Online: http://www.socialresearchmethods.net/kb/ NSF – User-friendly Handbook for Mixed Methods
Designs Online: http://
www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153
Optimal Design -- Descriptive
No single preferred design
Common features Focus on individual and collective experiences of
students Context is described Search for patterns Developmental Multiple assessment methods and assessment
opportunities Subgroup analysis
Optimal Design -- Inferential
Randomized “Clinical” Trial
Real Life
Quasi-experimental designs Non-random assignment to groups
Frequently used (and effective designs Single group repeated measures design
Non-equivalent groups pre-test/post-test design
Cohort design
Single group crossover (ABAB) design
Uninterpretable Designs
One-shot case study
Single-group pre-test/post-test design
Static group comparison
X O
O1 X O2
X O O
Single Group Repeated Measures/Time Series
Add multiple observations to track learning/change
Goal is to study the trajectory of learning Depth of knowledge Understanding of concepts Application Integration/synthesis Testing of ideas
O1 O2 X O3 O4 O5 O6
Non-equivalent Groups Pre-test/Post-test Design
Compare two classes New vs. standard
Willing/interested colleagues Commitment to method
Pre-test/Post-test essential
Targeted measurement
O1 X O2
O1 O2
Cohort Design
What if you are the only one who teaches this? Or colleagues are not “willing”
Compare outcomes to yourself But be very, very careful
Risk – nothing else can change Assumes stable cohort
O1 X
O2
Single Group Cross-over Design
Multiple units/cases
Replicate (ABAB)
Standard assessment strategy
XA1O1O2O3 XB1O4O5O6 XA2O7 O8O9XB2O10O11O12
Less “Control”
Rule out competing explanations of findings
Argument/Logic
Measurement Student demographics Possible pre-test differences Characteristics of “treatment”
Design
Statistical controls
Preventive action
Sound Measurement
Existing standards
Independence of observations, where possible
Conceptual comparison
Multiple dimensions/skills
Your Turn…
Choose design that best “fits” your case
Type of class
Possibility of comparison
Assessment strategy
IRB Review
Educational research Human participants
Type of research Non-research/exempt/expedited
Primary Issues Coercion Informed consent
Everything IRB
www.irb.emory.edu
Social-behavioral studies
Multi-disciplinary team Rebecca Rousselle Carol Corkran
Non-Research
New category
Systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge
PI can make this determination
If it is just for your own evaluation, it is not research
Exempt vs. Expedited
Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant.
IRB determines this
Coercion
Major issue
Students must be free not to participate
Informed Consent
Description of the purpose of the study
Who is being studied
What will be required
Risks/benefits
Compensation
Legal review
Contact information
Summary of findings