conducting randomized evaluations in schools
DESCRIPTION
Conducting Randomized Evaluations in Schools. Lisa Markman Associate Director Education Research Section Princeton University. Education Research Section. Promotes the use of research in education decision-making. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/1.jpg)
Conducting Randomized Evaluations in Schools
Lisa MarkmanAssociate DirectorEducation Research SectionPrinceton University
![Page 2: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/2.jpg)
Education Research Section
• Promotes the use of research in education decision-making.
• Conducts and promotes high quality education research through the use of experimental and quasi-experimental research designs
![Page 3: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/3.jpg)
No Child Left Behind, Schools and Research
• No Child Left Behind Act (2001) requires programs and practices to be based on “scientifically based research”.
• Term “scientifically based research” appears throughout the law over 100 times.
• Applies to curricular interventions, professional development programs, psychosocial interventions and supplemental education services.
![Page 4: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/4.jpg)
What is Scientifically Based Research ?
The term scientifically based research means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs
(According to NJ Dept. of Ed)
![Page 5: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/5.jpg)
Necessary Components of SBR
• Employs systematic, empirical methods that draw on observation or experiment
• Involves rigorous data analysis that are adequate to test the stated hypotheses and justify the general conclusions drawn
• Relies on measurement or observable methods that provide reliable and valid data
• Is evaluated using experimental or quasi-experimental designs with appropriate control or comparison groups (with a preference for random assignment)
• Reports results with sufficient detail and clarity• Has been subject to external (peer) review
(According to NJ Dept. of Ed)
![Page 6: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/6.jpg)
Evaluations in Education
Undesirable Features of Many Evaluations
• No comparison group or comparison group is not comparable to treatment group
• Only “successfully” treated schools/students are included in the analysis
• The evaluators are not independent of the developers of the program
Desirable features of Evaluations
• Participants are randomly assigned to “treatment” status
• All initial participants are included in the analysis
• The evaluators are independent of the developers of the program
![Page 7: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/7.jpg)
Recipe for an Ideal Measurement
Step 1: Randomly assign students.
Step 2: Obtain PERFECT test.
Step 3: Test students in fall.
Step 4: Test students in spring.
Step 5: Set non-school parameters back to original conditions.
Step 6: Repeat Steps 3 & 4.
Step 7: Impact =
(Step 4 – Step 3) Year 2 - (Step 4 – Step 3) Year 1
![Page 8: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/8.jpg)
The “2nd Best” OptionRandomization
Randomly assign some students, classes, school districts, or states to the reform in question; the others are randomly assigned to the status quo (or another reform).
![Page 9: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/9.jpg)
Why Randomize ?
To obtain an appropriate control group
Allows one to answer what would happen in the absence of the intervention
(creates a credible counterfactual)
![Page 10: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/10.jpg)
Common Concerns About Randomized Evaluations
• It is too disruptive
• Unethical to deny service to some children
• Legal obligations require providing services to all children
• It does not answer all questions
• Fear of “random”
![Page 11: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/11.jpg)
Is It Too Disruptive ?
• It can be.
• Good researchers work with teachers, schools and school districts to understand school culture, and school needs.
![Page 12: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/12.jpg)
Example of a Randomized Research Evaluation
• To determine the effectiveness of a computerized math curriculum
• 3 Urban School districts
• Randomize on the class level by school
![Page 13: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/13.jpg)
Teacher Period 1 Period 2 Period 3 Period 4
A Alg 1 Alg 1A Alg 1
B Alg 1 Alg 1A Alg 1A
C Alg 1A Alg I
D Alg I Alg 1 Alg 1A
Randomized Evaluation of A Computerized Math Curriculum
Current School Schedule
![Page 14: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/14.jpg)
Teacher Period 1 Period 2 Period 3 Period 4
A Alg 1 Alg 1A Alg 1
B Alg 1 Alg 1A Alg 1A
C Alg 1A Alg I
D Alg I Alg 1 Alg 1A
Teacher Period 1 Period 2 Period 3 Period 4
A Regular Regular LAB
B LAB LAB Regular
C Regular Regular
D Regular Regular LAB
Randomized Evaluation of A Computerized Math Curriculum
Current School Schedule
Below is a sample schedule that would be returned to each school after random assignment
![Page 15: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/15.jpg)
It’s Unethical/Illegal to Deny Service
• We don’t know if program/reform actually works
• Ways around completely excluding all students
• All children receive “something” or can contrast two programs
![Page 16: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/16.jpg)
It Doesn’t Answer All Questions
• True
• Benefit of Mix-Methods Approach
![Page 17: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/17.jpg)
The Fear of “Random”
• Usually can not treat all students
• Way to organize groups to get meaningful information
![Page 18: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/18.jpg)
To Sum….
• Need more high-quality impact evaluations to help guide everyday decisions
• Conducting evaluations does not have to be super disruptive
• Schools should make sure that researchers work with them
• It’ll take cooperation from all to improve the quality of education research on impacts
![Page 19: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/19.jpg)
Information for Librarians
• What Works Clearinghouse
• Education Databases
![Page 20: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/20.jpg)
What Works Clearinghouse• Created as a resource for decision makers in education. Provides
databases and reports regarding the effectiveness scientifically based programs and practices.
• Current topics – – Adult Literacy, – Character Education, – Delinquency, Disorderly and Violent behavior, – Dropout Prevention, – English Language Learners, – Math (Elementary and Middle School)– Peer Assisted Learning– Reading, – Early Childhood
• http://www.whatworks.ed.gov
![Page 21: Conducting Randomized Evaluations in Schools](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814866550346895db574cf/html5/thumbnails/21.jpg)
Examples of Education Databases
• National Center for Education Statistics (NCES) – National Assessment of Educational Progress (NAEP) – High School and Beyond (HS&B)– The Common Core of Data (CCD)
– http://nces.ed.gov
• National Longitudinal Study of Youth (NLSY)– NLSY97– NLSY79
– http://www.bls.gov/nls
– Check websites for a list of all databases