rigorous research on the effects of learning technology ... · titles commercially available. thus,...
Post on 20-Jul-2020
2 Views
Preview:
TRANSCRIPT
Rigorous Research on the Effects of Learning Technology:Implications for Policy
Barbara MeansCenter for Technology in Learning
SRI InternationalPresentation for the National Conference of State Legislators
San FranciscoJune 13 2009June 13, 2009
NATIONAL STUDY OF THE EFFECTIVENESS OFEDUCATIONAL TECHNOLOGY INTERVENTIONSEDUCATIONAL TECHNOLOGY INTERVENTIONS
Sponsored byI tit t f Ed ti S iInstitute of Education Sciences U.S. DEPARTMENT OF EDUCATION
Conducted by
WithWith
http://ies.ed.gov/ncee/pubs/20074005Some of this material is based on work supported by the US Department of Education under Contract ED-01-CO-0039/007 with Mathematica Policy Research Any opinions, conclusions or recommendations expressed in this presentation are those of the presenter and do not necessarily reflect the views of the Department of Education or Mathematica Policy Research.
The ContextThe Context
C i l d t i NCLB i d th Ed tiCongressional mandate in NCLB required the Education Secretary to:
IES tasked with implementing the mandatep g
Design contract made key decisions Focus on mathematics and reading software with commonly used
d di d hi l istandardized achievement tests as learning outcome Emphasize schools serving Title I students Cluster similar products; individual product results confidential Use an open self-nomination process to select products for inclusion
The Technologies Studied
1 2 3 4 6 8 9 10 11 12
g
K 1 2 3 4 5 6 7 8 9 10 11 12
A B C DAEarly Reading
1. Headsprout2. Plato Focus
BReading Comprehension
6. Academy of
CPre-Algebra
10. SmartMath11. Achieve Now12 L P l b
DAlgebra
13. Carnegie Cognitive Tutor14 PLATO Al b3. Waterford
4. Academy of Reading5. Destination
Reading
Reading7. Read 1808. KnowledgeBox9. Leaptrack
12. Larson Prealgebra 14. PLATO Algebra15. Larson Algebra
ead g
Copyright © 2007 SRI International
Research Questions
Does the use of software products increase reading and mathematics achievement?
Wh t diti d ti i t d ith ff ti
I li i i h h i i li d h
What conditions and practices are associated with effective software implementations?
Implicit in the way these questions were operationalized was the decision to study the kinds of effects on achievement that can be expected when a district purchases a reading or mathematics software product.
The EETI study attempted to test the effectiveness of typicaly p yptechnology implementations under real-world conditions.
Sample & Recruiting
Schools recruited through their districts: 33
6
Schools recruited through their districts: 33 districts, 132 schools, 439 teachers, and over 9,400 students participated. Within each school, volunteering teachersWithin each school, volunteering teachers
were randomly assigned to use the software or teach as normal. If necessary to run the software efficiently,If necessary to run the software efficiently,
schools received technology upgrades from the study budget. Vendors trained teachers before
implementation started and provided ongoing support as they deemed appropriate.
Learning Outcome MeasuresLearning Outcome Measures
1st Grade: SESAT & TOWRE
4th G d SAT R di 4th Grade: SAT Reading
6th Grade: SAT Math
Algebra: ETS Algebra Test
Findings: The Short StoryFindings: The Short Story
A hi i ifi l hi h i Achievement test scores were not significantly higher in classrooms using the selected reading and mathematics software products than in control classrooms. p
There was substantial variation between schools in theimpacts on student achievementimpacts on student achievement.
Although the study collected data on many school andclassroom characteristics and practices they did notclassroom characteristics and practices, they did notexplain the variation in impacts between schools onstudent achievement except for two factors in reading.
Experimental Results for First Grade
Source: Dynarski et al., 2007.
Issues of Alignment & Integrationg g
Vendors said that their products were aligned with nationalVendors said that their products were aligned with nationalstandards, and most of the products could be tailored to state ordistrict standards if teachers wished.
Teachers’ reports of whether or not the product had beenaligned with local standards did not predict effectiveness.
P d t t d ith th ti iti d d f Products competed with other activities and agendas forteachers’ and students’ time and attention.
12 out of 15 of the products are described as “supplemental”12 out of 15 of the products are described as supplemental instruction rather than the core curriculum.
There were cases where students were confused by differencesin representation or terminology between regular class workand the software.
Understanding the findings in contextg g
The EETI study involved a certain class of educational The EETI study involved a certain class of educational technology applications. Most of them were drill-and-practice at heart. The study does not show that these kinds of software can’t
be effective. Sometimes they were and sometimes they weren’tweren t. Report analyzes effectiveness in treatment teachers’ first year
of using the product.of using the product.
Recently released report on teachers’ second year of implementation found significant average effect size for twoimplementation found significant average effect size for two products.
Actual Headlines
EDUCATION WEEK: Federal Study Finds No Edge for Students Using Technology-Based Reading and Math Products
W hi t P t Software’s Benefits On Tests In Doubt: StudyWashington Post: Software s Benefits On Tests In Doubt: StudySays Tools Don’t Raise Scores
CoSN: Washington Update: USDE Releases ControversialCoSN: Washington Update: USDE Releases Controversial Study on Efficacy of Educational Software
Associated Press: No Benefit of Adding Math and SciencegSoftware Products to Classrooms, Study Says
CNN.com: Study: No benefit going high-tech for math and science
Study Mythsy y
Myth: The study showed that using technology to teach science produces b fi f hino benefit for achievement.
There was NO SCIENCE software in the study.
Myth: The U.S. Department of Education wanted this study to show that y p ytechnology is ineffective to justify cutting off educational technology funding. IES had hoped that this study would find convincing positive effects i d d h d ili f d i d fi ldin order to demonstrate the power and utility of randomized field trials.
Myth: The study authors had it in for educational technology.We wanted to do an exemplary RCT study.
Myth: The study did not take implementation factors into account.O 200 i l t ti i bl d i hOver 200 implementation variables were measured in each classroom using the software. Very few variables had a significant correlation with effect size.
Response to Industry Criticismsespo se to dust y C t c s s
CRITIQUE: “only surveyed 16 educational software titles out of hundreds of suchCRITIQUE: only surveyed 16 educational software titles out of hundreds of such titles commercially available. Thus, it is very hard to state that all educational software does not impact learning.”
RESPONSE: What the report actually says: This study . . . was not designed to assess the effectiveness of educational technology across its entire spectrum of uses, and the study’s findings do not
t l i b t t h l ’ ff ti b d th t d ’ t tsupport conclusions about technology’s effectiveness beyond the study’s context, such as in other subject areas.
M b l ti b t f i l d t ith th b t il blMoreover, by selecting a subset of commercial products with the best available prior evidence of effectiveness, the study was if anything biased toward finding a positive effect relative to what could be expected if all software products were included.
Response to Industry Criticisms cont’dp y
CRITIQUE: “the study itself essentially admits that the teachers using the softwareCRITIQUE: the study itself essentially admits that the teachers using the software did not feel comfortable with it, even after training.” [emphasis added]
RESPONSE: What the report actually says: “by the time of the first classroom observation (generally about mid fall) when mostby the time of the first classroom observation (generally about mid-fall) when most teachers had begun to use products, the proportion of teachers indicating that the initial training had adequately prepared them had declined from 95 percent at the end of initial training to 60 percent.”
Moreover, the product vendors did the teacher training and ongoing support themselves.
CRITIQUE: “the teachers reported that they used the product only 10% to 15% of all classroom time over the course of one year devoted to a particular subject.”
RESPONSE: This is what happens in the real world. Further, amount of time onRESPONSE: This is what happens in the real world. Further, amount of time on the product did not correlate with effect size in Grades 1, 6 or algebra. There was a small association in Grade 4.
So where does this leave us?So where does this leave us?
What can research tell us about whether to invest in educational technology?educational technology?
Insights from a contrasting case . . .
16
The Technology StudiedThe Technology StudiedV
Integration of Curriculum, Technology & PD
Key Strategy -Visual, dynamic, linked representations
“Democratizing access to the mathematics of change” - Jim Kaput17
A decade of SimCalc research
SRI’s large-scale RCT built on many years of prior research.
Studies grew in size from 10 students to 10 teachers to 10 schools to 10 regions.
Many research methods were used to investigate how best to support students’investigate how best to support students understanding of proportionality and rate.
Research Questions
Does SimCalc support complex math learning with a wide variety of teachers and students?
Which kinds of variability in classrooms are most important to the effectiveness of the programimportant to the effectiveness of the program with students?
This study attempted to test the robustness of theapproach under realistic conditions of scaling up.
Scaling Up g p
L l d i d t l Large-scale randomized control study
1,621 students, 95 teachers, 73 h lschools
Research team developed i l l t itcurriculum replacement units,
instructional strategies, assessments and teacher trainingg
Ongoing professional development provided by UT’s Dana Center
Experimental DesignSUMMER SCHOOL YEAR
TeacherCl I i
TreatmentPart 1
Mathematical
Professional Development Classroom Instruction
TeachSimCalc Replacement
Part 2Teaching with
Part 3PlanningTreatment
SimCalcIntervention
Foundations(TEXTEAMS)
2 days
pUnit
[Technology + Curriculum]
about 3 weeks
gSimCalc
3 days
g
1 day
Control
Teach
Rate and Proportionality
Part 1MathematicalFoundationsControl Rate and Proportionality
as usual
about 3 weeks
Foundations(TEXTEAMS)
2 days
StudentPretest
StudentPosttest
NS3
Slide 21
NS3 Added diagramNicole Shechtman, 5/23/2007
Student learning measuresg
P ti l R i P ti fProportional Reasoning as Preparation for Algebra and Calculus“Simple” proportionality “Simple” proportionality– Solving for a specific value in a/b=c/d or y=kx– Reading a specific value in a graph or table
“Complex” proportionality– Solving problems that invoke the function y=kx
• Function implies a mapping• Function implies a mapping• Within various representations• Making connections or comparisons across varying rates,
f ti d t tifunctions, and representations
NS2
Slide 22
NS2 slight editsNicole Shechtman, 5/23/2007
SimCalc students scored higher2323
The overall effect size (classroom level) was 0.84, considered large in ed cation st dieseducation studies
(t(93) = 9.1, P < 0.0001)
Outcomes by EthnicityOutcomes by Ethnicity
(Student-Level Data)
NS4
Slide 24
NS4 changed graph again (change did not persist from yesterday for some reason)Nicole Shechtman, 5/23/2007
Implications for Ed Tech Implementation
Move beyond promotion of technology for its own sakey p gy
Need to think in terms of Instructional Activity Systems Content Content
Learning activities, some of which are technology based
Articulation with other aspects of instruction
Teacher professional development and collaboration around implementation
Assessment for learning Assessment for learning
Use of data to refine the Instructional Activity System on an ongoing basis
Need ways to measure the things we care about
THE NATIONAL EDUCATIONAL TECHNOLOGY PLANTECHNOLOGY PLAN
Prior National Ed Tech Plans
19962004
2000
Planning Activitiesg
Outreach to key stakeholders
Technical Working GroupTechnical Working Group
Commissioned research and data analyses
Solicitation of white papers, viewpoints and comments
Public web site
Plan Development Processp
ED Internal Interviews
Information Gatheringgg
Potential Theme Draft Plan
Web Site with Potential
O h I f P bli
Identification Draft PlanPotential Themes/Goals
Outreach Activities
Input from Public & the Field
29
Four Bases for Goal Settingg
Research on
implementation
effective technology
implementation
New Education
New and emerging NATIONAL EDUCATIONAL Education
Prioritiesg g
technology capabilities
TECHNOLOGY GOALS
TechnologyTechnology-induced changes in
30
qeducation
requirements
Education Priorities
Rigorous college- and career-ready standards and high-qualityRigorous college and career ready standards and high quality assessments that are valid and reliable for all students, including English language learners and students with disabilities;
Pre-K-to college and career data systems that track progress andPre K to college and career data systems that track progress and foster continuous improvement;
Improvements in teacher effectiveness and in the equitable distribution of qualified teachers for all students, particularlydistribution of qualified teachers for all students, particularly students who are most in need;
Intensive support and effective interventions for the lowest-performing schools.p g
Universal access to affordable, high-quality education, including early learning and college
R&D and infrastructure to dramatically accelerate innovation − the creation and adoption of effective educational solutions.
New Technology Capabilitiesgy p
Web 2.0 capabilities for knowledge building Learning 24/7g Virtual schools and open educational
resourcesresources Immersive environments
M bil ti d i Mobile computing devices Sensor networks
Much wider options for educational content and ways to learncontent and ways to learn
Technology-induced changes in education requirementseducation requirements
Predictable functions, even those requiring complex processes, can be automated or outsourced.
Effective participation as a worker, citizen, or community member requires new skills, including use of technology toolstools.
The nature of subject domains—particularly in the sciences—is being changed by technologysciences is being changed by technology.
An Examplep
Biology Today(bioinformatics)(bioinformatics)
SS
BioCyc Pathway/Genome DatabasesThe BioCyc databases are divided into three tiers, based on their quality.
BioCyc Pathway/Genome DatabasesThe BioCyc databases are divided into three tiers, based on their quality.
Tier 1 databases have received person-decades of literature-based curation, and are the most accurate. Tier 2 and Tier 3 databases contain computationally predicted metabolic pathways, predictions as to which genes code for missing enzymes in metabolic pathways, and predicted operons.
BioCyc Tier 1: Intensively Curated Databases
Tier 1 databases have received person-decades of literature-based curation, and are the most accurate. Tier 2 and Tier 3 databases contain computationally predicted metabolic pathways, predictions as to which genes code for missing enzymes in metabolic pathways, and predicted operons.
BioCyc Tier 1: Intensively Curated DatabasesDatabase Scope Highlights
EcoCyc Escherichia coli K-12 MG1655Model-Organism Database
* Literature curation of complete genome* Information from 17,000 publications* Transcriptional regulatory network* Protein complexes and transporter functions* Gene Ontology assignments
MetaCyc Multiorganism Metabolic Pathway and Enzyme * 1 200 metabolic pathways
Bi C Ti 2 C t ti ll D i d D t b S bj t t M d t C tiBi C Ti 2 C t ti ll D i d D t b S bj t t M d t C ti
MetaCyc Multiorganism Metabolic Pathway and Enzyme Database
1,200 metabolic pathways* Pathways elucidated from 1,500 organisms* Extensive commentary* Information from 19,000 publications
BioCyc Tier 2: Computationally-Derived Databases Subject to Moderate Curation23 databases are available. [list of tier 2 DBs]
BioCyc Tier 3: Computationally-Derived Databases Subject to No Curation
389 databases are available [list of tier 3 DBs] and ready for adoption [more] by interested scientists for curation and updating.
C t Y O P th /G D t b
BioCyc Tier 2: Computationally-Derived Databases Subject to Moderate Curation23 databases are available. [list of tier 2 DBs]
BioCyc Tier 3: Computationally-Derived Databases Subject to No Curation
389 databases are available [list of tier 3 DBs] and ready for adoption [more] by interested scientists for curation and updating.
C t Y O P th /G D t bCreate Your Own Pathway/Genome Database
Interested in creating a pathway/genome DB for your genome? [learn more]
Create Your Own Pathway/Genome Database
Interested in creating a pathway/genome DB for your genome? [learn more]
Research on effective education technology implementationstechnology implementations
Technology by itself is not the answer.
We need to think about instructional activity systems, not individual technology tools by themselves.
Like any education innovation, those involving technology need tho ghtf l implementation plans reflecting a s stemicneed thoughtful implementation plans reflecting a systemic perspective.
It makes sense to try out technology-based innovations on It makes sense to try out technology based innovations on a modest scale with provision for examining both practice and outcomes to identify areas for improvement.
All too often our policies assume we can solve our fproblems if we just buy the right technology
Insert into Schools &Make T h Insert into Schools &
ClassroomsTech
Purchase
Better instruction & improved outcomes for students
38
We actually need something more like thisWe actually need something more like this
We actually need something more like thisWe actually need something more like this• What learning goal do we want to address?• What research-based approaches show promise in addressing this goal?addressing this goal?• What technology is part of that?• What else do we need to change to implement this intervention?
We actually need something more like thisWe actually need something more like this
• Try out the intervention in a small number of schools or classrooms.• Involve teachers in deciding how to implement.
We actually need something more like thisWe actually need something more like this
• Gather information on both implementation and learning outcomes.• Involve teachers and students in
l i h i ievaluating the intervention.
We actually need something more like thisWe actually need something more like this
• Aggregate data and feedback across pilot classrooms and schoolsclassrooms and schools.
We actually need something more like thisWe actually need something more like this
• Determine whether intervention merits scaling up.• Identify ways in which the intervention can be improved.
We actually need something more like thisWe actually need something more like this
• Plan for larger-scale implementation. I t t th i t ti ith th • Integrate the intervention with other
curriculum, assessment and pd activities.
Closing Thoughtg g
It’s time we moved from a “Silver Bullet” mentality to a “Continuous Improvement” mindset around educational technology.
top related