ixl california dashboard research report...california research report 10 table 2. analysis on...

The Impact of IXL Math and IXL ELA on the California School Dashboard

CALIFORNIA RESEARCH REPORT

IXL LEARNING 777 Mariners Island Blvd., Suite 600, San Mateo, CA 94404

650-372-4040 | www.ixl.com

TABLE OF CONTENTS

_____________________________________________________________________________

Executive Summary 1

Objectives 2

Study Design 3

Methodology 5

Study Sample 5

Analysis 5

Results 5

TheIXLMathEffectontheCaliforniaDashboardforMath 5

TheIXLMathEffectontheCaliforniaDashboardforELA 7

References 8

Appendix 9

California Research Report

1

Executive Summary _____________________________________________________________________________

IXLisapersonalizedlearningplatformdesignedtohelpstudentsbuildacademicskills.ToquantifytheeffectivenessofIXLMathandIXLEnglishLanguageArts(ELA)onimprovingstudentlearning,researchersconductedanevaluationstudyacrossthestateofCaliforniausingaquasi-experimentalcontrolgroupdesign.ThisstudyaimedtounderstandtheimpactofIXLMathandIXLELAontheCaliforniaSchoolDashboardforMathandELA.TheCaliforniaDashboardshowsschoolperformanceonindicatorsincludedinthestate’sschoolaccountabilitysystem.Foreachindicator,aschoolisassignedacolorrating(fromhighesttolowest):Blue,Green,Yellow,Orange,orRed.StudiedsamplesusedIXLbetween2015and2019.Thekeyfindingsinclude:

• SchoolsthatusedIXLMathdemonstratedbettermathperformancethannon-IXLschools,basedontheirhigherprobabilityofachievingBlueorGreenratingsontheCaliforniaMathDashboard.


2

• SchoolsthatusedIXLELAdemonstratedbetterELAperformancethannon-IXLschools,basedontheirhigherprobabilityofachievingBlueorGreenratingsontheCaliforniaELADashboard.

Objectives _____________________________________________________________________________

ThisresearchaimedtoevaluatetheimpactofIXLMathandIXLELAontheCaliforniaSchoolDashboardforMathandEnglishLanguageArtsandLiteracy.Thisevaluationconsideredtworesearchquestions:

RQ1:WhatistheassociationbetweentheusageofIXLMathandschoolacademicperformanceasmeasuredbytheCaliforniaSchoolDashboardforMath?

RQ2:WhatistheassociationbetweentheusageofIXLELAandschoolacademicperformanceasmeasuredbytheCaliforniaSchoolDashboardforEnglishLanguageArtsandLiteracy?


3

Study Design _____________________________________________________________________________

ThisstudyevaluatedtheimpactofIXLMathandIXLELAonstudentacademicperformanceattheschoollevelinelementaryandmiddleschoolsinCalifornia.

TheCaliforniaDashboardshowsschoolperformanceonstateandlocalindicatorsincludedinCalifornia’sschoolaccountabilitysystem.Foreachindicator,aschoolisassignedacolor(i.e.,Blue,Green,Yellow,Orange,orRed)indicatingtheirperformancelevelsbasedonafive-by-fivecoloredtable(Figure1),acombinationoftheschool’scurrentperformancestatus(i.e.,VeryHigh,High,Medium,Low,orVeryLow),andchangestatus(i.e.,IncreasedSignificantlyfromPriorYear,IncreasedfromPriorYear,MaintainedfromPriorYear,DeclinedfromPriorYear,orDeclinedSignificantlyfromPriorYear).Forexample,ifaschool’s2019mathperformancelevelis“Medium”onthe2019SmarterBalancedAssessmentsandthisperformanceis“Maintainedfromtheprioryear,”thisschoolisassignedaYellowratingontheDashboardforMath(Figure1).AsshowninFigure1,Bluerepresentsthehighestperformancelevel,followedbyGreen,whileRedrepresentsthelowestperformancelevel.MoredetailsabouttheCaliforniaDashboardcanbefoundontheCaliforniaDepartmentofEducationwebsite(CaliforniaDepartmentofEducation,2019).

Figure 1. Dashboard determination (copied from California Department of Education, 2019)


4

Thisstudyusedaquasi-experimentalcontrolgroupdesign(Figures2-4)andcomparedtheperformanceofthetreatmentgroupandthecomparisongrouponthe2019,2018,and2017DashboardsforMathandELA,respectively.Forthe2019Dashboardanalysis,thetreatmentgroupincluded1-,2-,3-,or4-yearIXLschoolsthatusedIXLbetween2015and2019(Figure2).Forthe2018Dashboardanalysis,thetreatmentgroupincluded1-,2-,or3-yearIXLschoolsthatusedIXLbetween2015and2018(Figure3).Forthe2017Dashboardanalysis,thetreatmentgroupincluded1-or2-yearIXLschoolsthatusedIXLbetween2015and2017(Figure4).IXLusagebytheschoolsinthisstudyrangedfromlessthanoneminuteperstudent,perweektoover100minutesperstudent,perweek.SchoolsthatdidnotuseIXLduringthestudyperiod(non-IXLschools)wereusedasthecomparisongroup.Itisnotedthatnon-IXLschoolsmayormaynothaveusedotheronlineprogramsduringthestudyperiod.

Figure 2. Study design for the 2019 Dashboard analysis



5


Methodology_____________________________________________________________________________

STUDY SAMPLE

Thisstudyanalyzeddatafromallschoolsthatincludedstudentsinatleastonegradelevelfrom3-8.ResearchersexaminedDashboardperformanceformathandELAattheschoollevel.Intotal,therewere809,704,and695schoolsidentifiedasIXLMathschoolsforthe2019,2018,and2017MathDashboardanalysis,respectively;therewere426,315,and276schoolsidentifiedasIXLELAschoolsforthe2019,2018,and2017ELADashboardanalysis,respectively.Table1intheAppendixshowsthenumberofstudysamplesintotalandbyschoollocation(i.e,city,suburb,town,ruralareas,asdefinedbytheInstituteofEducationSciences).TheDashboardperformance,enrollment,anddemographicdatawereobtainedfromtheCaliforniaDepartmentofEducationwebsiteandtheInstituteofEducationSciences.

ANALYSIS

Thisstudycomputedchi-squarestatisticsandoddsratiostatisticstoexaminewhetherIXLschoolsweremorelikelytoreceiveperformancelevelsofBlueorGreencolorsontheirDashboardforMathandforELAthannon-IXLschools.Thetestsofstatisticalsignificancewereexamined.Ap-valuelessthan.05indicatesstatisticalsignificance—evidencethattheIXLeffectexists.

Results_____________________________________________________________________________

THE IXL MATH EFFECT ON THE CALIFORNIA DASHBOARD FOR MATH

IXLMathusagewasassociatedwithbettermathperformance.Specifically,schoolsthatusedIXLMathhadsignificantlyhigherprobabilitiesofreceivingperformancelevelsofBlueorGreenontheDashboardforMaththannon-IXLschools.


6

2017 Dashboard for MathFigure5showsthe2017DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2017DashboardforMath(x2=32.35,p<.01).Specifically,43%ofIXLschoolsthatusedIXLMathfor1or2yearsduringthe2015-2017schoolyearsreceivedperformancelevelsofBlueorGreenonthe2017DashboardforMath,comparedtojust32%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.6timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2017DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).

2018 Dashboard for MathFigure6showsthe2018DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2018DashboardforMath(x2=27.3,p<.01).Specifically,45%ofIXLschoolsthatusedIXLMathfor1,2,or3yearsduringthe2015-2018schoolyearsreceivedperformancelevelsofBlueorGreenonthe2018DashboardforMath,comparedtojust35%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.5timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2018DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).

Figure 5. 2017 Dashboard for Math with 5 colors (left) and with 2 color categories (right)



7

THE IXL ELA EFFECT ON THE CALIFORNIA DASHBOARD FOR ELA

IXLELAusagewasassociatedwithbetterELAperformance.Specifically,schoolsthatusedIXLELAhadsignificantlyhigherprobabilitiesofreceivingperformancelevelsofBlueorGreenontheDashboardforELAthannon-IXLschools.

2017 Dashboard for ELA Figure8showsthe2017DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2017DashboardforELA(x2=16.58,p<.01).Specifically,44%ofIXLschoolsthatusedIXLELAfor1or2yearsduringthe2015-2017schoolyearsreceivedperformancelevelsofBlueorGreenonthe2017DashboardforELA,comparedtojust32%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.7timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2017DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).

Figure 8. 2017 Dashboard for ELA with 5 colors (left) and with 2 color categories (right)

2019 Dashboard for Math Figure7showsthe2019DashboardforMathwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLMathusageandthe2019DashboardforMath(x2=10.33,p<.01).Specifically,44%ofIXLschoolsthatusedIXLMathfor1,2,3,or4yearsduringthe2015-2019schoolyearsreceivedperformancelevelsofBlueorGreenonthe2019DashboardforMath,comparedtojust38%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.3timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2019DashboardforMaththannon-IXLschools(seeTable2intheAppendixfordetails).



8

2019 Dashboard for ELAFigure10showsthe2019DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2019DashboardforELA(x2=12.30,p<.01).Specifically,47%ofIXLschoolsthatusedIXLELAfor1,2,3,or4yearsduringthe2015-2019schoolyearsreceivedperformancelevelsofBlueorGreenonthe2019DashboardforELA,comparedtojust39%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.4timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2019DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).


2018 Dashboard for ELAFigure9showsthe2018DashboardforELAwith5colors(left)andwith2colorcategories(right).TherewasastatisticallysignificantassociationbetweenIXLELAusageandthe2018DashboardforELA(x2=15.06,p<.01).Specifically,50%ofIXLschoolsthatusedIXLELAfor1,2,or3yearsduringthe2015-2018schoolyearsreceivedperformancelevelsofBlueorGreenonthe2018DashboardforELA,comparedtojust39%ofnon-IXLschoolsreceivingperformancelevelsofBlueorGreen.IXLschoolswere1.6timesmorelikelytoreceiveperformancelevelsofBlueorGreenonthe2018DashboardforELAthannon-IXLschools(seeTable3intheAppendixfordetails).



9

Appendix_____________________________________________________________________________

Table 1. Characteristics for the state of California and IXL schools

Math ELA

State IXL schools State IXL schools

California Dashboard 2019

• Thenumberofschools 6,511 809 6,420 426

• %ofschoolsincities 42% 34% 42% 34.5%

• %ofschoolsinsuburbs 42% 46% 42% 45.5%

• %ofschoolsintowns 6% 9% 6% 9%

• %ofschoolsinruralareas 9% 11% 9% 11%



• %ofschoolsincities 42% 37% 42% 36%

• %ofschoolsinsuburbs 42% 42% 42% 43%

• %ofschoolsintowns 6% 9% 6% 8%




• %ofschoolsincities 42% 37% 42% 38%

• %ofschoolsinsuburbs 42% 43.5% 42% 45%

• %ofschoolsintowns 6% 8.5% 6% 5%


References_____________________________________________________________________________ CaliforniaDepartmentofEducation(2019).2019CaliforniaSchoolDashboardTechnicalGuide. Fromhttps://www.cde.ca.gov/ta/ac/cm/documents/dashboardguide19.pdf

https://www.cde.ca.gov/ta/ac/cm/documents/dashboardguide19.pdf


10

Table 2. Analysis on Dashboard for Math

Table 3. Analysis on Dashboard for ELA

Year Schools

% of schools that received Green or Blue on Dashboard

% of schools that received Red, Orange, or Yellow on Dashboard

Chi-square Odds Ratio

2019IXLSchools 44% 56%

10.33* 1.3*Non-IXLSchools 38% 62%


27.33* 1.5*Non-IXLSchools 35% 65%


32.35* 1.6*Non-IXLSchools 32% 68%

Year Schools

% of schools that received

Green or Blue on Dashboard

% of schools that received Red, Orange, or Yellow on Dashboard

Chi-square Odds Ratio


12.30* 1.4*Non-IXLSchools 39% 61%


15.06* 1.6*Non-IXLSchools 39% 61%


16.58* 1.7*Non-IXLSchools 32% 68%

Note. *p<.05. For 2019 analysis, IXL schools refer to schools that used IXL Math for 1, 2, 3 or 4 years. For 2018 analysis, IXL schools refer to schools that used IXL Math for 1, 2, or 3 years. For 2017 analysis, IXL schools refer to schools that used IXL Math for 1 or 2 years.

Note. *p<.05. For 2019 analysis, IXL schools refer to schools that used IXL ELA for 1, 2, 3 or 4 years. For 2018 analysis, IXL schools refer to schools that used IXL ELA for 1, 2, or 3 years. For 2017 analysis, IXL schools refer to schools that used IXL ELA for 1 or 2 years.

ixl california dashboard research report...california research report 10 table 2. analysis on...

Documents