INDIANATEACHERAPPRAISALSYSTEMSOFSUPPORT(INTASS)STATEBOARDOFEDUCATIONRESEARCHREPORTFORTEACHEREVALUATIONPLANIMPLEMENTATION

2012-13THROUGH2015-16

• REPORTISONEOFSEVERALONTEACHEREVALUATIONININDIANASUPPORTEDBYTHEINDIANASTATEBOARDOFEDUCATIONANDTHEJOYCEFOUNDATION

• RESEARCHDESIGNEMPLOYEDAMULTI-YEARAPPROACHWITHADATASETINCLUDINGMORETHAN2,352,964+ STUDENTRECORDS/199,588+EDUCATORRECORDS• DATASETINCLUDESINDIVIDUALEDUCATORANDSTUDENTATTRIBUTES,DISTRICT,SCHOOL,CLASSROOMDEMOGRAPHICS,TEACHEREVALUATIONPLANQUALITY,TEACHEREVALUATIONRATINGSANDSTUDENTOUTCOMES

• RESEARCHDESIGNINCLUDESMULTIPLEMODELSANDSTATISTICALANALYSESTOADDRESSRESEARCHQUESTIONSCONCERNINGTEACHERRATINGS,STUDENTOUTCOMESWITHPLAN QUALITY,CLASSROOM,SCHOOL,ANDDISTRICTS,ANDSTUDENTANDEDUCATORDEMOGRAPHICVARIABLES

CONTEXT:ACCOUNTABILITYENVIRONMENT2012-2016

•SENATEBILL1CHANGEDEXPECTATIONSFOREDUCATOREVALUATION•CHANGESINASSESSMENT,STANDARDSANDACCOUNTABILITY

•PERVASIVEISSUESWITHASSESSMENTADMINISTRATION

• INCONSISTENCYIN:üTEACHEREVALUATIONPOLICY&GUIDANCEINTERPRETATION

üDEVELOPMENTANDIMPLEMENTATIONOFTEACHEREVALUATIONPLANS

RESEARCHMODELCOMPONENTS

•DISTRICT,SCHOOL,EDUCATOR,ANDSTUDENTDEMOGRAPHICS,STUDENTDAYSATTENDANCE,YEARSOFPLANIMPLEMENTATION

•STUDENTANDEDUCATOROUTCOMEMEASURESRANGINGFROMSTUDENTSCALESCORES,STUDENTPROFICIENCYLEVELS,STUDENTGROWTHPERCENTILES ANDBOTHEDUCATORIGMANDSUMMATIVERATINGS

•STATE,DISTRICT,CLASSROOM,EDUCATOR,ANDSTUDENTLEVELANALYSES

LEVELSOFSIGNIFICANCETHEMULTIPLEMODELSANDANALYSESOFTENPROVIDEDFINDINGSTHATDIFFEREDINCONSISTENCYANDSIGNIFICANCE.THEFOLLOWINGSCALEWASUSEDTOINTERPRETTHE“COLLECTIVE”SIGNIFICANCE*OFSPECIFICVARIABLESWITHSTUDENTANDTEACHEROUTCOMES**:

üNONE:SIGNIFICANTINNONEOFTHESPECIFICATIONS&YEARS

üWEAK:SIGNIFICANTIN25%ORLESSOFTHESPECIFICATIONS&YEARS

üMEDIUM:SIGNIFICANTIN75%ORLESSOFTHESPECIFICATIONS&YEARS

üSTRONG:SIGNIFICANTINOVER75%OFTHESPECIFICATIONS&YEARS*WEAK&MIXEDRESULTSWERENOTFELTTOALLOWDISCERNIBLEPATTERNSOFSIGNIFICANCE

**1)STUDENTOUTCOMESREFERTOMETRICSDERIVEDFROMSTUDENTISTEPPERFORMANCEAND2)TEACHEROUTCOMESREFERTOIGMAND/ORSUMMATIVERATINGS.

STUDENTDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS

• THEREISANEGATIVERELATIONSHIPBETWEENSTUDENTFREEANDREDUCEDLUNCHSTATUS,RACIAL/ETHNICMINORITYGROUPMEMBERSHIP,ANDSTUDENTOR TEACHEROUTCOMES.• THEREISAPOSITIVERELATIONSHIPBETWEENSTUDENTATTENDANCEANDSTUDENTORTEACHEROUTCOMES.• HAVINGANIEPISNEGATIVELYASSOCIATEDWITHISTEPSCORESANDSUMMATIVERATINGS.• THEREISANEGATIVERELATIONSHIPBETWEENLIMITEDENGLISHPROFICIENTANDISTEPSCORESBUTAPOSITIVERELATIONSHIPWITHISTEPGROWTH.• THEREISNOTADISCERNIBLERELATIONSHIPBETWEENDISTRICTGROWTHWEIGHT,STUDENTDEMOGRAPHICSANDSTUDENTORTEACHEROUTCOMES.

TEACHERDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS

• THEREISNORELATIONSHIPBETWEENTEACHERRACE/ETHNICITYANDSTUDENTORTEACHEROUTCOMES.

• THEREAREPOSITIVERELATIONSHIPSBETWEENTEACHERSALARY,STUDENT PRIORISTEPSCORESANDATTENDANCEWITHSTUDENTANDTEACHEROUTCOMES.

• TEACHERYEARSOFEXPERIENCEHAVEDIFFERENTRELATIONSHIPSWITHSTUDENTANDTEACHEROUTCOMES.

• TEACHERGENDER,GRADEASSIGNMENTSHAVEDIFFERENTRELATIONSHIPWITHSTUDENTANDTEACHEROUTCOMES.

• THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHT,TEACHERDEMOGRAPHICSANDSTUDENTORTEACHEROUTCOMES.

DISTRICTDEMOGRAPHICS,STUDENTOUTCOMESANDTEACHEREVALUATION

•THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHT,DISTRICTDEMOGRAPHICS,ANDSTUDENTORTEACHEROUTCOMES

•THEREISASTRONGRELATIONSHIPBETWEENDISTRICTSTUDENTOUTCOMESINCLUDINGPRIORYEARISTEPSCORESANDTEACHEROUTCOMES

•THEREISARELATIONSHIPBETWEENDISTRICTFREEANDREDUCEDLUNCH,IGMRATINGSANDSUMMATIVERATINGS

PRINCIPALCHARACTERISTICS,STUDENTOUTCOMESANDTEACHEREVALUATIONRATINGS

• PRINCIPALSRATEDEFFECTIVERATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE

• PRINCIPALSRATEDNEEDSIMPROVEMENTRATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE

• PRINCIPALSRATEDINEFFECTIVERATETHEIRTEACHERSLOWERTHANPRINCIPALSRATEDHIGHLYEFFECTIVE

• THEREISNODISCERNIBLERELATIONSHIPWITHDISTRICTGROWTHWEIGHTANDSTUDENTORTEACHEROUTCOMES

• PRINCIPALGENDER,ETHNICITYOREXPERIENCEHAVENOSIGNIFICANTRELATIONSHIP WITHTEACHERRATINGS

CLASSROOMCOMPOSITIONANDSTUDENTANDTEACHEROUTCOMES

COMPARINGFOURTHANDFIFTHGRADECLASSROOMSWITHHIGHFREEANDREDUCEDLUNCHVERSUSLOWFREEANDREDUCEDLUNCHSHOWS:

üTHEREISASTRONGRELATIONSHIPWITHTHEPROPORTIONOFSTUDENTSONFREEANDREDUCEDLUNCHINACLASSROOMANDSTUDENTANDTEACHEROUTCOMES.

üTEACHERSINCLASSROOMSWITHHIGHERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHHAVELOWERIGMANDSUMMATIVERATINGS.

CLASSROOMCOMPOSITIONANDSTUDENTPERFORMANCE

• FOURTHANDFIFTHGRADECLASSROOMSFREEANDREDUCEDLUNCHDISTRIBUTIONDECILECOMPARISONS

üSEPARATINGTHEDISTRIBUTIONOFFOURTHANDFIFTHGRADECLASSROOMS BYDECILESSHOWSTHEREARE“BREAK”POINTSINCLASSROOMCOMPOSITIONWHEREIDENTIFIABLEDIFFERENCESINSTUDENTPERFORMANCEAREASSOCIATEDWITHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCH.

• DECILECOMPARISONSOFCLASSROOMSBY%OFFREEANDREDUCEDLUNCH:

üDECILECOMPARISONSSHOWDISTINCTDIFFERENCESBETWEENCLASSROOMSBASEDONPERCENTOFSTUDENTSONFREEANDREDUCEDLUNCH.

üTHEREISAPOINTBEYONDWHICHINCREASINGFRLHASNOCORRESPONDINGDECREASEINSTUDENTOUTCOMES- TYPICALLYAROUNDTHE7TH DECILEOR65%.

üCLASSROOMSINTHESECONDDECILE(18%- 28.5%FRL)HAVESTATISTICALLYLOWERIGMSCORESTHANCLASSROOMSINTHEFIRSTDECILE(0%- 18%).

üCLASSROOMSWITH65%ORMOREOFSTUDENTSONFREEANDREDUCEDLUNCHSHOWNOSTATISTICALLYSIGNIFICANTDIFFERENCES.

üTHISSUGGESTSTHEREMAYBEIMMEDIATEDETECTABLEEFFECTSONIGMSCORES ARISINGFROMLOWSESSTUDENTSPRESENCEINACLASSROOM,BUTTHEEFFECTOFADDITIONALLOWSESSTUDENTSTAPERSOFFASTHEPROPORTIONGROWS.

IGMRATINGSANDSUMMATIVERATINGS

2013PercentofTeachers

2014PercentofTeachers

2015*Percentofteachers

SummativeratingofEffectivewithanIGMratingsofIneffective

13% 11% 12%

SummativeratingofHEwithanIGMratingsofIneffective

5% 4% 6%

SummativeratingofEffectivewithanIGMratingofNeedsImprovement

29% 33% 33%

SummativeratingofHEwithanIGMratingofNeedsImprovement

19% 16% 23%

PLANQUALITY,PLANCOMPONENTS,STUDENTOUTCOMESANDTEACHERRATINGS

•THESIXDISTRICTSRECOGNIZEDFORTHEIRPLANQUALITYALSOHADBETTERSTUDENTGROWTH

•THEREISAMEDIUMTOSTRONGRELATIONSHIPBETWEENPLANQUALITYANDTEACHERSUMMATIVERATINGSFORDISTRICTS.

•THEREISASTRONGRELATIONSHIPBETWEENPRIORYEARSTUDENTISTEPSCORESANDTEACHERRATINGSFORDISTRICTS.

TEACHERMOBILITYANDEVALUATIONRATINGS

• THEREISNORELATIONSHIPBETWEENTEACHERMOBILITYANDWHETHEROR NOTTEACHERSAREINANACCOUNTABILITYGRADEORSUBJECTAREA.

• TEACHERSTENDTOMOVEFROMDISTRICTSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCH.

• THEREISNOTANASSOCIATIONWITHTEACHERMOBILITYFROMDISTRICTS WITHHIGHERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTONFREEANDREDUCEDLUNCHANDCHANGESINIGMRATINGS.

• THEREISARELATIONSHIPBETWEENMOVEMENTFROMDISTRICTSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHTODISTRICTSWITHLOWERPERCENTAGESOFSTUDENTSONFREEANDREDUCEDLUNCHANDSUMMATIVETEACHEREVALUATIONRATINGS.

• MOREEXPERIENCEDTEACHERSARELESSLIKELYTOMOVEDISTRICTS.

RECOMMENDATIONSMANAGEMENT&IMPLEMENTATION

• ENSUREDATAINTEGRITYINORDERFORVALIDRESEARCH-BASEDDECISIONMAKINGFORPOLICYANDGUIDANCE.

• ENSUREEXPLICITGUIDANCEONASSIGNINGSTUDENTSTOTEACHERSFORACCOUNTABILITYANDINCLUDETHISINTHEMONITORINGPROCESS.

• ENSUREVENDORMANAGEMENTANDCOMPETENCEWITHIMPACTFULPENALTIESENFORCEDFORLACKOFPERFORMANCE.

• ESTABLISHANEFFECTIVEMONITORINGSYSTEMTHATINCLUDESANANALYSISOFTEACHEREVALUATIONRATINGSANDSTUDENTOUTCOMESTOINFORMSTATESUPPORTFORTEACHERDEVELOPMENT.

RECOMMENDATIONSGROWTHMODEL

•REVIEWTHECURRENTGROWTHMODELTOENSURETHATSTUDENTCHARACTERISTICSAREEFFECTIVELYACCOUNTEDFORWHILEMAINTAININGHIGHEXPECTATIONSFORALLSTUDENTS.

•CONSIDEREXPLORINGATEACHEREVALUATIONGROWTHMODELTHATLOOKSATLIKECLASSROOMCOMPARISONSASTHEBASISFORTHEIGMRATING.

•CONSIDEREXPLORINGATEACHEREVALUATIONCLASSROOMGROWTHMODELTHATLOOKSATGROWTHBASEDUPONACLASSROOMMETRICOTHERTHANSTUDENTGROWTHPERCENTILES.

RECOMMENDATIONSTECHNICALSUPPORTFORPLANDEVELOPMENTAND

IMPLEMENTATION• PROVIDEONGOINGTRAININGANDSUPPORTTOENSUREINTER- RATERCONSISTENCYINTHETEACHEREVALUATIONPROCESS.

• ELEVATEPRINCIPALANDSUPERINTENDENTEVALUATIONQUALITYASSURANCETOTHESAMELEVELOFIMPORTANCEASTEACHEREVALUATION.

• ESTABLISHASETOFCRITERIABASEDUPONRESEARCHFINDINGSTOEVALUATEPLANDEVELOPMENTANDPLANIMPLEMENTATION.

• ESTABLISHINCENTIVESFORDISTRICTSTOENGAGEINATEACHEREVALUATIONPROCESSTHATFOCUSESONTEACHERDEVELOPMENTTHROUGHPROFESSIONALDEVELOPMENT.

• ESTABLISHAPROCESSTOENSURETHATTEACHERSAREINVOLVEDINTEACHEREVALUATIONASACOLLABORATIVEPROCESS.

• CONSIDERESTABLISHINGSTATEPROVIDEDINCENTIVESFORTEACHERSTOACCEPTTEACHINGASSIGNMENTSINDISTRICTSANDCLASSROOMSWITHHIGHPERCENTAGESOFSTUDENTSONFREEANDREDUCED LUNCH.

• PROVIDESTATESUPPORTFORPROFESSIONALDEVELOPMENTINCULTURALLY RESPONSIVEINSTRUCTION.

FUTURERESEARCHAGENDA• CONDUCTMORECOMPREHENSIVERESEARCHONTHERELATIONSHIPBETWEENHIGHQUALITYPLANIMPLEMENTATION,TEACHEREVALUATIONEXPERIENCESANDSTUDENTOUTCOMES.

• ENGAGEINQUALITATIVERESEARCHTODOCUMENTANDINCORPORATEEDUCATOREXPERIENCESINTOTHEANALYSISOFTEACHEREVALUATIONPROCESSESANDOUTCOMES.

• EVALUATETHEEFFECTIVENESSOFTECHNICALASSISTANCEANDSUPPORTONTHETEACHEREVALUATIONPROCESSININDIANA.

• CONDUCTRESEARCHONTHEINTER-RELATIONSHIPSOFCLASSROOM,SCHOOLANDDISTRICTCHARACTERISTICSONTEACHEREVALUATIONANDSTUDENTOUTCOMES.

• CONDUCTRESEARCHONSCHOOLANDDISTRICTRESOURCESANDSUPPORTSONTEACHEREVALUATIONANDSTUDENTOUTCOMES.

• CONDUCTRESEARCHONTHEDIFFERENTIALIMPACTOFTEACHEREVALUATIONONIGMVS.NON-IGMTEACHERS.

• PROVIDEFUNDSFORDISTRICTSTOENGAGEINASTATESUPPORTEDPILOT PROJECTTOREVISEANDIMPROVETHETEACHEREVALUATIONPROCESS.

AMULTI-DISCIPLINARYRESEARCHTEAM• HANNAHBOLTE,STATISTICALCONSULTANT/LECTURER,IUDEPARTMENTOFSTATISTICS

• MICHAELFRISBY,STATISTICALCONSULTANT,IUDEPARTMENTOFSTATISTICS

• DEMETREESL.HUTCHINS,MANAGEMENTANALYST,IUPUISCHOOLOFEDUCATION

• GARYPIKE,PROFESSOR,HIGHEREDUCATION&STUDENTAFFAIRS,IUPUI• SARAHPIES,RESEARCHASSOCIATE,INTASSPROJECT,INDIANAUNIVERSITY• HARDYMURPHY,IUPUIEDUCATIONALLEADERSHIPPROGRAM,EXECUTIVEDIRECTORINDIANAURBANSCHOOLSASSOCIATION,CODIRECTOROFINTASS

• SANDICOLE,DIRECTORCENTERONEDUCATIONANDLIFELONGLEARNINGIUB,CODIRECTORINTASS

• SPECIALACKNOWLEDGEMENT:THERESEARCHTEAMACKNOWLEDGESHAMMADRAHMAN,DATAMANAGEMENTSPECIALIST,INDIANADEPARTMENTOFEDUCATION,FORTHEIMPORTANTCONTRIBUTIONSMADETO THISRESEARCH.

QUESTIONS?