the impact of the 2012 tncore math training on … · the impact of the 2012 tncore ath training on...

The Impact of the 2012 TNCore Math Training on Teaching Practices and Effectiveness

POLICY BRIEF NOVEMBER 2013OFFICE OF RESEARCH AND POLICY

CONTENTS

1 EXECUTIVESUMMARY

2 INTRODUCTION

3 BACKGROUND

4 DATA

6 FINDINGS

10 INTERPRETINGTHERESULTS

12 CONCLUSIONS

AcknowledgementsLauraNeergaardBookerwasprimaryauthorofthisreport.BradWalkerdesignedthereport.

Click here to view the Technical Appendix.

http://www.tn.gov/education/research/doc/Core_Coach_Technical_Appendix.pdf

1 The Impact of the 2012 TNCore Math Training on Teaching Practices and Effectiveness

EXECUTIVE SUMMARY

Duringthespringandsummerof2012,theTennesseeDepartmentofEducationtrainedabout200CommonCoreCoacheswhothenwentontofacilitatesummertrainingsforthousandsofthestate’s

thirdthrougheighthgrademathteachers.Thefollowingsummer,thetrainingsessionsreachednearly30,000teachersacrossthestate,coveringmath,Englishlanguagearts,andliteracyinscience,socialstudies,andcareerandtechnicaleducation.

TheTennesseeCoreCoachtrainingmodelwasdesignedtodevelopanetworkofteacherswithadeepcontentandpedagogicalknowledgeoftheCommonCoreStateStandardswhocouldpasstheknowledgeontotheirpeersduringformaltrainingsessionsandinformalinteractionsthroughouttheyear.CoacheswereTennesseeteachersselectedviaacompetitiveapplicationandinterviewprocess.Coachesreceivedeightdaysofintensivegrade-leveltrainingprovidedbytheInstituteforLearningattheUniversityofPittsburgh,engagingwiththematerialfirstaslearnersandthenasteachertrainers.Coachesthendeliveredtrainingtoparticipantsatthree-day,grade-levelworkshopsheldthroughoutthesummer.

ThismemoprovidesthefirstanalysisoftheeffectivenessoftheTNCoretrainingonteacherpracticeandstudentachievement.Thefindingssuggestthatattendanceatthesummertrainingsessionsmadeasignificantdifferencetostudentachievementandteachereffectiveness,asmeasuredbybothobserverratingsandvalue-addedscoresbasedonstatewideTCAPmathtests.Theseresultsremainconsistentevenasweapplyevaluativeapproachesthatusemultipleyearsofteacherdataandcontrolforpriorscoresandschooleffects,suggestingthattheseresultsarenotbiasedbyparticipantselection.

KEY FINDINGS• WeconsistentlyfindpositiveandsignificanteffectsoftheTNCoremathtrainingonparticipants’

instructionalpracticeandontheireffectivenessatraisingstudenttestscores.Theseresultsremainconsistentusingmethodsthatcontrolforpreviousyearscores,school-levelinputs,andforthefixedcharacteristicsofteachers.

Participants’gainsonobservationscoreswereequivalenttoabouthalfofthegainsmadebytheaverageteacherbetweenthefirstandsecondyearofteaching.

Thegainsininstructionalpracticeratingswerelargestforthepracticesemphasizedinthetrainingsessions,includingskillssuchasquestioning,providingacademicfeedback,andteachingproblem-solvingtechniques.

Participants’gainsineffectivenessasmeasuredbytheTennesseeValue-AddedAssessmentSystem(TVAAS)translateintotheequivalentofapproximatelyoneextraweekoflearningforeachoftheirstudentsthanwewouldhaveexpectedhadtheynotattendedthetrainingsessions.

• ParticipantswhohadaCoreCoachworkingattheirschoolmadesignificantlygreaterestimatedincreasesinquestioningpracticescomparedtoparticipantswithoutthissupport.

• ForCoreCoaches,wefindsomeevidencethatthecoachingprocesswasassociatedwithimprovementsintheirownclassroomteaching;however,weareuncertainwhethertheseimprovementscanbeattributedtotheirroleascoaches.


INTRODUCTION

Overthenextseveralyears,45of50statesandtheDistrictofColumbiaareplanningtotransitiontoanewsetofinstructionalstandardsdesignedtoguideteachingineachgradeandsubject.TheCommon

CoreStateStandards(CCSS)aremeanttofocusontheskillsthatstudentswillneedtobereadyforcollegeandcareerandhelpourchildrenandourcountrystaycompetitive.However,thereisconsiderableevidencethatmanyteachersacrossthecountryfeelunpreparedtotakeontheincreaseddemandsofthesenewlearningstandards.1Forinstance,anationalsurveyfortheAmericanFederationofTeachersconductedinMarch2013foundthat44percentofteacherssaidtheirdistrictswere“somewhatprepared”or“notprepared”toimplementthestandards.Anothersurvey,conductedacross45states,foundthatlessthanone-fifthofmathteachersinthesamplehadreceivedmorethan20hoursofprofessionaldevelopmentrelatedtotheCCSS.2

TheTennesseeDepartmentofEducation(TDOE)hastakenauniqueapproachtoCCSSimplementationbysponsoringintensivestatewideprofessionaldevelopment(TNCoretraining)designedtoimprovestudentachievementbyeasingthetransitiontothenewstandards.Inthespringof2012,throughapartnershipbetweenthedepartmentandtheUniversityofPittsburgh’sInstituteforLearning,theTennesseeDepartmentofEducationselectedandtrainedabout200currentTennesseemathteacherstoserveas“CoreCoaches,”whowouldpreparetheirpeerstoteachthenewCommonCoreStateStandards.Thecontentofthetrainingfocusedonresearch-basedinstructionalshiftsthatwouldincreasestudentachievementinmathematics.Thatsummer,thousandsofthirdthrougheighthgrademathteachersparticipatedintheTNCoremathtrainingsessionsledbythefirstroundofstateCoreCoaches.In2013,thestateintensifieditscommitmenttothepeertraining,selectingandtrainingnearly700mathandliteracyCoreCoachestoleadtrainingsformorethan30,000educatorsacrossthestate.

Thismemooffersthefirstrigorousevaluationofthestate-ledeffortsatTNCoretraining.Wefocusontwocentralresearchquestions:

• DidservingasaCoreCoachleadtobetterclassroompracticeand/orgreatereffectivenessatraisingtheirownstudents’achievement?

• Didattendingthecoach-ledTNCoremathtrainingleadtobetterclassroompracticeand/orgreatereffectivenessatraisingstudentachievement?

WebeginwithsimplecomparisonsofoutcomesbetweenCoreCoaches,trainingparticipants,andnon-participants,movingtoaseriesofmorecomplexmodelstotestthevalidityoftheinitialconclusions.Acrossmodels,weconsistentlyfindpositiveandsignificanteffectsofthesessionsledbyCoreCoachesontheparticipants’observedinstructionalpracticeandontheireffectivenessatraisingstudenttestscores.Forcoaches,wefindsomeevidencethatthecoachingprocesswasassociatedwithimprovementsintheirownclassroomteaching;howeverweareuncertainwhethertheseimprovementscanbeattributedtotheirroleascoaches.

1 BillandMelindaGatesFoundation.(2012).Primarysources:American’steachersontheteachingprofession.NewYork:Scholastic,Inc.

2 Davis,J.,Choppin,J.,McDuffie,A.R.,&DrakeC.(2013).CommonCoreStateStandardsforMathematics:Middleschoolmathematicsteachers’perceptions.UniversityofRochester:WarnerCenterforProfessionalDevelopmentandEducation.

Many teachers

across the country

feel unprepared

to take on the

increased demands

of these new

learning standards.


BACKGROUND

Tennessee Core Coach Training Model

TheTennesseeCoreCoachtrainingmodelwasdesignedtodevelopanetworkofteacherswithadeepcontentandpedagogicalknowledgeoftheCommonCoreContentStandardsforMathematicsand

StandardsforMathematicalPracticewhocouldthenpasstheknowledgeontotheirpeers.CoacheswereTennesseeteachersselectedviaacompetitiveapplicationandinterviewprocess.Coachesreceivedeightdaysofintensivegrade-leveltrainingprovidedbytheInstituteforLearning.Byengagingfirstaslearners,coacheshadopportunitiestodeepentheirowncontentandpedagogicalknowledge,aswellastheirunderstandingofthewaysinwhichadultslearn.Duringthetraining,coachesengagedinmathlessonsandreflectedonthecontentlearned,howitwaslearned,andwhatsupportedthatlearning.Theyalsoanalyzedteachingandlearningthroughthelensofpedagogicalandcontenttools.Coachesthendeliveredtrainingtoparticipantsatthree-day,grade-levelworkshopsheldthroughoutthesummer.Aspartofthemodel,principalCoreCoacheswerealsoselectedandtrainedtoleadonedayoftrainingspecificallyforschoolleadersattendingthethree-daytraining.

Thetraininggiventobothcoachesandparticipantswasdesignedtochallengeteachers’assumptionsusingatransformativeapproach.Transformationfocusesondesigninglearningexperiencesthatchallengeparticipants’currentthinkingandoftenstartlethemintonewbeliefs.3Withinthetraining,thegoalwasforteacherstoengagedirectlywithstudentworkandassessthestrengthoftheircurrentteachingmethodswhilebeginningtheprocessofconstructingnewandmorepowerfulpractices.

Duringthecoach-ledtrainingsessions,coacheshelpedtheirpeersbetterunderstandnewfocusareasforgrades3-8mathbyworkingthroughhighlevelconstructedresponseassessmenttasksandfacilitatingmodellessonsusinganinstructionaltask.Participantswereaskedwhichcontentandpracticestandardswereaddressedthroughthetasksandlessons.Theydiscussedtheirideastodeepentheirunderstandingofthemeaningandexpectationsofthestandards.

Themethodofusingpeercoachestodeliverthetraininghasseveralpotentialadvantages.Thecoachesgainlastingknowledgethatshouldpositivelyinfluencetheirownteachingandstudents’learning.Thecoachesthenpassthatknowledgetoparticipantstoimplementintheirownpractice.Inaddition,thecoachesareavailableattheschoolanddistrictleveltosharetheirexpertisewithotherteacherstobroadentheimpactofthetraining.TDOEandtheIFLalsomadeallmaterialsusedinthetrainingavailableonlinetoeducators.Thesematerialsincludedconstructedresponseassessmenttasks,instructionaltasks,ataskanalysisguideusedtodeterminethecognitivedemandofwrittentasks,andexamplesoflessonstructureandroutines.

Itisworthnotingthatthismodelofusingdistrictpersonnelasteachertrainersmakesitmoredifficulttoevaluatetheresultsofthetraining.Uponreturningtotheirdistricts,CoreCoachesandparticipantswereoftenaskedtoprovidetrainingtoandsharematerialswithteacherswithintheirdistrictsandschoolswhodidnotattendthetraining(seeFigure1).Inthiscase,wewoulddetectlessofadifferenceinoutcomesbetweenthosewhodidanddidnotattendtraining,evenifthetrainingsactuallydidhaveapositiveeffectsinceourdatacannotentirelydistinguishbetweentrainingparticipantsandnon-participants.ThisphenomenonsuggeststhatourestimatedresultsmightbelowerthantheactualimpactoftheTNCoretraining.

Core Coaches Participants

Non-Participants

Figure 1. Who received the training?

3 Kaser,J.,Mundry,S.,Stiles,K.,&Loucks-Horsley,S.(2002).Leadingeverday:124actionsforeffectiveleadership.ThousandOaks,CA:CorwinPress.

By engaging first as

learners, coaches had

opportunities to deepen

their own content

and pedagogical

knowledge, as well as their

understanding of the ways

in which adults learn.


DATA

Common Core Math Coaches, Training Participants, and Non Participants

The194CommonCoremathcoachesin2011-12represented163schoolsin56districts.ThedarkblueinFigure2indicatesthecountieswherecoacheswerelocated,andthelightblueindicatescounties

thathadparticipantsbutnocoaches.OnlyonedistricthadnoCoreCoachesortrainingparticipants.MostCoreCoacheswereclassroomteachers,althoughsomeheldotherrolesinschoolsanddistricts,suchasinstructionalcoachesoradministrators.Inthisstudy,wefocusonCoreCoacheswhowereclassroomteachers.

Whileteachersfromprivateschools,schoolanddistrictmathcoachesandadministrators,anduniversity

instructorsattendedthetraining,thefocusofthisstudyisonparticipantswhoarepublicschoolclassroommathteachers.About6,000participantshadanobservationscorefrom2012or2013,meaningtheywereapublicschoolteacher.Theseparticipantsrepresentedalmost1,700schoolsacross135districts.About11percentoftheparticipantshadacoachlocatedintheirschool,andabout5percentofparticipantshadaprincipalwhoservedasaprincipalCoreCoach.Themajorityofparticipants(77percent)hadaschoolleaderwhoalsoattendedthetraining.

Non-participantsaremathteachersingrades3-8whodidnotattendthetraining.Teacherswereidentifiedasnon-participantsiftheyhadTVAASmathestimatesforgrades4-8from2011-12schoolyearorgrades3-8fromthe2012-13schoolyearbutwerenotontheparticipantlist.ThirdgradeteachersdidnothaveaTVAASestimatesin2011-12.

Outcome MeasuresWefocusontwocentralmeasurestoidentifytheeffectsofthe2012TNCoremathtrainingonteachingandlearning,teacherobservationratingsandameasureofteachercontributiontostudentachievement.Thesemeasuresareavailableaspartofthestatewideeducatorevaluationsystemfirstimplementedinthe2011-12schoolyear.Whiletheserepresentourbestsetofcurrentmeasures,itisimportanttonotethatstudentachievementismeasuredusingTennesseeComprehensiveAssessmentProgram(TCAP),andwhiletheTCAPtestshavebeenfocusedtomorecloselyaligntotheCCSS,theyarenotyetperfectlyaligned(seeFigure3).Thus,itispossiblethat,asaresultofthetraining,teachersmightbecomebetteratteachingtheskillsthatwillbetestedbythenextiterationofTennesseetestsbutthatarenotpickedupbythecurrenttests.

Classroom Instructional Practices.InTennessee,observationsofclassroominstructionalpracticescountas50percentofthestatewideeducatorevaluationsystem.Teachersareobservedmultipletimeseachyearusingastate-approvedobservationrubric.Mostdistricts(comprisingabout80percentofteachersstatewide)havechosentousethestateobservationrubricthatisbrokenintofourdomains:instruction,environment,professionalism,andplanning.Eachdomaincontainsmultipleindicators,scored1through5duringclassroomobservationswith5beingthehighestpossiblerating.Forthepurposesofthisstudy,we

No Participants or Coaches

Participants

Participants and Coaches

Figure 2. Tennessee Counties with 2012 Math Common Core Coaches


focusontheInstructiondomainandonthefourInstructionindicatorsthataremostalignedwithteachingpracticesemphasizedinthetraining:Questioning,AcademicFeedback,Thinking,andProblemSolving.Table1providestheaverageobservationscoresfor3-8mathteachersin2012-13.4

Teacher Effectiveness at Increasing Student Achievement.TheTennesseeValue-AddedAssessmentSystem(TVAAS)measurestheimpactschoolsandteachershaveontheirstudents’academicprogress.TVAAS,whichisbasedonstudentresultsfromtheTennesseeComprehensiveAssessmentProgram(TCAP),controlsforstudents’previousachievementlevelstoisolategrowthduringaparticularclassroomyear.Forteachersintestedgradesandsubjects,a1to5compositeTVAASscorecountsas35percentoftheteacherevaluationmodel.TVAASscoresforgrades3-8arecalculatedatthesubjectandgradelevelandmeasureprogressusing2009asthebaseyearcomparison.Table1alsoshowstheaverageTVAASscoreinstandarddeviationunits.5

4 2012-13classroominstructionalpracticemeansonlyincludeteacherswhoalsohave2011-12practicescores.

5 2012-13TVAASmeansonlyincludeteacherswhoalsohave2011-12TVAASscores.

MeanStandard Deviation Range

Instruction Domain 3.89 0.59 1 to 5

Questioning 3.74 0.71 1 to 5

Thinking 3.70 0.76 1 to 5

Problem Solving 3.69 0.82 1 to 5

Academic Feedback 3.82 0.71 1 to 5

TVAAS Level 3.74 1.41 1 to 5

TVAAS in SD units 0.70 1.00 -4.1 to 4.4

Table 1. Average 2012-13 Classroom Observation Ratings and TVAAS Scores

Challenges to Evaluating Professional Development

Thereareseveralsignificantchallengestoevaluatingtheeffectsof professional development (PD) on teaching and learning.Whilean idealevaluationwouldcomparea seriesof long-termoutcomes between two equivalent groups of teachers, one ofwhich had been randomly assigned to receive a certain set oftrainingsandoneofwhichhadgonewithoutthistreatment,PDrarely takesplace in thisway.As a result, anyevaluation facesseveralthreatstoitsoverallvalidity.

First,theoutcomemeasuresmaynotbealignedwiththetargetorfocusofthePD.Forexample,ifPDiscenteredondeepeningquestioningpractices,buttheoutcomemeasureevaluatesquestioningpracticesbasedonfrequencyratherthandepth,wemayconcludethatthePDhasnotbeeneffectiveeventhoughteachersaretrulychangingtheirpractice.

Second, it isdifficult toestablishanappropriatecomparisongroup.TeacherswhovoluntarilyattendPDsessionsmaystartatahigherlevelorhaveagreaterpropensitytoimprovethanthosewhochoosenottoattendpriortothetrainingitself.Moreover,teacherswhodid

notattend the trainingmay still receiveaspectsof the trainingthroughotherroutes.

Third, teachers may attend multiple training opportunities ormayreceiveothersupportsthatimpacttheirpractice,makingitdifficulttoisolatetheeffectsofasinglePDexperience.

Finally,changesinteacherpracticemaytaketime,meaningthatimmediate impacts are difficult to detect. Thismemouses thefollowingquestionstoguideourevaluationoftheTNCoremathtraining.

Questions to ask when evaluating PD:

1. Are the measures used to evaluate the PD aligned with the PD goals?

2. Who is the appropriate comparison group?3. What other supports do teachers receive that make it

difficult to determine the impact of PD?4. Have teachers had the time to integrate the PD into their

practice?

Figure 3.


FINDINGS

Figures4and5presentthe2012-13observationandTVAASscoresofcoaches,participants,andgrades3-8mathteacherswhodidnotparticipateinthetraining.6Asthefiguresshow,coachesscored

significantlyhigheronteacherobservationandTVAASthantrainingparticipantswho,inturn,scoredhigherthannon-participants,in2012-13afterattendingtheTNCoremathtraining.Theproblemwithjustexaminingoutcomesintheyearfollowingthetrainingisthatitdoesnotaccountforthedifferencesbetweencoaches,participants,andnon-participantsthatwerealreadyinplacebeforethetraining.Asfigures4and5show,observationandTVAASscoreswerealsohigherforcoachespriortothetraining,whichisnotsurprisingconsideringthatthesefactorswerepartofthecoachselectionprocess.Trainingparticipantshadsignificantlyhigherinitialaverageobservationscoresthannon-participantsbutsimilarinitialTVAASscores,suggestingeitherthatschoolleadersselectedteacherstoattendthetrainingwhotheyperceivedasbetterteachersorthoseteachersself-selectedintothetraining.

Togetattheimpactofthetraining,weneedtotakeonthiscomparisongroupproblembyfindingawaytocontrolforthesetofinitialdifferencesacrosseachtraininggroupthatmayhavecontributedtochangesinoutcomes(seeFigure3).

Method 1: Controlling for Past PerformanceToevaluatetheimpactofthe2012TNCoremathtrainingoncoaches’andparticipants’observationandTVAASscores,weneedtotakeintoaccountthedifferentstartingpointsofcoaches,participants,andnon-participantsinourstudy.Thus,weuseregressionanalysistoestimatetheimpactoftrainingon2012-13scorescontrollingfor2011-12scores.Regressionanalysisisamethodforestimatingtherelationshipamongdifferentfactors.Here,itprovidesanestimateofhowmuchoutcomeschangeifateachereitherservedasaCoreCoachorparticipatedinthetraining.Regressionanalysisalsoallowsotherfactorstobeheldconstant(i.e.,controlledfor).Thismeanswecancomparescoresbetweencoaches,participants,andnon-participants

4.11

4.63

3.80 4.23

3.66

4.24

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Observation TVAAS

Coaches Participants Non-participants

Figure 4. 2011-12 Observation Ratings and TVAAS Scores Before Training

6 Foreachoftheoutcomes,onlyteacherswhohadbothyearsofdataareincludedintheaverages.Moredataisavailableacrossbothyearsonclassroompracticesthanteachereffectivenessbecauseallteachers,regardlessofgradeorsubjectornumberofstudentstaught,receiveobservationscores.

7 Ingeneral,TVAASscoreswerelowerin2012-13than2011-12duetosmallergainsinstudenttestscores.

4.23 4.30 3.95 3.97 3.80 3.85

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Observation TVAAS

Coaches Participants Non-participants

Figure 5. 2012-13 Observation Ratings and TVAAS Scores After Training7


in2013whileholdingconstanttheinfluenceofinitial2012effectivenessratings.Wealsocontrolledfortheteacher’sgradeandexperiencelevel.8

Holding2011-12observationscoresconstant,wefindthatbeingacoachoraparticipatinginthetrainingwasassociatedwithstatisticallysignificantincreasesinalloftheclassroominstructionalpracticesexamined.Thegainswereatleasttwiceaslargeforcoachesasforparticipants.Thisisnotsurprisinggiventhatcoachesreceivedmoreintensiveexposuretothetrainingviatheeightdaysoftraininganddeliveringthecontentinthesummersessions.Interestingly,therelationshipappearsevenlargerwhenwelookattheindividualindicatorsratherthantheinstructionaldomainoverall.Thissuggeststhatthetrainingmighthavebeensuccessfulinleadingteacherstoimplementtheteachingpracticesmostemphasizedinthetraining.

Onaverage,thismethodofanalysissuggeststhatservingasaCoreCoachwasassociatedwithathirdofapointincreaseonobservationscoresforteachersinpracticesforteachingproblemsolvingandaroundaquarterofapointincreaseonpracticesaroundacademicfeedback,studentthinking,andquestioning(seeFigure6).Themagnitudeoftherelationshipwassmaller,butstillstatisticallysignificant,forparticipants.Participantsgainedbetweenone-tenthandone-twentiethofapointacrossthefourinstructionalindicatorsandoverallinstructionaldomain.

Whenweusethesamemethodtolookateffectsonteachers’value-addedscores,wesimilarlyfindthatcoachingorparticipatinginthetrainingwasassociatedwithstatisticallysignificantincreasesin2012-13TVAASscores.9Servingasacoachwasassociatedwithaboutone-fifthofastandarddeviationincreaseinteachereffectivenessandparticipatinginthetrainingwasassociatedwithaboutone-twelfthofastandarddeviationincreaseineffectiveness(seeFigure7).MoredetailsaboutthemodelsandresultsarelocatedintheAppendix.

WhilethesefindingsprovidesomeevidencethatperhapsTNCoremathtrainingdidhavemeaningfuleffectsonteachingpracticeandstudents’achievement,thismethodofaddingcontrolsforteachers’startingpointsisnotsufficientforisolatingtheimpactofthetraining.Coaches,participants,andnon-participantsdonotnecessarilysharethesameteachingsupports(seeFigure3).Forexample,coachesandparticipantsmighttendtocomefromschoolsthatarehighlyinvestedintheCCSSandthathavedeployedahostofresourcestowardtheimplementationofthenewstandards,whereasnon-participantsmightinsteadbedrawnfromtheschoolsthatarefocusedonotherissues.Ournextmethodbuildsuponmethodonebycontinuingtocontrolforinitialstartingpoint,butalsotakingtheseadditionalconcernsintoaccount.

Figure 6. Estimated Difference in 2012-13 Observation Ratings for Coaches and Participants Compared to Non-Participants, Controlling for Past Performance

0.05

0.08

0.06

0.08

0.07

0.12

0.22

0.24

0.28

0.31

0 0.1 0.2 0.3 0.4 0.5

Instruction

Academic Feedback

Questioning

Thinking

Problem Solving

Observation Rating (1 -5)

Coach

Participant

8 Abeginningteacherisdefinedashavingonlyoneortwoyearsofexperience(beingintheirsecondorthirdyearofteaching).Firstyearteacherswerenotincludedinthestudybecausetheydonothavepriorscores.

9 TVAASscoresarecalculatedatthegradelevelsoteacherswhoteachmultiplegradelevelsreceiveestimatesforeachgradelevel.Resultscouldbebiasedifteacherswhoteachmultiplegradesareeithermoreorlesslikelytoattendtrainingandmoreorlesslikelytobeeffective.Therefore,weweightedtheanalysistoaccountforthisvariation.

Figure 7. Estimated Difference in TVAAS Scores for Coaches and Participants Compared to Non-Participants, Controlling for Past Performance

0.08

0.22

0 0.1 0.2 0.3

Participant

Coach

TVAAS Performance (standard deviation units)

The Impact of the 2012 TNCore Math Training on Teaching Practices and Effectiveness

Method 2: Taking Environment into AccountThefollowingresultscontinuetocontrolforgradelevel,teachers’experiencelevelandtheirpreviousscoresonobservationandTVAAS,buttheyaddanadditionalconstraintinthatallcomparisonsoccuronlybetweencoaches,participants,andnon-participantswhotaughtwithinthesameschool.Thus,theeffectsofthetrainingaremeasuredonlyinschoolsthathadbothnon-participatingmathteachersandparticipatingmathteachersorcoaches.Thismodel,knownasaschoolfixedeffectsmodel,controlsforfactorsthatarethesamewithinschools,suchashavingthesameadministrativestaffconductingobservations,havingthesameschoolorientationtowardtheCommonCoreStateStandards,andteachingsimilarstudentpopulations.

Resultsfromtheschoolfixedeffectmodelsarequitesimilartoresultsofthebasicmodelspresentedabove,strengtheningthecasethattheseresultsactuallyrepresenttheimpactoftheTNCoremathtraining.

Inthecaseofteacherobservationratings,resultsaregenerallyunchangedwhenincludingschoolfixedeffects.Figure9demonstratesthemagnitudeoftheestimatedgainsinclassroomobservationratingsforteacherswhoeitherservedasCommonCoreCoachesorparticipatedintrainingsessionsascomparedwiththoseinthesameschoolwhodidnotattendthetraining.

Estimatesoftheprogram’simpactonteachervalue-addedalsolookquitesimilarwiththeadditionofschoolfixedeffects.Again,beingacoachwasassociatedwithaboutafifthofastandarddeviationincreaseinteachereffectiveness,whileparticipatinginthetrainingwasassociatedwithsmallerbutstillstatisticallysignificantincreases(seeFigure10).

TheseresultsaddconfidencethattheTNCoremathtrainingwassignificantlyassociatedwithimprovementsinteacherquality.Afinalconcern,however,isthatthedifferencebetweencoaches,participants,andnon-participantsliesnotintheirstartingpointorintheirenvironmentbutratherintheirlikelihoodofimprovementovertime.Inotherwords,coachesandparticipantsmightrepresentthegroupofteacherswhoaremorelikelytoachievetheseincreaseswithouttraining,whilenon-participantsmighthaveimprovedlesseveniftheyhadattendedthetraining.

What do improvements in classroom instructional practices mean for students?

• Studentsreceivedmorefrequent,high-qualityfeedback.

• Studentsreceivedinstructionindifferenttypesofthinking,suchasanalytical,creative,andresearch-basedthinking.Studentsalsoreceivedmoreopportunitiestoanalyzeproblemsfrommultipleperspectives.

• Studentswereaskedquestionsthatadvancedtheirunderstandingandaskedtociteevidencefortheirreasoning.

• Studentsengagedinactivitiesrequiringthemtouseavarietyofproblem-solvingstrategies,includingpredictingoutcomes,generatingideas,andimprovingsolutions.

Figure 8.

Figure 9. Estimated Difference in Observation Ratings for Coaches and Participants Compared to Non-Participants, Controlling For Environment and Past Performance

0.05

0.08

0.11

0.10

0.09

0.16

0.24

0.21

0.29

0.35

0 0.1 0.2 0.3 0.4 0.5

Instruction

Academic Feedback

Questioning

Thinking

Problem Solving

Observation Rating (1-5)

Coach

Participant

Figure 10. Estimated Difference in TVAAS Scores for Coaches and Participants Compared Non-Participants, Controlling For Environment and Past Performance

0.06

0.21

0 0.1 0.2 0.3

Participant

Coach



Method 3: Adjusting for Teacher CharacteristicsInbothpreviousmethods,ourcomparisongroupwasthegroupofteacherswhohadnotparticipatedinTNCoremathtraining,eitherascoachesorparticipants.However,thesemethodsdidnotallowustoconsiderthepossibilitythatcoachesandparticipantsmighthavemovedoncompletelydifferentpathstowardteachingimprovementthannon-participantsregardlessofthespecifictrainingtheyattended.

Here,weshiftcourseandcompareindividualstotheirownselvesintheyearsbeforetheyparticipatedintraining.Thismodel,knownasanindividualfixedeffectmodel,holdsconstantanythingwithinteachersthatdoesnotchangeovertime,therebycontrollingforunobservablecharacteristicssuchastheirlikelihoodforself-improvement.WeuseTVAASdatadatingbacktothe2007-08schoolyear.UsingmultipleprioryearsofTVAASperformanceallowsustoexaminewhethertheteachersimprovedmoreduringtheyearfollowingparticipationinthetrainingthanwewouldhaveexpectedgiventheirpreviousscores.10Sincestatewideteacherevaluationonlybeganin2011-12inTennessee,weareunabletoemploythismethodforteachers’observationratings.

Interestingly,theseresultsaredifferentthanthoseinprevioussections.Servingasacoachisnolongersignificantlyassociatedwithanincreaseinteachereffectivenessscores(seeFigure11).Whileourpreviousestimatesshowedthatcoaches’gainsonTVAASoutpacedparticipantsandnon-participants,theselaterresultssuggestthatthegainsmightnothavebeencausedbythecoachingexperience.Coacheswerealreadyonatrajectorytooutperformparticipantsandnon-participants,evenbeforethetrainingtookplace.

Atthesametime,theestimatesoftheimpactofthetrainingonTVAASscoresfortrainingparticipantsarenearlyidenticaltotheresultswereportedinprevioussections.Again,beingaparticipantasassociatedwithanincreaseofabout0.08standarddeviationsinteachereffectiveness.Inotherwords,trainingparticipantsimprovedsignificantlymorein2012-13thanwewouldhaveexpectedevenknowingtheirTVAASscoresfromthepreviousseveralyears.ThesematchingfindingsacrossmodelsprovideevidencethattheeffectwearedetectingcanbeascribedtotheTNCoremathtrainingthattheparticipantsattendedinsummer2012.

0.08

not significant

0 0.1 0.2 0.3

Participant

Coach


Figure 11. Estimated Gains in TVAAS Scores for Coaches and Participants, Adjusting for Teacher Characteristics

10 Themodeladditionallyincludescontrolsforyearandgradeleveltoaccountfordifferencesinaggregatetrendsovertimebyyearandgradelevel.

Training participants improved significantly more in 2012–13 than we would have

expected even knowing their TVAAS scores from the previous several years.


INTERPRETING THE RESULTS

Eachmethoddescribedaboveusesadifferenttechniquetoaddressthechallengesofevaluatingprofessionaldevelopment.Sowhatcanweconcludewhenwelookacrossmodels?Coachesexperiencedgainsinobservationscoresandinstudentgrowthbuttheyappeartobeaselectgroupwhomayhavemadegreatergainsthantheirpeersin2012-13withorwithouttheirparticipationincoachingsessions.

Forthosewhoparticipatedinthetraining,wefindaconsistentandsignificanteffectofattendingthetrainingonboththeirinstructionandclassroomimpact.Participantsscoreaboutone-tenthofapointhigheronthe1-5observationscaleforvariousobservationindicators,andtheyrateabout0.08standarddeviationshigheronTVAASscoresthanwewouldhavepredictedabsentthetraining.

Nowthatwehaveevidencethatthetraininghadapositiveimpact,itishelpfultothinkabouttheimplicationsforstudents.Onewaytothinkabouttheimpactofthetrainingisbycomparingtheaverageeffectivenessofcoachestonon-coaches.Onaverage,coachesareaboutone-thirdofastandarddeviationhigherinteachereffectivenessthannon-coaches(i.e.,participantsandnon-participants).Afterattendingthetraining,participantsareestimatedtoimprovetheireffectivenessatincreasingstudentachievementby0.08orone-twelfthofastandarddeviation.Thismeansparticipantswouldbeabout25percentclosertocoaches’estimatedeffectivenessthantheywerebeforethetraining.Figures8and12alsoofferhelpinterpretingthemagnitudeoftheseresultsintermsthatoffermoreinsightintotheimpactoftheeffect.

InthetermsdescribedinFigure12—abouthalfofthedifferencebetweenafirst-andsecond-yearteacherasmeasuredbyobservationratings,andoneweekofstudentachievementasmeasuredbyTVAAS—thegainsthatparticipantsmadedonotseemparticularlylarge.Atthesametime,theserepresentresultsfromasinglethree-daytrainingsessionthatparticipantsattendedduringthesummermonths,andtheycontrastwithseveralotherrigorousstudiesofrecentprofessionaldevelopmentinitiativesthathavefoundnosignificanteffectsonteachingorlearning.11Moreover,the2012trainingevaluatedhererepresentedthefirsttimethatthestateattemptedsummertrainingsessionsatsuchscale,withthesubsequent,largertrainingin2013benefitingfromseverallessonslearnedduringtheprevioussummer.

How large are the effects?

Classroom Instructional Practices. On the 1-5 observationscale,CoreCoachesscoreaboutone-thirdofapointhigherandparticipants scoreaboutone-tenthofapointhigher comparedtonon-participants.Whilethesegainsmayseemquitesmall,itisimportanttonotethatobservationscoresforTennesseeteacherstend to rangebetween threeandfive,meaning thataquarter-point gain can be quite meaningful. Another way of thinkingaboutthegainisbylookingataveragedifferenceacrossteachercohorts. Theone-thirdofapointpredictedchangeincoaches’problem solving scores is about the same size as the average increase on problem solving for teachers in their second year compared to their problem solving scores in their first year. Themagnitude of the impactwas smaller for participants. Forexample, participants are associated with a 0.08 increase inacademic feedback. This is abouthalf the size of the average increase in academic feedback for second year teachers.

Teacher Effectiveness. Wedrawon learningtime calculationsusedbyTheEducationTrusttointerpretourfindings. Researchexaminingstudentachievementonanationallynorm-referencedtest reported expected increases, on average, by about 0.42standarddeviations inmathover thecourseofacalendaryear.ResearchconductedbySASInstitute,Inc.foundthatinTennesseea one standard deviation (SD) increase in teacher effectivenessresultsina0.15to0.22SDincreaseinstudentachievement.Usingthe lower estimate, a one SD increase in teacher effectivenessis associated with an increase in student outcomes of about0.15standarddeviations,whichequatestoabout3.5monthsoflearning inmath. Using this interpretation, being a core coach was associated with student achievement gains of about three weeks and participating in the training was associated with student achievement gains for about one week. If the higherestimateof the relationship between teacher effectiveness andstudentachievementwasused, theestimated impactwouldbe4.2weeksofstudentachievementforstudentsofCoreCoachesand1.6weeksofstudentachievementforstudentsofparticipants.

Figure 12.

11 Yoon,K,S.,T.,Lee,S.W.Y.,Scarloss,B.,&Shapley,K.L.(2007).Reviewingtheevidenceonhowteacherprofessionaldevelopmentaffectsstudentachievement.NationalCenterforEducationalEvaluationandRegionalAssistance,InstituteofEducationSciences,U.S.DepartmentofEducation


Importantly,thecalculationsabouttheeffectsofthetrainingonstudentachievementalsoallowustoprovidearoughcost-benefitcalculationusingdatafromotherresearchstudiesabouttheeconomicimpactofhigherstudenttestscores.Whilethecost-benefitprojectionsrequireseveralassumptions,theyalsoofferausefulgaugeastowhetherthestateisspendingitsRacetotheTopdollarswisely.

Thetotalcostofthe2012summermathtrainingwasabout$3.2million.Thisincludedmaterials,trainingsitescosts,lodgingfortrainers,andpaymentstoCoreCoachesforleadingthetrainingsessions.Weestimatethatthetrainingwasprovidedtoabout6,000publicschoolteachersinthestateofTennessee.Thus,thecostaveragestoabout$500perteacher.12Hanushek(2004)calculatesthataonestandarddeviationincreaseinstudentachievementtranslatesintoabout$60,000increaseinlifetimeearnings.13Wefoundthatparticipatinginthetrainingwasassociatedwith0.08standarddeviationincreaseinteachereffectivenessora0.01increaseinstudentachievement.Therefore,studentsofteacherswhoparticipatedinthetrainingwouldbeexpectedtoexperienceabouta$600lifetimeincreaseinearnings.Ifthe6,000teacherswhoattendedthetrainingtaughtabout30studentseach,thiswouldtranslateintoabout180,000studentsearninganadditional$600overtheirlifetimeor$108million.Thisestimatedbenefitisbasedonasingleyearofimprovedteachereffectiveness.Weexpectthattheimprovementsattributedtothetrainingwouldcontinuetobenefitstudentsofteacherswhoattendedthetraininginfutureyears.Whilethesecalculationsprovideausefulmethodofestimatingthefinancialbenefitofthetraining,theyarejustestimatesandshouldbeinterpretedwithcaution.

Impact for Participants with Access to Coaches at their SchoolItisalsousefultothinkaboutwhethertherewereparticularattendeeswhobenefitedmostfromthetraining.BecauseCommonCoreCoacheswerealsoTennesseeteachers(andprincipals),theparticipantswhohadcoachesactuallyworkingintheirschoolslikelyhadgreateraccesstoongoingfeedbackandsupportforimplementingtheprinciplesofthetrainingthanthosewhoonlyattendingthethree-daysummertraining.Thissectionaddressesthequestion:Didparticipantsincoachschoolsbenefitfromthetrainingmorethanparticipantsinnon-coachschools?

SincehavingaccesstoaCoreCoachwouldbethesameforallparticipantsinthesameschool,weareunabletouseaschoolfixedeffectsapproachcomparingteachersinthesameschool.Therefore,weusedadistrictfixedeffectsspecificationthatcomparesteachersinthesamedistricttoeachother,accountingfordifferencesinsupportsthataresimilaracrossdistricts.WecomparedparticipantswithCoreCoachesintheirschoolstoparticipantswithoutCoreCoachesintheirschools.

ParticipantswithaCoreCoachattheirschoolhadsignificantlygreaterestimatedincreasesinquestioningcomparedtoparticipantswithoutthissupport.Participantswithcoachesintheirschoolswerepredictedtoimprovequestioningpracticesby0.08comparedtoparticipantswithoutaccesstocoaches.Thedifferencesinpredictedincreaseswerenotstatisticallysignificantbetweenthetwoparticipantgroupsfortheotherinstructionalpractices.Thus,weconcludethathavingaccesstoaCoreCoachwasassociatedwithgreaterimprovementsinquestioningpracticesfortrainingparticipants.Weconductedasimilaranalysistolookattheimpactforparticipantshavingaccesstoacoachonteachereffectivenessoutcomes.However,duetothesmallersamplesizeofteacherswithTVAASscoresfrombothyears,thegroupsofparticipantswithandwithoutcoachesattheirschoolsweretoosmalltodetectdifferences.

12 Whilethe2012summertrainingactuallyservedabout13,000participants(includingschoolanddistrictleadersandotherpersonnel),wearemostinterestedintheimpactofthetrainingonclassroomteachers’instructionalpracticesandeffectivenessatincreasingstudentachievement.Therefore,wecalculatethecostoftrainingforjustTennesseepublicschoolclassroomteachers.

13 Hanushek,E.A.(2004).Somesimpleanalyticsofschoolquality(No.w10229).NationalBureauofEconomicResearch.


CONCLUSIONS

ThismemoreportsontheresultsofastudyoftheimpactofTennesseeDepartmentofEducationledtrainingsessionsforgrades3-8mathteachersacrossthestateinthesummerof2012.Usingavariety

ofanalysistechniques,weconsistentlyfoundastatisticallysignificantrelationshipforattendingthethree-daytrainingsessiononteachers’instructionalpracticesintheareasofquestioning,feedback,problemsolving,andthinking,andonteachers’contributionstostudentachievement.Thoughthemagnitudefindingsmayappearsmallforanindividualteacher,whencompoundedacrossthe6,000participantswhoeachteachnumerousstudents,thetrainingseemstohavehadagreatbenefitforstudentsinTennessee.

Whilewehavetakenanumberofstepstoidentifythetrueimpactofthetraining,itisimportanttorememberthatthechallengesofevaluatingprofessionaldevelopmentmakethistaskdifficult.Duetothenatureofthetrainingmodel,wecannotbecertainwhethernon-participantsaretherightcomparisongroupassomenon-participantsmayhavereceivedaccesstothetrainingprinciples.Ifsomenon-participantsreceivedthetrainingandtheirpracticesimprovedasaresult,thiswouldbiastheestimateoftheimpactofthetrainingoncoachesandparticipantsdownward,meaningthattheimpactcouldbelargerthanweareestimating.Additionally,wemayseeagreaterrelationshipforattendingthetrainingonteachereffectivenessifstudentachievementwasmeasuredwithanassessmentthatwasmorecloselyalignedwiththeCommonCoreStateStandards.

Lastly,thetrainingwasdesignedtoshiftteachers’thinkingaboutteachingandoftenteachersneedtimeforthisshifttotakeplaceandtointegratethelessonslearnedfromthePDintotheirpractice.Ourinvestigationfocusedononlytheinitialyearafterthetraining.However,itmayactuallytakelongerthanayearforteacherstochangetheirpracticeandforthosechangestoresultinincreasesinstudentachievement.Therefore,itisimportanttocontinuetoinvestigatethelong-termimpactsofthetraining.

If you have questions or comments regarding this report, please email [email protected].

Click here to view the Technical Appendix.

mailto:Education.Research%40tn.gov?subject=RE%3A%20Common%20Core%20Report

http://www.tn.gov/education/research/doc/Core_Coach_Technical_Appendix.pdf

the impact of the 2012 tncore math training on … · the impact of the 2012 tncore ath training on...

Documents