![Page 1: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/1.jpg)
HighStakesQualityMeasuresinEarlyCareandEducation:ReconsideringtheEvidenceRachelA.Gordon,Professor
SociologyandInstituteofGovernmentandPublicAffairsUniversityofIllinoisatChicago
TheChallengeofUsingObserva3onalQualityRa3ngToolsinAccountabilitySystemsand
StrategiestoAddressThemNa#onalResearchConferenceonEarlyChildhood
July13,2016(v3)
![Page 2: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/2.jpg)
Acknowledgments• Thisworkdrawsprimarilyfromcollaborativeexaminationsofthe
psychometricpropertiesofmeasuresofclassroomqualityandchildren’ssocio-emotionaldevelopmentinearlychildhoodfundedbyIESandNIH:
– IESR305A090065– NIHR01HD060711– IESR305A130118– IESR305A160010– IESR305H130012
• Resultsreflectourteams’interpretation(notnecessarilythoseofour
funders).• Presentationreflectsmysynthesis(notnecessarilyindividualteam
members).
![Page 3: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/3.jpg)
PreviewofTalk
• Briefreminderofpolicycontext,especiallyhighstakesuseofmeasures.
• Highlightsfromresearchfindings.
– Generalimportanceoffreshlyconsideringtheevidencebasespecificallyforeachuseofameasure.
• Highstakes,professionaldevelopment,research,selfassessment.
– Currentmeasuresinhighstakesuse.• Dotheypredictlargeschoolreadinessgains?• Dotheysharplymeasureconstructsalignedwithreadinessgains?• Aretheyconstructedformaximalprecision(highsignalvs.noise)?
![Page 4: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/4.jpg)
BriefReminder:PublicInvestmentsand
HighStakesUseofMeasures
![Page 5: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/5.jpg)
ExpandingPublicInvestmentsinEarlyCareandEduca8on
NIEER:State-FundedPrekEnrollment
Percentageof4yearoldsenrolledinstatepre-kdoubled,2002to2010
![Page 6: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/6.jpg)
ExpandingPublicInvestmentsinEarlyCareandEducation
Federalandstatechildcarespendingtripled,1997to2003
![Page 7: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/7.jpg)
PolicyFocusonQualityEarlyCareandEducation
• Policyinitiativesfocusonhigh-qualityearlycareandeducation.
• Typicallywithatleastpartofthegoalbeingsupportforchildren’sschoolreadinessandlaterschoolandlifesuccess.
![Page 8: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/8.jpg)
![Page 9: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/9.jpg)
![Page 10: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/10.jpg)
![Page 11: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/11.jpg)
PolicyFocusonQualityEarlyCareandEducation
• Thisreflectsasensibledesiretoensurethatpublicdollarsinvestinhighqualitysettings.
• But,adesirethatisdifficulttoputintopractice.Ideally,consider:
– Whatarethepolicygoals?– Whataspectsofqualityalignwiththesegoals?– Whatarethebeststrategiesforassessingtheseaspectsof
qualityforthisparticularuse?
• Asthedesiretoassurequalityinpubliclyfundedprogramsgrewrapidly,decisionmakersturnedtoexistingmeasures.
![Page 12: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/12.jpg)
ECERS-RandCLASS• Twoobservationalmeasuresmostwidelyused:ECERS-RandCLASS.
• Similaritiesanddifferences:
– Bothhaveobserversvisitclassroomsforseveralhourstorateactualclassroomactivitiesandinteractions.
– Bothproduceratingsona1to7scale.– But,differentoriginsandstructures.
![Page 13: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/13.jpg)
ECERS• Developedin1970sfromachecklisttohelp
practitionersimprovethequalityoftheirsettings.
• Reflectstheearlychildhoodeducationfield’sconceptofdevelopmentallyappropriatepractice(wholechildapproach,child-initiatedactivities,teacherfacilitationresponsivetoindividualneeds).
• ECERS-R:43scoresassignedbasedonobserved400+indicators.
• Newversion:ECERS-3.
![Page 14: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/14.jpg)
![Page 15: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/15.jpg)
CLASS• Developedin1990s/2000sbeginninginaresearchstudyandlateraimedatprofessionaldevelopmentandcoaching.
• Reflectsdevelopmentaltheoryandresearchandemphasizesteacher-student(adult-child)interactionsastheprimarymechanismofdevelopmentandlearning.
• Observersassimilatewhattheyseetoassignscorestojustafewitems.
![Page 16: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/16.jpg)
Themanualadvises:“BecauseofthehighlyinferentialnatureoftheCLASS,scoresshouldneverbegivenwithout
referringtothemanual.”(Pianta,LaParo&Hamre,p.17,boldin
original)
![Page 17: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/17.jpg)
UseinStateQualityRatingandImprovementSystems
http://qriscompendium.org/top-ten/question-3/
2010~85%ECERS-R~7%CLASS
2014:~87%ECERS-R~37%CLASS
ERS=Broadersuiteofmeasuresforpreschools(ECERS-R),infant/toddlercenters(ITERS-R)andhomes(FCCERS-R).
![Page 18: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/18.jpg)
![Page 19: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/19.jpg)
Example:Illinois’QRISLearningEnvironment
http://www.excelerateillinois.com/docman/resources/2-gold-excelerate-illinois-chart/file
![Page 20: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/20.jpg)
UseinHeadStart
![Page 21: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/21.jpg)
Highlights:EvidenceforHighStakesUse
Iambrieflyhighlightingfindingsintheinterestoftime.Ihavelistedcitations,andwouldbehappytosharefull
publicationsoradditionaldetails.
![Page 22: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/22.jpg)
EvidenceforHighStakesUse• Alternativeoptionsforhighstakesuse.
– Choosetherelativelybestmeasureavailableatthetimeanduse“asis”(evenifevidencelimited)?
– Chooseexistingmeasurebutbuildinrigorousevidencebuildingandpotentialformodificationstomeasureduringuse?
– Requireanabsolutelevelofevidencebeforeuse?
• PotentialforlimitsofECERS-RandCLASSbasedonabsolutelevelofevidence.– Bothweredesignedforpurposesdifferentfromtheircurrenthighstakesuse.– Bothwereembeddedinpractice/professionalandconceptual/empirical
knowledge.– But,neitherusedafullymodernmeasurementdevelopmentand
psychometricapproach(e.g.,IRT)duringdevelopment,especiallyonealigningwiththisparticularhighstakesuse.
![Page 23: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/23.jpg)
DoECERS-RandCLASSpredictlargeschoolreadinessgains?
![Page 24: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/24.jpg)
AccumulatingEvidence:SmallAssociations(OftenNotSig.)
• Burchinal,KainzandCai(2011)– Effectsizes(adjustedcorrelations)of.14andbelowacrosspublishedstudieswitha
rangeofchildoutcomes.– Evenwhenfocusingonlow-incomechildrenandaligningsubscaleswithlanguage,
math,social,andbehavioraloutcomesinnewanalyses,32of36adjustedcorrelationsatorbelow.10.
• Keysandcolleagues(2013)– Averageeffectsizesbetween.01to.05forlanguage,math,social,andbehavioral
outcomes.
• Someevidenceofthresholds(strongereffectsinhigherqualityregion;Burchinal,Zaslow,&Tarullo,2016).– ButstillsmallforECERS-RandCLASS.– Anddemonstratedsmallsamplesizesacrossregionsofqualityandneedfordata
collectedspecificallytotestthisquestion(oversamplehigherquality).
![Page 25: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/25.jpg)
NewDataSyntheses(IESR305A130118)
• Meta-analyses– 13datasetswithmultiplewavesandsubgroups
• Integrativedataanalysis(stackeddatasets)– FACES2000,2003,2006and2009– Greatersamplesizesacrossthequalitycontinuum
• Usednumerousstrategiestoexaminenon-linearity(dummyvariable,quadratic,piecewise,non-parametric).
• Usednumerousstrategiestoexaminesensitivity(e.g.,multipleapproachestomissingdata,complexsamples,qualityandoutcomescoring).
![Page 26: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/26.jpg)
Example:PredictingGrowthinChildren’sVocabulary(PPVT)
Source:StackedanalysisoffourFACEScohorts.AdjustingforfallPPVTscoreaswellaschild’sgender,race-ethnicity,anddisabilitystatus,familyincome,whether3or4atHSenrollment,monthoffallassessments,monthsbetweenfallandspringassessment,andageinmonthsatspringassessment.
ECERS-R Teaching and Interactions
78
80
82
84
86
88
90
92
PPV
T (S
tand
ardi
zed)
1 2 3 4 5 6 7ECERS-R Teaching and Interactions
Linear
Sprin
g PP
VT
(Sta
ndar
dize
d)
78
80
82
84
86
88
90
92
PPV
T (S
tand
ardi
zed)
1 1.5 2 2.5 3 3.5 4 4.5 5CLASS - Instructional Support
Linear
Sprin
g PP
VT
(Sta
ndar
dize
d)
78
80
82
84
86
88
90
92
< 3 3 to < 4 4 to < 5 5 to < 6 6 to 7
PPV
T (S
tand
ardi
zed)
ECERS-R Teaching and Interactions
Dummy Variable
ab cde bef cf ad
Sprin
g PP
VT
(Sta
ndar
dize
d)
78
80
82
84
86
88
90
92
< 1.5 1.5 to < 2 2 to < 2.5 2.5 to < 3 >= 3
PP
VT
(S
tand
ardi
zed)
CLASS - Instructional Support
Dummy Variable
a b abc c Sprin
g PP
VT
(Sta
ndar
dize
d) CLASS Instructional Support
Effect size: .05
Effect size: .04
Max Std. Diff: .13
Max Std. Diff: .17
![Page 27: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/27.jpg)
Example:PredictingGrowthinChildren’sVocabulary(PPVT)
![Page 28: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/28.jpg)
DoECERS-RandCLASSsharplymeasureconstructsalignedwith
readinessgains?
![Page 29: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/29.jpg)
ImportanceofDimensionsofQuality• Ideally,highstakesmeasureswouldbecreated
specificallytomeasureaspectsofqualityalignedwithpolicygoals.
– Content-focusedaspectsofqualityalignedwithparticularreadinessdomainsmayshowstrongerrelationships.
• Alternatively,ifmeasuresdesignedforotherpurposesareused,theyshouldhavecleardefinitionsoftheaspectsofqualitymeasuredandempiricalevidencethatitemsmeasurethem.
– Yet,evidenceforthesubscalesdefinedinECERS-RandCLASSmanuals(andoftenusedinpolicy)islimited.
![Page 30: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/30.jpg)
ECERS-RDimensions:One,Seven,orTwo(Three)?
• TheERSpresumesaqualityprogramsupportsthreebasicneeds(health/safety,positiverelationships,opportunitiesforlearningfromexperience)and“nooneismoreorlessimportantthantheothers”http://ers.fpg.unc.edu/
– TheECERS-Rscaledeveloperssometimesdescribethemeasureascapturingasingleglobalaspectofquality.
– Butitemsareorganizedintosevensubscales,someofwhichonthesurfacealignwithparticularaspectsofquality(personalcare,interaction,activities).
– SomeQRIS,likeIllinois,relyoneitherthetotalorsubscalescores.
• Ontheotherhand,factoranalyseshaveidentified2-3dimensions,andthemostalignedoftheseareoftensomewhatmorehighlycorrelatedwithoutcomes;thesedimensionsaresometimesusedinQRIS.
![Page 31: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/31.jpg)
TwoDimensionsReplicate(IESR305A130118)
• Tosolidifyevidencerelatedtodimensionality,andencourageconsistentpractice/policyuse,wefactor-analyzeddatafromeightsurveys(with14waves)andsynthesizedtheresults.
• Twobroaddimensionsreplicatedacrossthedatasets:– Language-Reasoning/Interaction(LR:Items16-18;Int:Items29-33).
– SpaceFurnishings/Activities(SF:Items2-6;Act:Items19-26&Item28).
• Butassociationswithoutcomesstillsmall.
![Page 32: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/32.jpg)
CLASSPreKDimensions:ThreeDomainsora“Bi-Factor”?
• CLASSPreKmanualproducesscoresinthreebroaddomains:EmotionalSupport,ClassroomOrganization,InstructionalSupport.(http://teachstone.com/)
• InstructionalSupportcapturesdimensionsofteacherpracticemeanttocutacrosscontent(ConceptDevelopment,QualityofFeedback,LanguageModeling).
• Dimensionssometimesmorestronglyrelatedtooutcomeswhenalignedbydomain(butstillsmall).
![Page 33: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/33.jpg)
CLASSPreKDimensions:ThreeDomainsora“Bi-Factor”?
– CLASSdevelopersrecentlypublishedadifferent“bi-factor”structurefortheCLASSPreK(Hamre,Hatfield,Pianta&Jamil,2014)thatdiffersfromthesubscaleswrittenintopolicy.• Onegeneraldimension(responsiveteaching).• Twospecificdimensions(proactivemanagementandroutines;
cognitivefacilitation).• Somewhatmoreconceptuallyconsistentpatternwith
outcomes,althougheffectsizesstillsmall(<.10).
– Wereplicatedthisbi-factorstructure,althoughliketheCLASSteamhadproblemswithconvergence(perhapsduetoitemskewnessandcorrelation).
![Page 34: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/34.jpg)
AreECERS-RandCLASSconstructedformaximal
precision(highsignalvs.noise)?
![Page 35: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/35.jpg)
ScoringStrategiesMayProduceNoise
• ThestructuresofECERS-RandCLASSarequitedifferent,buteachmayincreasenoise.
– ECERS-Rchecklistoriginof400+indicators,butused“stopscoring”sonotallwererated.
– CLASSahighlyinferentialapproach,wherecodersassimilatedallthey’veseenintheirheads(ratherthaninchecklists).
![Page 36: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/36.jpg)
ECERS-RStandard“StopScoring”
§ Conditionsintheindicatorsoflowerscoresmustbemetbeforeindicatorsofhigherscoresareevaluated.
§ Higherscoremaynotalwaysreflecthigherquality,especiallyforaspectsofqualityrelevantforschoolreadiness.
![Page 37: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/37.jpg)
ECERS-RItem10:Meals/Snacks
Source: Harms, T., Clifford, R.M., & Cryer, D. (1998). Early Childhood Environment Rating Scale, Revised Edition. New York, NY: Teachers College Press.
![Page 38: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/38.jpg)
WhatWouldOrderLookLike?
• Ifhigherscoresreflecthigherquality,thenaveragequalityscoresshouldbehigherforcentersratedinhighercategoriesversuslowercategories.
![Page 39: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/39.jpg)
![Page 40: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/40.jpg)
WhatWouldOrderLookLike?
• Ifhigherscoresreflecthigherquality,thenaveragequalityscoresshouldbehigherforcentersratedinhighercategoriesversuslowercategories.
• Alternatively,mayseeunexpectedflatregionsordipsinaveragequalityatsomehighercategoryscores.
![Page 41: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/41.jpg)
![Page 42: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/42.jpg)
Non-OrderinCategoryAverages(IESR305A130118)
• Categoryaveragesoutoforderforsomeitemsinall8datasets.
• Inanalysisofstackeddatafiles(withgreatestprecision)overtwo-thirdsofitemshadatleastonepairofadjacentcategorymeansthatdidnotprogressinastairstepfashion.
• Mostcommonlocationofnon-orderwascategories2-to-3followedbycategories4-to-5.
• TheproblemwasmostevidentinthePersonalCareRoutinesitemsandleastevidentintheLanguage-Reasoningitems.
• Fornearly¾(26of36)items,atleastonecategory-totalpointbiserialcorrelationswasnegative.
![Page 43: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/43.jpg)
IRTModelsAlsoIdentifyDisorder(IESR305A130118)
• PartialCreditModel– Foreveryitem,atleastonepairofadjacentthresholds-latentlevelof
qualitywherearaterisequallylikelytochoosebetweenadjacentcategories–wasoutoforder.
– Thisdisordergenerallyinvolvedthe3rdand5thcategories.
• NominalResponseModel– Categoryboundarydiscriminationsnegative(andscoringfunctionvalues
outoforder)inatleastoneplacefor15of36items(42%).– Categoryboundarydiscriminationssmall(0to0.5)(andscoringfunction
valuesprogressedminimally)for35of36items.– Thisnonordertypicallyoccurredbetweenscores2and3andscores4and5.
![Page 44: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/44.jpg)
Visualiza8on
![Page 45: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/45.jpg)
Visualiza8on
![Page 46: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/46.jpg)
Visualiza8on
![Page 47: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/47.jpg)
Visualiza8on
![Page 48: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/48.jpg)
Interpreta8ons
• Consistentloca8onmayreflectdifferentscoringrulesforevenandoddcategories(requiringhalfvsallindicatorstobemet).
• Greatestproblemsevidentforpersonalcarerou8nesmayreflectmixingofdifferentaspectsofquality(alsoindicatoranalysisinGordonetal.,EED,2015).
![Page 49: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/49.jpg)
Implica8ons• Theseresultscau#onuseofthesimplesum(average)oftherawscores,includinginrela#ontohighstakescutoffs.
• PrestonandReise(2015,p.392)– WhenCBDvaluesarenotposi8ve,aresponseinahighercategory“doesnotindicatemoreofthetrait”thanaresponseinalowercategory.
– “whencategorydis8nc8onsfailtodiscriminate,aresearcherwouldnotwanttouseascoringstrategythataggregatesrawintegeritemscores.”
Preston,K.S.J.&Reise,S.P.(2015).Detec8ngfaultywithin-itemcategoryfunc8oningwiththenominalresponsemodel.InS.P.Reise&D.A.Revicki.(Eds.),Handbookofitemresponsetheorymodeling:Applica#onstotypicalperformanceassessment(pp.386-405).NewYork:Routledge.
![Page 50: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/50.jpg)
CLASSInter-RaterReliability:Is“WithinOne”GoodEnough?
TheCLASS(liketheECERS-Randotherobservationalsystems)assessesagreement“withinone”encompassingbroadregionsofthe7-pointscale.
Ascoreof5,6or7isconsideredinagreementwithamasterscoreof6.
Ascoreof3,4or5isconsideredinagreementwithamasterscoreof4.
Ascoreof1,2or3isconsideredinagreementwithamasterscoreof2.
![Page 51: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/51.jpg)
Pianta,R.C.,LaParo,K.M.,&Hamre,B.K.(2008).ClassroomAssessmentScoringSystem–PreKManual.Bal8more,MD:BrookesPublishing.
ExactAgreementDifficultontheHighlyInferentialSystem
![Page 52: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/52.jpg)
![Page 53: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/53.jpg)
![Page 54: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/54.jpg)
ChallengeofRaterVariance• BasedonHeadStarttraining,CLASSdevelopers(Cash,
Hamre,Pianta,&Myers,2012)reported:
– Exactagreementwaslow:41%overallexactagreementwithmasterscoreintrainingofover2,000HeadStartstaff.
– BlackandLatinoratersplacedtheirInstructionalSupportscoresfartherfromthemasterscoreasdidraterswhodisagreedwithintentionalteachingbeliefs.
• RecentreportonratererrorsinCLASS-S(McCaffreyet
al.,EducationalMeasurement).
![Page 55: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/55.jpg)
Conclusions
![Page 56: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/56.jpg)
Summary:LimitsofAdoptingExistingMeasuresforHighStakesUse
• Whenscrutinizingthesemeasureswhichweredevelopedforotherpurposes,itisnotsurprisingthattherearelimitationsforthewaysinwhichtheyhavebeenadoptedforhighstakespolicyuses.
• ThelimitationsofthereliabilityandvalidityevidenceforECERS-RandCLASSmay,inpart,explaintherelativelylowassociationswithchildren’sdevelopmentalgainsduringpreschool.
![Page 57: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/57.jpg)
AlternativeApproachtoEvaluatingLevelofEvidence
• ConsistentwiththelatestStandardsforEducationalandPsychologicalTestingmayneedtostepbackandconsider:– theintentsofeachresearch,practiceandpolicyuse,– weighthefullbodyofreliabilityandvalidityevidenceagainsteachuse,
– buildincontinuousandlocalvalidationofmeasuresselectedfortheseuses,
– allowfortherefinementofmeasuresoverplaceandtime.
http://www.apa.org/science/programs/testing/standards.aspx
![Page 58: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/58.jpg)
Inshort• Ameasureisnotsta8cally“reliableandvalid”
• Theevidencemustbefullyevaluatedandregularlyrevisited(includinglocally)foreachuse.
– Thebodyofevidenceneededtodemonstratereliabilityandvalidityforprogramself-assessment
– Maybedifferentfromreliabilityandvalidityforteacherprofessionaldevelopment
– Whichmaybedifferentfromreliabilityandvalidityforpolicydecisionmakingandaccountability
![Page 59: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/59.jpg)
AlternativeApproachtoEvaluatingLevelofEvidence
• Asaconcreteexample,ifitisdesirabletodistinguishclassroomsthatfallaboveandbelowspecificcutpoints,asincurrentpolicyuses,thenmeasureswithveryhighinformation(andlowerror)atthosecutpointsareneeded.
• Ifthepolicygoalistoimprovechildren’sschoolreadiness,thenneedagreementondefinitionsofreadinessandtheaspectsofqualitythatsupportthem,andmeasuresdesignedandevaluatedtoassessthoseaspectsofquality.
![Page 60: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/60.jpg)
ConsiderAlternativeApproachestoAccumulatingEvidence
• Continuousandlocalvalidationandimprovementofmeasures.
• Thisapproachcouldpotentiallybenefitfromviewingmeasuresas:– Notfixedinstone(movingawayfromsinglecopyrightedmeasurecontrolledbypublisher).
– Jointlyowned(movingawayfromfinancial/professionalstakeinafixeditem/measure).
![Page 61: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/61.jpg)
LocalValidationCanAlsoEncompassContextualDiversity
• Doesameasurecaptureasingleconceptionofquality?– Isthatconceptionexplicitorimplicit?– Doesthatconceptionmatchwithpolicygoalsandwithon-the-groundpractice,acrosslocalcontexts?
• Doesameasurecapturewellallchildren’sexperiences?– Or,“average”teacherquality,“average”child,“substantialportionoftheday”
![Page 62: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/62.jpg)
FutureDirections
• NewECERS-3– Someimportantchanges(e.g.,expandedcontentonlanguage,literacyandmath).
– Othersremainedthesame(encouragesscoringallindicators,andalternativescoringindevelopment,butmanualretainsstopscoring).
– IES-fundedvalidationstudyofECERS-3.
• Newmeasuredevelopment– IESResearchNetworkonEarlyChildhoodEducation.– Includesassessmentteam(headedbyCarolConnor).– NewmeasuredevelopmentforQRIS.
![Page 63: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/63.jpg)
FutureDirec8ons(cont)• EarlyInvestmentsIni8a8ve(Gordon,Zinsser,Sheridan,Main,
Curby,etal.,IGPA)&EMOTERS(Zinsser,Curby,Gordon,etal.IESR305A160010)– NewmeasureofSocialandEmo8onalTeaching.
• Thevarietyofac8vi8esandprac8cesthatpromoteSELinchildren.
– Oneofthenoveldesignfeatures:videoweeks• Videofullweekperclassroom(panoramic&closeup).• Easiertohavemanycoders,toexamineIRR.• Facilitateiden8fica8onofwithin/acrossdayvaria8on.• Usinginitera8vemeasuredevelopment.• Usinggeneralizabilitytheorytoparsesourcesofvaria8on.
![Page 64: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/64.jpg)
FutureDirec8ons(cont)• IESR305H130012
– Researcher-Prac88onerPartnershipsinEduca8onResearchgrant– Crea8ngaMonitoringSystemforSchoolDistrictstoPromote
Academic,Social,andEmo8onalLearning:AResearcher-Prac88onerPartnership
– CASEL&WashoeCountySchoolDistrict
• UsingIRT(Rasch)approachduringitera8veitemdevelopment.– Refiningconstructdefini8on– Developingitempool– Con8nuousrefinement– Anchor(common)andnew(unique)items
![Page 65: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/65.jpg)
TheRaschRuler
CompetenciesthatarereallyHARDformost
kids
CompetenciesthatarereallyEASYformostkids
KidswhohavetheMOSTcompetency
KidswhohavetheLEASTcompetency
Marks=Competencies
Measures=Kids’Levels
![Page 66: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/66.jpg)
TheRaschRuler
2+2Ifwehadmarksonlyatthebojomoftheruler–justtheeasymathitems–wecouldn’tseparatethestudentswithmoderatelytohighlycompetentmathskills.
![Page 67: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/67.jpg)
-2.00
-1.50
-1.00
-0.50
0.00
0.50
1.00
1.50
2.00
ThewayWCSDdepictedtheruler,showingourmostimprovedsetofitems:
RelaPonshipSkills.
Hardest to Do
Easiest to Do
![Page 68: High Stakes Quality Measures in Early Care and Education · development and coaching. • Reflects developmental theory and research and emphasizes teacher-student (adult-child)](https://reader034.vdocuments.us/reader034/viewer/2022052000/6011fb2f7992420013012ecc/html5/thumbnails/68.jpg)
1.SharingwhatIamfeelingwithothers.
2.JoiningagroupIdon’tusuallysitwithatlunch.
3.TalkingtoanadultwhenIhaveproblemsatschool.
4.Introducingmyselftoanewstudentatschool.
6.Beingpolitetoadults.
5.Genngalongwithmyclassmates.
1=HardesttoDo 6=EasiesttoDo