11

Do We Have To Choose Do We Have To Choose Between Accountability Between Accountability

and Program and Program Improvement?Improvement?

NECTAC’s Measuring Child and Family NECTAC’s Measuring Child and Family Outcomes ConferenceOutcomes Conference

20062006

Kristie Pretti-FrontczakKristie Pretti-FrontczakKent State UniversityKent State University

kprettif@kent.edukprettif@kent.edu Jennifer Grisham-BrownJennifer Grisham-Brown

University of KentuckyUniversity of Kentuckyjgleat00@uky.edujgleat00@uky.edu

22

Overview of SessionOverview of Session Discuss the need for measuring child Discuss the need for measuring child

outcomes as it relates to programming and outcomes as it relates to programming and accountability purposesaccountability purposes

Discuss three issues and associated Discuss three issues and associated recommendations and related researchrecommendations and related research

Discussion is encouraged throughoutDiscussion is encouraged throughout Time will remain at the end for questions Time will remain at the end for questions

and further discussion of what was and further discussion of what was presented.presented.

33

Introductions and Setting a Introductions and Setting a ContextContext

Kristie Pretti-FrontczakKristie Pretti-Frontczak Kent State UniversityKent State University

Jennifer Grisham-BrownJennifer Grisham-Brown University of KentuckyUniversity of Kentucky

Belief/Bias/Recommended PracticeBelief/Bias/Recommended Practice Authentic assessment is critical Authentic assessment is critical

regardless of purposeregardless of purpose

44

CENTRAL QUESTION FOR CENTRAL QUESTION FOR TODAY’S PRESENTATIONTODAY’S PRESENTATION

Can instructional data be used for Can instructional data be used for accountability purposes?accountability purposes?

The Short Answer: Yes (IF) ….The Short Answer: Yes (IF) ….

55

Linked System ApproachLinked System Approach

•Authentic•Involves families•Comprehensive•Common

•Based upon children’s emerging skills•Will increase access and participation

•Systematic

•Ongoing

•Guides decision-making

Assessment Goal Development

Instruction

•Developmentally and individually appropriated

•Comprehensive and common

Evaluation

66

If you….If you…. If you assess young If you assess young

children using a high children using a high quality authentic quality authentic assessment…assessment…

Then you’ll be able to Then you’ll be able to develop high quality develop high quality individualized plans to individualized plans to meet children’s unique meet children’s unique needs….needs….

If you identify the individual If you identify the individual needs of children….needs of children….

77

You’ll want to use the You’ll want to use the information to guide information to guide curriculum development…curriculum development…

If you have a curriculum If you have a curriculum framework that is designed framework that is designed around the individual around the individual needs of the children…needs of the children…

Then you’ll want to Then you’ll want to document that children’s document that children’s needs are being met…needs are being met…

88

Then you’ll need to monitor Then you’ll need to monitor children’s performance children’s performance over time using your over time using your authentic assessment…authentic assessment…

And when you have done And when you have done the authentic assessment the authentic assessment for a second or third time, for a second or third time, you’ll want to jump for joy you’ll want to jump for joy because all of the children because all of the children will have made progress!will have made progress!

99

Three IssuesThree Issues SelectionSelection

ImplementationImplementation

InterpretationInterpretation

1010

Questions around Selecting an Questions around Selecting an AssessmentAssessment

Which tools/processes?Which tools/processes? Which characteristics should be considered?Which characteristics should be considered? What about alignment to state standards or What about alignment to state standards or

Head Start Outcomes?Head Start Outcomes? Use a single/common assessment or a list?Use a single/common assessment or a list? Allow for choice or be prescriptive?Allow for choice or be prescriptive? Who should administer?Who should administer? Where should the assessment(s) be Where should the assessment(s) be

administered?administered?

1111

RecommendationsRecommendations Use an assessment for its intended Use an assessment for its intended

purposepurpose Avoid comparing assessments to one Avoid comparing assessments to one

another – rather compare them to another – rather compare them to stated/accepted criteriastated/accepted criteria Alignment to local/state/federal standardsAlignment to local/state/federal standards Reliable and validReliable and valid Comprehensive and flexible Comprehensive and flexible Link between assessments purposesLink between assessments purposes Link between assessment and interventionLink between assessment and intervention

1212

Recommendations Recommendations ContinuedContinued

Allow for state/local choice if possible Allow for state/local choice if possible Increases likelihood of a match Increases likelihood of a match Increases fidelity and useIncreases fidelity and use Avoids a one size fits all approachAvoids a one size fits all approach

if assessment is flexible and comprehensive if assessment is flexible and comprehensive 1 might work1 might work

Authentic, authentic, authenticAuthentic, authentic, authentic People who are familiarPeople who are familiar Settings that are familiarSettings that are familiar Toys/materials that are familiarToys/materials that are familiar

1313

Generic Validation ProcessGeneric Validation Process Step 1– Create a Master Alignment MatrixStep 1– Create a Master Alignment Matrix

Experts create a master matrixExperts create a master matrix Establish inclusion and exclusion criteriaEstablish inclusion and exclusion criteria

Step 2 –Create Expert Alignment MatrixesStep 2 –Create Expert Alignment Matrixes Experts blind to the master matrix create their own Experts blind to the master matrix create their own

alignment matrixesalignment matrixes

Step 3 – Validate Master Alignment MatrixStep 3 – Validate Master Alignment Matrix Compare master and expert matrixesCompare master and expert matrixes Ensure that all items that should be considered were Ensure that all items that should be considered were

placed on the final matrixesplaced on the final matrixes Examine the internal consistency of the final matrixesExamine the internal consistency of the final matrixes

Allen, Bricker, Macy, & Pretti-Frontczak, 2006; Walker, & Pretti-Allen, Bricker, Macy, & Pretti-Frontczak, 2006; Walker, & Pretti-Frontczak, 2005Frontczak, 2005For more information on crosswalks visit: For more information on crosswalks visit: http://www.fpg.unc.edu/~ECO/crosswalks.cfm or http://aepslinkedsystem.com

1414

Concurrent ValidityConcurrent Validity Purpose:Purpose:

To examine the concurrent validity between a traditional norm-To examine the concurrent validity between a traditional norm-referenced standardized test (BDI-2) and an curriculum-based referenced standardized test (BDI-2) and an curriculum-based assessment (AEPSassessment (AEPS®®))

Subjects:Subjects: 31 Head Start children 31 Head Start children Ranged in age from 48 months to 67 months (Ranged in age from 48 months to 67 months (MM=60.68, =60.68,

SDSD=4.65)=4.65) Methods:Methods:

Six trained graduate students administered the BDI-2 and six Six trained graduate students administered the BDI-2 and six trained Head start teachers administered the AEPStrained Head start teachers administered the AEPS® ® during a during a two-week period. Conducted seven (7) bivariate 2-tailed two-week period. Conducted seven (7) bivariate 2-tailed correlations (Pearson’s and Spearman’s)correlations (Pearson’s and Spearman’s)

Results:Results: Five correlations suggested a moderate to good relationship Five correlations suggested a moderate to good relationship

between the BDI-2 and the AEPSbetween the BDI-2 and the AEPS Two correlations suggested a fair relationship between the BDI-Two correlations suggested a fair relationship between the BDI-

2 and the AEPS2 and the AEPS

Hallam, Grisham-Brown, & Pretti-Frontczak, 2005Hallam, Grisham-Brown, & Pretti-Frontczak, 2005

1515

Concurrent Validity ResultsConcurrent Validity Results AdaptiveAdaptive

Self Care items from the BDI (Self Care items from the BDI (MM = 66.03, = 66.03, SDSD = 6.67) = 6.67) were moderately correlated with Adaptive items from were moderately correlated with Adaptive items from the AEPS (the AEPS (MM = 62.03, = 62.03, SDSD = 13.57), = 13.57), rr = .57, = .57, nn = 31, = 31, pp =.01.=.01.

SocialSocial Personal Social items from the BDI (Personal Social items from the BDI (MM = 175.15, = 175.15, SDSD = =

22.74) had a fair correlation with Social items from the 22.74) had a fair correlation with Social items from the AEPS (AEPS (MM = 80.06, = 80.06, SDSD = 16.33), = 16.33), rr = .50, = .50, nn = 31, = 31, pp =.01. =.01.

CommunicationCommunication Communication items from the BDI (Communication items from the BDI (MM = 121.06, = 121.06, SDSD = =

16.22) were moderately correlated with Social 16.22) were moderately correlated with Social Communication items from the AEPS (Communication items from the AEPS (MM = 88.61, = 88.61, SDSD = = 14.20), 14.20), rr = .54, = .54, nn = 31, = 31, pp =.01. =.01.

1616

Concurrent Validity Results Concurrent Validity Results ContinuedContinued

MotorMotor Gross Motor items from the BDI (Gross Motor items from the BDI (MM = 82.76, = 82.76, SDSD = 4.70) = 4.70)

had a fair correlation with Gross Motor items from the had a fair correlation with Gross Motor items from the AEPS (AEPS (MM = 30.10, = 30.10, SDSD = 6.62), = 6.62), rr = .48, = .48, nn = 31, = 31, pp =.01. =.01.

Fine Motor items from the BDI (Fine Motor items from the BDI (MM = 52.45, = 52.45, SDSD = 5.30) = 5.30) were moderately correlated with Fine Motor items from were moderately correlated with Fine Motor items from the AEPS (the AEPS (MM = 26.39, = 26.39, SDSD = 5.68), = 5.68), rr = .58, = .58, nn = 31, = 31, pp =.01. =.01.

Perceptual Motor items from the BDI (Perceptual Motor items from the BDI (MM = 27.73, = 27.73, SDSD = = 3.63) were moderately correlated with Fine Motor items 3.63) were moderately correlated with Fine Motor items from the AEPS (from the AEPS (MM = 26.39, = 26.39, SDSD = 5.68), = 5.68), rr = .58, = .58, nn = 31, = 31, pp =.01.=.01.

CognitiveCognitive Cognitive items from the BDI (Cognitive items from the BDI (MM = 135.85, = 135.85, SDSD = 23.44) = 23.44)

were moderately correlated with Cognitive items from were moderately correlated with Cognitive items from the AEPS (the AEPS (MM = 81.26, = 81.26, SDSD = 24.26), = 24.26), rr = .71, = .71, nn = 31, = 31, pp =.01.=.01.

1717

Head Start/University Partnership grant Head Start/University Partnership grant (Jennifer Grisham-Brown/Rena Hallam)(Jennifer Grisham-Brown/Rena Hallam)

Purpose: To build the capacity of Head Start Purpose: To build the capacity of Head Start programs to link child assessment and programs to link child assessment and curriculum to support positive outcomes for curriculum to support positive outcomes for preschool childrenpreschool children Focus on mandated Head Start Child OutcomesFocus on mandated Head Start Child Outcomes

Concepts of PrintConcepts of Print Oral LanguageOral Language Phonological AwarenessPhonological Awareness Concepts of NumberConcepts of Number

Project LINKProject LINK

Grisham-Brown, Hallam, & Brookshire, in press; Hallam, Grisham-Brown, Hallam, & Brookshire, in press; Hallam, Grisham-Brown, Gao, & Brookshire, in pressGrisham-Brown, Gao, & Brookshire, in press

1818

PRELIMINARY RESULTS FROM PRELIMINARY RESULTS FROM PROJECT LINK: Classroom PROJECT LINK: Classroom

QualityQuality No significant differences between No significant differences between

control and intervention classrooms control and intervention classrooms on global quality (ECERS-R)on global quality (ECERS-R)

The quality of the language and The quality of the language and literacy environment (ELLCO) was literacy environment (ELLCO) was superior in intervention classrooms; superior in intervention classrooms; significant in pilot classroomssignificant in pilot classrooms

1919

PRELIMINARY RESULTS FROM PRELIMINARY RESULTS FROM PROJECT LINK: Child OutcomesPROJECT LINK: Child Outcomes

Change scores in Change scores in InterventionIntervention classrooms are classrooms are significantlysignificantly higher than Control classrooms on higher than Control classrooms on letter-word recognition subscale of FACES letter-word recognition subscale of FACES battery.battery.

The mean change scores were The mean change scores were higher higher (although (although not significantly so) on not significantly so) on sevenseven additional additional subscales (11 total) of FACES battery - nearing subscales (11 total) of FACES battery - nearing significance on PPVTsignificance on PPVT

Results would probably have been greater with Results would probably have been greater with larger samplelarger sample

Results will be duplicated this year Results will be duplicated this year

2020

Questions Around Training, Questions Around Training, Implementation, and UseImplementation, and Use

Who will implement?Who will implement? What level of training and support will staff What level of training and support will staff

need?need? What will be topics of training?What will be topics of training? Who will provide training and support?Who will provide training and support? How will you know if staff are reliably How will you know if staff are reliably

collecting data?collecting data? How ill you know if staff are procedurally How ill you know if staff are procedurally

collecting data with fidelity?collecting data with fidelity?

2121

Recommendations:Recommendations: Training/Follow-upTraining/Follow-up

FormatFormat TopicsTopics Classroom and administrativeClassroom and administrative

Valid and reliableValid and reliable Will require training and supportWill require training and support Will require seeing assessment as a Will require seeing assessment as a

critical part of intervention/curriculum critical part of intervention/curriculum planningplanning

2222

What it takes!What it takes! Who?Who?

All classroom staffAll classroom staff Administrators/consultantsAdministrators/consultants

What?What? InstrumentInstrument Methods (e.g., observations, anecdotals, work Methods (e.g., observations, anecdotals, work

samples)samples) Data entry/managementData entry/management Relationship to everything else (I.e., Linked Relationship to everything else (I.e., Linked

system)system)

2323

What it takes (cont.)What it takes (cont.) How?How?

Training that is “chunked”Training that is “chunked” Self-assessmentSelf-assessment Follow-up, follow-up, follow-up Follow-up, follow-up, follow-up

MentoringMentoring On-site technical assistanceOn-site technical assistance Access to someone to call!Access to someone to call!

Involvement of administrationInvolvement of administration

2424

Can preschool teachers (with Can preschool teachers (with appropriate training) collect appropriate training) collect reliable data with fidelity?reliable data with fidelity?

Reliability studyReliability study Fidelity studyFidelity study Accuracy studyAccuracy study

Brown, Kowalski, Pretti-Frontczak, Uchida, & Sacks, 2002Brown, Kowalski, Pretti-Frontczak, Uchida, & Sacks, 2002 ; Grisham-Grisham-Brown, Hallam, & Pretti-Frontczak, in preparationBrown, Hallam, & Pretti-Frontczak, in preparation

2525

Inter-Rater ReliabilityInter-Rater Reliability Subjects:Subjects:

7 Head Start Teachers7 Head Start Teachers 7 Head Start Teaching Assistants7 Head Start Teaching Assistants

Method:Method: Practiced scoring AEPS items from videoPracticed scoring AEPS items from video Scored AEPS items; Checked against master score Scored AEPS items; Checked against master score

provided by authorprovided by author Results:Results:

7 of 7 teachers reached reliability at 80% or higher 7 of 7 teachers reached reliability at 80% or higher (range 85% - 93%)(range 85% - 93%)

5 of 7 teaching assistants reached reliability at 80% or 5 of 7 teaching assistants reached reliability at 80% or higher (range 75% - 90%)higher (range 75% - 90%)

2626

Fidelity StudyFidelity Study Subjects:Subjects:

Six (6) Head Start teachers/teaching assistants who Six (6) Head Start teachers/teaching assistants who reached 80% or higher on interrater reliability studyreached 80% or higher on interrater reliability study

Method:Method: Used fidelity measure to check teachers’ implementation of Used fidelity measure to check teachers’ implementation of

authentic assessment within seven (7) planned activitiesauthentic assessment within seven (7) planned activities Six (6) Authentic Assessment VariablesSix (6) Authentic Assessment Variables

set up and preparation; decision making; materials; set up and preparation; decision making; materials; choice; embedding; and procedure choice; embedding; and procedure

ProceduresProcedures Observed participants collecting AEPS® data during each 7 Observed participants collecting AEPS® data during each 7

small group activities small group activities Observed participants 7 times for up to 10 minutes per Observed participants 7 times for up to 10 minutes per

activityactivity

2727

Average Ratings on Six Authentic Assessment Variables Average Ratings on Six Authentic Assessment Variables across Observations and Activities by Teacheracross Observations and Activities by Teacher

0

0.5

1

1.5

2

2.5

3

Abby Amanda Kate Reba Sarah Vicky

Teachers

Aver

age

Ratin

gs

Setup and Preparation

Decision Making

Materials

Embedding

Child Choice

Procedures

2828

Average Ratings on Six Authentic Assessment Variables Average Ratings on Six Authentic Assessment Variables across Observations for Seven Different Activitiesacross Observations for Seven Different Activities

0

1

2

3

Outdoor Play DramaticPlay

A BookAbout Me

Playdough Manipulatives Story Time Snack

Authenic Assessment Activities

Ave

rage

Rat

ing

Setup and PreparationDecision MakingMaterialsEmbeddingChild ChoiceProcedures

2929

Accuracy StudyAccuracy Study Study designed to investigate the accuracy Study designed to investigate the accuracy

of teachers’ assessments of children’s skills of teachers’ assessments of children’s skills and abilities using observational and abilities using observational assessmentassessment Examined the degree of agreement between Examined the degree of agreement between

assessments of children’s Language and Literacy assessments of children’s Language and Literacy and Early Math skills made by their teachers and Early Math skills made by their teachers using an observational assessment instrument using an observational assessment instrument and assessments of the same skills made by and assessments of the same skills made by researchers using a demand performance researchers using a demand performance instrument. instrument.

Brown, Kowalski, Pretti-Frontczak, Uchida, & Sacks, 2002Brown, Kowalski, Pretti-Frontczak, Uchida, & Sacks, 2002

3030

MeasuresMeasures Observational Measure - Galileo System’s Observational Measure - Galileo System’s

Scales (Bergan, Bergan, Rattee, & Feld, 2001)Scales (Bergan, Bergan, Rattee, & Feld, 2001) Language & Literacy-Revised Ages 3-5 (n=68 items full scale)Language & Literacy-Revised Ages 3-5 (n=68 items full scale) Early Math-Revised Ages 3-5 (n=68 items full scale)Early Math-Revised Ages 3-5 (n=68 items full scale)

Demand Performance MeasureDemand Performance Measure Items that could be readily assessed in individual, one-session, Items that could be readily assessed in individual, one-session,

performance-based interviews with children were selected performance-based interviews with children were selected from the Galileo System’s scales and converted into demand from the Galileo System’s scales and converted into demand performance tasks to create two performance measures performance tasks to create two performance measures

Language & Literacy (n=21 items) Language & Literacy (n=21 items) Early Math (n=23 items). Early Math (n=23 items).

Items varied in difficulty and knowledge domain assessed.Items varied in difficulty and knowledge domain assessed. Standardized sets of materials for administering tasks were Standardized sets of materials for administering tasks were

also developed (e.g., index cards with printed objects, books, also developed (e.g., index cards with printed objects, books, manipulatives, etc.). manipulatives, etc.).

The performance measures were piloted with preschoolers in The performance measures were piloted with preschoolers in two regions of the state and revised accordingly. two regions of the state and revised accordingly.

3131

ProceduresProcedures Trained research assistants visited sites across Trained research assistants visited sites across

the state:the state: collected data teachers entered into the relevant observation collected data teachers entered into the relevant observation

scales of the Galileo System; and scales of the Galileo System; and administered the Performance Measures. administered the Performance Measures.

In order to ensure that the most up-to-date In order to ensure that the most up-to-date information was obtained from the Galileo System, information was obtained from the Galileo System, data were collected during the 2 weeks prior to data were collected during the 2 weeks prior to and following a state mandated entry date. and following a state mandated entry date.

Order of administration of Performance Measures Order of administration of Performance Measures was counterbalanced across assessment domains.was counterbalanced across assessment domains.

3232

ParticipantsParticipants 122 children 122 children

ranged in age from 3 to 6 years (ranged in age from 3 to 6 years (MM=4 years, 11 =4 years, 11 months)months)

100% in state-funded Head Start programs100% in state-funded Head Start programs 66 teachers66 teachers

Areas in which children are servedAreas in which children are served 47% urban47% urban 41% suburban/small town41% suburban/small town 11% rural11% rural

Representation by use of the Galileo SystemRepresentation by use of the Galileo System 38% first-year users38% first-year users 32% second-year users32% second-year users 23% third-year users23% third-year users

3333

ConclusionsConclusions Overall, levels of Overall, levels of concordanceconcordance were moderate. were moderate.

In the domain in which teachers were most conservative in In the domain in which teachers were most conservative in attributing abilities to children, attributing abilities to children, Language & LiteracyLanguage & Literacy, , there was the there was the mostmost amount of amount of agreement agreement between data between data teachers entered into the Galileo System and the teachers entered into the Galileo System and the Performance Measure (71%).Performance Measure (71%).

In the domain in which teachers were most generous in In the domain in which teachers were most generous in attributing abilities to children, attributing abilities to children, Early MathEarly Math, there was the , there was the leastleast amount of amount of agreementagreement between the data teachers between the data teachers entered into the Galileo System and the Performance entered into the Galileo System and the Performance Measure (66%).Measure (66%).

ReliabilityReliability Teachers using the naturalistic observation instrument (the Teachers using the naturalistic observation instrument (the

Galileo System) are not providing inflated estimates of Galileo System) are not providing inflated estimates of children’s skills and abilities.children’s skills and abilities.

However, they may be underestimating children’s skills and However, they may be underestimating children’s skills and abilities in the domain of Language & Literacy.abilities in the domain of Language & Literacy.

3434

Questions Around Questions Around Interpreting the EvidenceInterpreting the Evidence

What is evidence?What is evidence? Where should the evidence come from?Where should the evidence come from? What is considered “performing as What is considered “performing as

same age peers”?same age peers”? How should decisions be made?How should decisions be made? Who should interpret the evidence?Who should interpret the evidence? How can the ECO child summary form How can the ECO child summary form

be used?be used?

3535

What is Evidence?What is Evidence? Information (observations, scores, Information (observations, scores,

permanent products) about a child’s permanent products) about a child’s performance across the three OSEP performance across the three OSEP outcomesoutcomes Positive social-emotional skills (including social Positive social-emotional skills (including social

relationships)relationships) Acquisition and use of knowledge and skillsAcquisition and use of knowledge and skills Use of appropriate behaviors to meet their needsUse of appropriate behaviors to meet their needs

The amount of type of evidence for each The amount of type of evidence for each outcome will varyoutcome will vary

3636

Where should the evidence Where should the evidence come from?come from?

Multiple time periods Multiple time periods Multiple settings Multiple settings Multiple peopleMultiple people

ParentsParents ProvidersProviders Those familiar with the childThose familiar with the child

Multiple measures (should be empirically aligned)Multiple measures (should be empirically aligned) ObservationsObservations InterviewsInterviews Direct testsDirect tests

3737

Required DecisionsRequired Decisions Decision for Time 1Decision for Time 1

Is the child performing as same age peers? Is the child performing as same age peers? YesYes NoNo

Decision for Time 2 Decision for Time 2 Did the child make progress?Did the child make progress?

YES – and performance is as you would expect of same age YES – and performance is as you would expect of same age peerspeers

YES – and performance is not as you would expect of same YES – and performance is not as you would expect of same age peersage peers

NO progress was madeNO progress was made

3838

Things to Keep in MindThings to Keep in Mind ““Typical/performing as same age peers” is Typical/performing as same age peers” is

NOT averageNOT average ““Typical” includes a very broad range of Typical” includes a very broad range of

skills/abilitiesskills/abilities Child can be “typical” in one OSEP area and Child can be “typical” in one OSEP area and

not anothernot another Progress is any amount of changeProgress is any amount of change

Raw score changed by 1 pointRaw score changed by 1 point A single new skill was reachedA single new skill was reached Child needs less assistance at time twoChild needs less assistance at time two

If using the Child Outcome Summary FormIf using the Child Outcome Summary Form Child’s rating score does NOT have to change from Child’s rating score does NOT have to change from

time 1 to time 2 to demonstrate progresstime 1 to time 2 to demonstrate progress Progress can be continuing to develop at a typical Progress can be continuing to develop at a typical

rate (i.e., maintain typical status)rate (i.e., maintain typical status)

3939

How Should the Required How Should the Required Decisions be Made?Decisions be Made?

Some assessments will make the decisionSome assessments will make the decision Standard scoreStandard score Residual Change ScoresResidual Change Scores Goal Attainment Scaling Goal Attainment Scaling Number of objectives achieved/Percent Number of objectives achieved/Percent

objectives achieved objectives achieved Rate of GrowthRate of Growth Item Response Theory (cutoff score)Item Response Theory (cutoff score) Proportional Change IndexProportional Change Index

4040

Making Decisions ContinuedMaking Decisions Continued Regardless - Team conclusions….Regardless - Team conclusions….

should be based on multiple sourcesshould be based on multiple sources should be based on valid and reliable should be based on valid and reliable

informationinformation should be systematicshould be systematic

Can use the Child Outcome Summary FormCan use the Child Outcome Summary Form Will help with required decision and provide more Will help with required decision and provide more

information for use at the local or state levelinformation for use at the local or state level

4141

Child Outcome Summary Child Outcome Summary FormForm Single rating scale that can be used to Single rating scale that can be used to

systematize information and make decisionssystematize information and make decisions After reviewing the evidence rate the child’s After reviewing the evidence rate the child’s

performance on each of the 3 outcomes from performance on each of the 3 outcomes from 1 to 71 to 7

Currently a score of 6 or 7 is considered to be Currently a score of 6 or 7 is considered to be performance that is similar to same age performance that is similar to same age peers.peers.

CompleteCompletelyly

SomewhatSomewhat EmergingEmerging Not YetNot Yet

77 66 55 44 33 22 11

4242

Getting from 7 to 3Getting from 7 to 3 Seven point rating scale just Seven point rating scale just

summarizes the evidencesummarizes the evidence The required interpretation is still The required interpretation is still

neededneeded

a.a. % of children who reach or maintain functioning % of children who reach or maintain functioning at a level comparable to same-age peersat a level comparable to same-age peers

b.b. % of children who improve functioning but are % of children who improve functioning but are not in “a”not in “a”

c.c. % of children who did not improve functioning% of children who did not improve functioning

4343

ExampleExample During a play-based assessment, IFSP/IEP team During a play-based assessment, IFSP/IEP team

administeredadministered a norm-referenced test a norm-referenced test a curriculum-based assessmenta curriculum-based assessment an interview with relevant caregiversan interview with relevant caregivers

The team then summarized the child’s performance The team then summarized the child’s performance using each method’s internal summary proceduresusing each method’s internal summary procedures Calculated a standard scoreCalculated a standard score Derived a cutoff scoreDerived a cutoff score Narratively summarized interviewNarratively summarized interview

Lastly the team rated the child’s overall performance Lastly the team rated the child’s overall performance using ECO’s Child Outcome Summary Form for each of using ECO’s Child Outcome Summary Form for each of the 3 OSEP outcomesthe 3 OSEP outcomes

Two years later as the child was being transitioned out Two years later as the child was being transitioned out of the program, the results from a comprehensive of the program, the results from a comprehensive curriculum-based assessment were reviewedcurriculum-based assessment were reviewed The child’s performance rated using ECO’s Child Outcome The child’s performance rated using ECO’s Child Outcome

Summary Form Summary Form The team made a determination of progressThe team made a determination of progress

4444

Example ContinuedExample ContinuedTime OneTime One

Outcome OneOutcome One Rating = 6Rating = 6 Interpretation = “Typical”Interpretation = “Typical”

Outcome TwoOutcome Two Rating = 5Rating = 5 Interpretation = “Not Interpretation = “Not

typical”typical” Outcome ThreeOutcome Three

Rating = 3Rating = 3 Interpretation = “Not Interpretation = “Not

typical”typical”

Time TwoTime Two Outcome OneOutcome One

Rating = 6Rating = 6 Interpretation = aInterpretation = a

Outcome TwoOutcome Two Rating = 5Rating = 5 Interpretation = b*Interpretation = b*

Outcome ThreeOutcome Three Rating = 5Rating = 5 Interpretation = bInterpretation = b

*Remember the Child Outcome Summary Form 7 point rating is a summary of performance not of progress. At time two, teams are also prompted to consider progress.

4545

Fact or FictionFact or Fiction

1.1. Someone has the answers and if I look Someone has the answers and if I look long enough I’ll have them too.long enough I’ll have them too.

2.2. Everything has to be perfect this first Everything has to be perfect this first time around.time around.

3.3. Research doesn’t matter – just getting Research doesn’t matter – just getting the data submitted.the data submitted.

4.4. I really do believe that garbage in is I really do believe that garbage in is garbage out but at the end of the day – garbage out but at the end of the day – just want the data.just want the data.

4646

Overall Synthesis and Overall Synthesis and RecommendationsRecommendations

Rigorous implementation of curriculum-Rigorous implementation of curriculum-based assessments requires extensive based assessments requires extensive professional development and support of professional development and support of instructional staff.instructional staff.

Findings suggest that CBAs, Findings suggest that CBAs, when implemented with rigor, have the when implemented with rigor, have the potential to provide meaningful child potential to provide meaningful child progress data for program evaluation and progress data for program evaluation and accountability purposes.accountability purposes.

4747

“And that’s our outcomes measurement system. Any questions?”

4848

ReferencesReferences Allen, D., Bricker, D., Macy, M., & Pretti-Frontczak, K. (2006, February). Allen, D., Bricker, D., Macy, M., & Pretti-Frontczak, K. (2006, February).

Providing Accountability Data Using Curriculum-Based AssessmentsProviding Accountability Data Using Curriculum-Based Assessments. Poster . Poster presented at the Biannual Conference on Research Innovations in Early presented at the Biannual Conference on Research Innovations in Early Intervention, San Diego, California.Intervention, San Diego, California.

Brown, R. D., Kowalski, K., Pretti-Frontczak, K., Uchida, C., & Sacks, D. (2002, Brown, R. D., Kowalski, K., Pretti-Frontczak, K., Uchida, C., & Sacks, D. (2002, April). April). The reliability of teachers’ assessment of early cognitive development The reliability of teachers’ assessment of early cognitive development using a naturalistic observation instrument.using a naturalistic observation instrument. Paper presented at the 17th Paper presented at the 17th Annual Conference on Human Development, Charlotte, North Carolina.Annual Conference on Human Development, Charlotte, North Carolina.

Grisham-Brown, J., Hallam, R., & Brookshire, R. (in press). Using authentic Grisham-Brown, J., Hallam, R., & Brookshire, R. (in press). Using authentic assessment to evidence children’s progress towards early learning assessment to evidence children’s progress towards early learning standards. Early Childhood Education Journal.standards. Early Childhood Education Journal.

Grisham-Brown, J., Hallam, R., & Pretti-Frontczak, K. Measuring child Grisham-Brown, J., Hallam, R., & Pretti-Frontczak, K. Measuring child outcomes using authentic assessment practices. outcomes using authentic assessment practices. Journal of Early Intervention Journal of Early Intervention (Innovative Practices)(Innovative Practices). Manuscript in preparation. . Manuscript in preparation.

Hallam, R., Grisham-Brown, J., Gao, X., & Brookshire, R. (in press). The Hallam, R., Grisham-Brown, J., Gao, X., & Brookshire, R. (in press). The effects of outcomes-driven authentic assessment on classroom quality. effects of outcomes-driven authentic assessment on classroom quality. Early Early Childhood Research and Practice.Childhood Research and Practice.

Hallam, R., Grisham-Brown, J., & Pretti-Frontczak, K. (2005, October). Hallam, R., Grisham-Brown, J., & Pretti-Frontczak, K. (2005, October). Meeting the demands of accountability through authentic assessmentMeeting the demands of accountability through authentic assessment . Paper . Paper presented at the International Division of Early Childhood Annual presented at the International Division of Early Childhood Annual Conference, Portland, OR.Conference, Portland, OR.

Walker, D., & Pretti-Frontczak, K. (2005, December). Walker, D., & Pretti-Frontczak, K. (2005, December). Issues in selecting Issues in selecting assessments for measuring outcomes for young childrenassessments for measuring outcomes for young children. Paper presented at . Paper presented at the OSEP National Early Childhood Conference, Washington, D.C.the OSEP National Early Childhood Conference, Washington, D.C. (http://www.nectac.org/~meetings/nationalDec05/mtgPage1.asp?enter=no)