Preventing the MisMeasure of Young Children:

LINKing Authentic Assessment and Early Childhood Intervention:

Best Measures for Best Practices

STEPHEN J. BAGNATO, Ed.D., NCSPProfessor of Psychology & PediatricsDirector, Early Childhood Partnerships

University of Pittsburgh/Office of Child Development (OCD)bagnatos@pitt.edu

www,earlychildhoodpartnerships.org

www.earlychildhoodpartnerships.org

Misrepresenting children through mismeasuring them

denies children their rights to beneficial expectations and

opportunities (Neisworth & Bagnato, 2004, p.198).

Workshop ObjectivesTo describe the difference between conventional tests/testing vs. the authentic assessment alternative in early childhood intervention (ECI)To define the 8 developmentally-appropriate standards to analyze the quality of measures to fulfill ECI purposes To identify and describe the highest quality consumer-rated authentic assessments for ECI purposes

Workshop Topics1. Why Are Changes Needed in Early Childhood

Measurement?2. What is Wrong with Conventional Testing of Preschool

Children?3. What Are the Purposes for Assessment of Young

Children?4. What is the Authentic Assessment Alternative to

Conventional Testing? 5. Are There Professional Standards for Authentic DAP

Assessment6. What Are Some Exemplary Authentic Assessment

Measures7. What is the Continuum of Measurement Contexts 8. What Are Guidepoints for Authentic Assessment in

Action?

Why Are Changes Needed in Early Childhood

Measurement?

Rationale for Changes in Conventional Early Childhood Testing Practices

Lag with philosophy, policies, reforms in early childhood and early intervention: developmentally-appropriate practice (DAP); inclusion; natural environments; equity; response to intervention (RTI);

Emphasize developmentally “inappropriate” practices: decontextualized environments; unnatural child behaviors; no universal design;

Fail to link or align with curricula and standards: No treatment validity;

Fail to collaborate with parents as “core” partners with professionals in appraisal process

What is Wrong with Conventional Testing of

Preschool Children?

Inauthentic Measurement in Early Childhood

“Much of developmental psychology [early childhood testing] as it now exists is the science of the strange behavior of children with strange adults in strange settings for the briefest possible periods of time.” (p.19)

(Bronfenbrenner, 1979)

What is Wrong with Conventional Tests and Testing?

Contrived situations & settings, not real-life activities; Testing by unfamiliar “experts”; Expected “test behavior” is unnatural and not play-based; Insensitive to developmental gains; Content not worthwhile or functional; No match with curricular content or standards “Scripted” materials & procedures; Norms on only typically developing children; Inequitable for children with differences and disabilities; Discourages parent input; Diagnostic for disabilities not capabilities; Lacks a research evidence-base in early childhood

Conclusion on Conventional Testing of High-Risk Students

National Academy of Sciences (2002)“… the committee regards the

effort to assess students’ decontextualized potential or ability as inappropriate and scientifically invalid (pp. 8-23)”

Why Do We Assess Young Children?

The Justifiable Purposes for Assessment in Early Childhood/Early Intervention

ASSESSMENT

“Assidere” – to sit beside and get to know; derives from

associate and assembly

Early Intervention Assessment Purposes

Finding and screening Determining service eligibility Assessing functional developmental and behavioral

competencies Facilitating parent-professional teamwork and

collaborative decision-making Determining program goals and services via

IEP/IFSP development Monitoring child and family progress Evaluating program impact and outcomes Diagnosing by response to intervention (RTI)

Justifiable Measurement Purposes for Early Childhood Intervention

Screening and early detection for instructional needs, not exclusionIndividual curriculum goal-planningMonitoring curricular progress toward standardsTracking individual and group progressDocumenting response to instruction, tutoring, therapy, interventionEvaluating program impact and outcomes

Primary Measurement Purpose in Early Childhood Intervention

“The overarching purpose of assessment in early childhood intervention is the design of individual plans for care, instruction and therapy” (Bricker etal., 2000; Sandall etal., 2000; Bagnato & Neisworth, 1997).

What is the Authentic Assessment Alternative to

Conventional Testing?

Definition

“Authentic assessment refers to the systematic recording of developmental

observations overtime about the naturally occurring behaviors of young

children in daily routines by familiar andknowledgeable caregivers

in the child’s life.” (Bagnato & Yeh Ho, 2006)

Authentic Assessment in Early Childhood Intervention

Natural observations of ongoing child behavior in everyday settings and routines vs. contrived arrangements;

Reliance on informed caregivers (teachers, parents, team) to collect convergent, multi-source data across settings;

Ongoing monitoring of skill acquisition in natural activities (i.e., preschool, home, community) over sufficient time periods and occasions

Curriculum-based measures linked to program goals, content, standards, & expected outcomes;

Universal design; equitable assessment content and methods;

Intra-individual child progress supplemented by inter-individual normative comparisons;

NAEYC/DEC/HS DAP Assessment Standards & Practices

Our Mantra:

No Tabletop Testing !

LeVan, R (2007). Testing Without Tests--

Shopping for Skills

in Bagnato (2007). Authentic Assessment for Early Childhood Intervention: Best Practices. New York: Guilford Press.

Adventures in Assessment with Clarence

What is the Continuum of Measurement Contexts to Guide

DAP Assessment for Early Childhood Intervention?

Authentic Assessment: Continuum of Measurement Contexts

Natural Clinical

Analog Simulated

Continuum of Measurement Contexts

Contextualized>>>>>Decontextualized

NATURALWHERE Everyday

routines

WHAT Spontaneous behaviors

HOW Direct observation;

report; interview

FBA; DOCS; ABAS

ANALOG Everyday routines Prompted natural

behaviors Direct observation;

report; interview CSBS;DOCS;

AEPS; CCCSN

Continuum of Measurement Contexts:

Contextualized>>>>>Decontextualized

SIMULATED Replica or setup

situations On-demand

behaviors Structured tests;

observation schedules

TPBA;BDI; Arena

CLINICAL Laboratory

situations Standard

responses to standard stimuli

Psychometric test instruments

SBIS;BSID;WPPSI

Authentic Assessment Activity View video segment(s) of assessment

situations--Jordan1. Identify the context type from the

continuum2. Identify the likely team model3. Describe the impact of the

context/model on the child and parent in the segment(s)

Are There Professional Standards for Choosing and Using

Authentic Assessment Measures in Early Childhood/Early

Intervention?

Selected Professional Standards for Early Childhood Assessment

(DEC, 2004; NAEYC, 1997, HS, 2000)

Reliance on developmental observations-ongoing observational assessments overtime

Performance on “authentic, not contrived, activities” Integration of assessment and curriculum Child progress on past performances as the reference, not

group norms Choose materials that accommodate the child’s special

functional needs Use only measures that have high treatment validity Rely on curriculum-based measures as the foundation or

“mutual language” for team assessments Defer a diagnosis until evaluation of a child’s response to a

tailored set of interventions Use scales with sufficient item density to detect even small

increments of progress

AUTHENTIC ASSESSMENT IN EARLY CHILDHOOD & EARLY INTERVENTION

8 Developmentally-Appropriate Standards

for Professional Practices

ACCEPTABILITYSocial validity; perceived worth or appropriateness of

the scale’s item content as perceived by parents & caregivers

Social competencies: Contains socially valued & relevant content?

Social detection: Yields socially noticeable functional changes in real-life?

Social appropriateness: Uses methods acceptable to parents and caregivers?

AUTHENTICITY

Extent to which assessment content & methods sample naturally occurring behaviors in everyday settings

Functional content: Necessary competencies for real-life participation?

Observational methods: In-vivo observations & reports of familiar people?

Natural situations: Captures data in familiar classroom, home, play, & community settings

Nonauthentic Content in Commonly Used Early Childhood Tests:

Removes/replaces pegs from pegboard

Sorts dry macaroni shapes into sorting tray

Stacks blocks horizontally/vertically

Strings 3-4 beads Colors within lines of a circle

Nonauthentic Content (Cont)

Completes three-hole puzzleUncovers toy hidden under cupPoints to the picture of “pouring”Points to picture of “what swims?”Eats items of a meal in an acceptable

order

Authentic Assessment Content:Carolina Curriculum for Young Children with Special

Needs (2004)

Engages in adult role-playingBuilds large structures from blocks or

chairs and centers play around themUses “I” instead of given nameFixes bowl of dry cereal with milk

independentlyIdentifies own feelings

Assessment, Evaluation, and Programming System (AEPS, 2004)

Locates objects, people, and events without contextual cues

Initiates communication with a familiar adult Moves around barrier to change location Conforms to game rules Demonstrates understanding of three different

shapes Initiates cooperative play with peers Uses more than one strategy to solve problems

COLLABORATIONParent/professional and interdisciplinary teamwork

Interdisciplinary procedures: Encourages parent/professional teamwork & models?

Family/culture-centered practices: Enables collaboration via “friendly” materials, family preferences, & respect for cultural values, actions?

EVIDENCEHas clear practice-based evidence for early childhood intervention

purposes; materials designed, developed, & field-validated for young children, particularly those with special needs

Professional standards: Adheres to unique ECI professional philosophy & practice expectations (e.g., NAEYC, DEC, HS)?

Diversity representation: Incorporates diverse children, including disabilities in the standardization group, if norm-referenced

Disability specificity: Provides evidence of pooled typical/atypical norms or disability-specific standardization samples

Early intervention validation: Shows field-validation studies for specific ECI purposes

MULTI-FACTORSCollection of data across multiple methods, sources,

settings, & occasions

Multiple situations: Gathers and records data across diverse places, routines, & times?

Multiple persons: Pools data from several familiar caregivers?

Multiple methods: Gathers data via several formats?

Multiple time-points: Synthesizes status/progress data over serial assessments?

SENSITIVITYDensity of items in the skill sequence

Functional hierarchy: Organizes content in sequences of graduated developmental competencies/instructional steps?

Sufficient item-floors: Contains sufficient number of items in sequence to record even low functional levels & small increments of progress?

Graduated scoring: Uses multi-point ratings to document status/gain & performance conditions

UNIVERSALITYDesign and/or accommodations which enable all children to show

their underlying and often unrealized functional skills

Equitable design: Develops items for child to show underlying competence via functional vs topographical content and universal design concepts?

Alternate materials: Allows use of substitute, multi-sensory objects to elicit child’s functional capabilities?

Alternate responses: Allows substitute ways for individual children to show competence despite limitations?

Universal Design(Center for Universal Design, NCSU, 1997;

Harcourt Assessment, 2003)

“The design of products and environments to be useable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (Mace, 1998).

Equitable Assessment Content:Function vs. Topography

for Universal Design

Function ____across the room ____on toys to make

them work _____________needs ________hidden toys ________objects or

pictures requested

Topography Walks across room Uses hands to make

toys work Talks to communicate Uncovers hidden toys Points to correct

pictures

UTILITYTreatment validity; usefulness of the scale and its assessment to

accomplish specific ECI purposes, especially planning and evaluating interventions

Curricular links: Functional competencies? Intervention content: What to teach? Intervention methods: How to teach? Progress monitor: Detects gains?

DEC ASSESSMENT STANDARDS

1. ACCEPTABILITY – Social worth & detection2. AUTHENTICITY – Natural methods & contexts3. COLLABORATION – Parent-professional

teamwork4. EVIDENCE– Disability design/evidence-base5. MULTI-FACTORS– Synthesis of ecological data6. SENSITIVITY – Fine content/measurement

gradations7. UNIVERSALITY– Equitable design/special

accommodations8. UTILITY – Usefulness for intervention

Overview of AA Measures

What Are Guidepoints for Sensible Assessment in Early Childhood?

Authentic Assessment “In Action”

Authentic Assessment in Action:

Choose an authentic curricular scale to unify interdisciplinary and interagency teamwork

Match team assessment model to the needs and preferences of each child and family

Share duties among parents, other caregivers, and interdisciplinary team members

Rely on parent observations & judgments Reframe your assessment role as “orchestrator”

of authentic assessment via others Spread team assessment activities over time

(e.g., 15-30 days) Sample skills only in natural or analog contexts

relying on several people;

Action Guidepoints

Use probes to sample expected skills in the proper context (e.g., math problems at a desk; social communication on the playground)

Ensure samples over several situations and occasions

Employ “friendly” materials that are jargon-free and a common language for all

Use several metrics to portray curricular performance (SS, RDQ, DA, IEI, CEI, HLM)

Use technology to collect ecological data for support and validation (e.g, computerized PDA; videotapes, audiotapes, pictures)

Guidepoints for the “Right Measure” of Children

Real situations give the best results.“Get Real.” Assessment requires teamwork.“Two Heads Are Better Than One.” Choose materials that link to intervention.“The Right Tool for the Right Job.” Assessment is decision-making.“Tests Don’t Make Decisions, People Do.”

The MisMeasure of Man (Stephen J. Gould, 1981)

“We pass through this world but once. Few tragedies can be more extensive than the stunting of life, few injustices deeper than the denial of an opportunity to strive or ever hope, by a limit imposed from without, but falsely identified as lying within” (p.28).

What Are Two Critical Elements In the Early Childhood Assessment

Process?

1. Context2. Teamwork

ASSESSMENT FOR EARLY INTERVENTION IS

NOT TEST-BASED

“Early childhood assessment is a flexible, collaborative decision-making process in which teams of parents and

professionals repeatedly revise their judgments and reach consensus about

the changing developmental, educational, medical, and behavioral healthcare needs

of young children and their families.” (Bagnato & Neisworth, 1991)

EARLY INTERVENTION TEAMWORK MODELS

MultidisciplinaryInterdisciplinaryTransdisciplinary

Assessment

Decision-making

Intervention

Multidisciplinary

SW ST PT OT PS T

SW

SW

ST

ST

PT

PT

OT

OT

PS

PS

T

T

Child

Assessment

Decision-making

Intervention

Interdisciplinary

SW ST PT OT PS T

SW

SW

ST

ST

PT

PT

OT

OT

PS

PS

T

T

Child

DEC Self-Appraisal Activity

Not everything that can be measured counts, and not

everything that counts can be measured (Einstein, 1951)

The MisMeasure of Young Children:

The Authentic Alternative for Assessment and Outcomes Research in Early Childhood

Intervention Programs

STEPHEN J. BAGNATO, Ed.D., NCSPProfessor of Pediatrics & PsychologyDirector, Early Childhood Partnerships

Children’s Hospital of PittsburghThe UCLID Center at the University of Pittsburgh

University of Pittsburgh School of Medicineemail: steve.bagnato@chp.edu

www.uclid.org