The Shared Data Experience How We Learn to Love Our Data!

Akron Ready Steps Assessment Presentation

AKRON, OHIP30 November 2009

Karen Burstein, PhD, MPHProject Evaluator

SWI 2

ALL SUCCESSFUL PROJECTS BEGIN AND END WITH THOUGHTFUL, ACCURATE ASSESSMENT.

3

Data Drives Our Lives

SWI

SWI 4

Teachers as Scientists

SWI 5

Rate Yourself- What data (or not) prompts you to choose a curriculum?

SWI 6

Evaluation as a Key Component

SWI 7

Let Your Application be Your Guide

SWI 8

Caveat 1

SWI 9

Caveat 2

SWI 10

SWI 11

So, Why Assess Children?

SWI 12

SWI 13

So, Why Assess Teachers?

SWI 14

SWI 15

Assessment drives the planning process;

which drives our teaching & instructional activities;

which are assessed for efficacy based on children’s performance.

AND THE CYCLE CONTINUES

SWI 16

Implications for Assessment

SWI 17

SWI 18

Reliability

SWI 19

Validity

SWI 20

Bias

SWI 21

Planning Annual ERF Assessments-

SWI 22

Set Up a System

SWI 23

Setup the Assessment Setting

A table for each assessment/ examiner

A place for children in between assessments

Limit visual and auditory distraction

Make sure teachers have comfortable chairs

Make sure that children’s feet are flat on the floor

SWI 25

Prepare a Dataset

SWI 26

Moving Children Through Assessments

Use the dataset to create name badges

A method by which to track the assessments given

A daily check sheet of assessments given

PROJECT RECOVERY DAILT ASSESSMENR LOG

NAME (pre-printed from dataset)

PPVT EOWPVT GRTR PALS TOPEL CAP

Amanda Smith KB/12-15-08 Stella Gales KB/12-15-08 Alisa Burroughs KB/12-15-08 Mary Smith KB/12-15-08

SWI 27

ARS Assessments

SWI 28

Kindergarten Screener

SWI 29

Kindergarten Screening Tool

• A screening tool is a short assessment of the presence of behaviors that is a reliable and valid indicator of a skill. A screening tool should be used to decide whether or not to take specific actions such as seeking referrals, further testing, and instructional activities. A test is a more rigorous and complex assessment of knowledge and skill with more extensive reliability and validity properties.

SWI 30

Why Screen?• Research shows that learning to read is a process that begins long

before children enter kindergarten. During the pre-kindergarten years, children develop the early literacy skills that help them to learn how to read during the first few years of elementary school. Early screening and intervention are the keys to overcoming reading difficulties and avoiding the problems that go along with them. Early literacy screening: Helps parents and early childhood professionals understand each child's progress towards developing literacy concepts and acquiring skills.

• Helps to identify developmentally appropriate experiences and teaching that can be used to support early literacy learning.

• Alerts early childhood professionals and parents to seek additional advice if a child does not appear to be making appropriate progress.

SWI 31

GRTR Screener

• Use this 20-question research-based screening tool with your 4-year-olds. The score will show if a child's pre-reading skills are weak, strong, or somewhere in between. And activities and resources to improve those skills will be provided.

• This tool is designed to screen a child twice during the year before kindergarten. Use the tool first in the fall one year before the child enters kindergarten, and again the next fall before kindergarten begins, to measure the child's progress. Don't use the tool more than three times in a year. It's not designed to measure small changes, and children develop new skills gradually.

SWI 33

SWI 34

Peabody Picture Vocabulary Test-IV

Measure of Receptive Language

Peabody Picture Vocabulary Test, Fourth Edition

By Lloyd M. Dunn, PhD and Douglas M. Dunn, PhD

SWI 36

SWI 38

TRAINING ITEMS Form B ages 4+

SWI 41

Assessment of Language and Literacy

Measure of Expressive Language, Phonological Awareness, Alphabet

Knowledge, Concepts of Print

SWI 42

SWI 43

• The Assessment of Literacy and Language (ALL™) aids in early detection of language disorders that could lead to reading difficulties. ALL assesses spoken language and written language skills, including:

• Listening comprehension• Language comprehension• Semantics• Syntax• Phonological awareness• Alphabetic principles/phonics• Concepts about print

With ALL, you can identify language disorders, language and emergent literacy deficits, emergent literacy deficits, and weak language and emergent literacy. View case studies to see how ALL can impact a child you know.

SWI 44

• Details the nature and severity of the problem• Aids in making recommendations for intervention• Provides information about where a child should be in his

or her reading and language skills based on grade level. Strategies for improving a child’s language skills in each of the areas assessed are included in the Examiner’s manual.The Parent Questionnaire provides additional information about a child’s language and literacy history and current skills.

• Norm-referenced scores• Criterion-referenced scores

SWI 46

GET It, Got It, Go!

Measure of Progress Monitoring

SWI 47

GGG (IGDI)

• Three subtests– Rapid Object Naming (1 minute)– Rhyming (2 minutes)– Alliteration (2 minutes)

SWI 48

Standardized Administration DOES Matter

• (These assessments are formally called Individual Growth and Development Indicators, or IGDIs.)

• Standardization of administration procedures, which includes: timing, appropriate prompting, administering sample items—is critical so that you get accurate results.

• By standardizing, we mean that IGDIs are administered in exactly the same way, for each child, across different assessment periods. This assures that IGDIs are administered accurately and reliably.

• Standardized administrations will minimize any systematic or unnecessary error. By standardizing, we are increasing the accuracy or reliability of our scores from one administrator to the next, from one child to the next, and for an individual child's scores from time a to b to c to d.

SWI 49

Each IGDI comes with a set of administration instructions.

• On the administration cards, bold print indicates exactly what the examiner should say.

SWI 50

Always Begin with Sample Items

• If the child meets the criteria indicated in the sample items, move to administration. Discontinue criteria are provided for the picture naming, rhyming, and alliteration measures. These criteria provide an indication of whether a child a). understands the task at hand and b). has sufficient skills for continuing with the administration.

• For example, the discontinue criteria for picture naming is as follows: The child is administered four sample cards (baby, bear, apple, cat). The administrator may stop administration if the child does not name all four of these pictures correctly.

SWI 51

Timing

• Each IGDI should be administered using a stopwatch according to the exact time specified.

SWI 52

Prompting

• Be Careful Not To Provide Corrected Feedback During Administration

• During IGDI "Administration" (not referring to sample card administrations) it is important not to prompt or provide feedback for correct or incorrect responses. Reinforcing statements such as "great," "good job," etc. are encouraged. However, feedback should not correspond to correct or incorrect responses made during administration (e.g., "that was right, " "that was wrong, etc)."

SWI 53

What Happens When Administration Is Not Standardized

"Melissa's" teacher administered the rhyming IGDI to Melissa on Oct. 14th and got a score of 7. She then administered rhyming again on Nov. 14th and got a score of 14. It appears that Melissa grew considerably in rhyming skill during this month. However, Melissa's teacher indicated that she didn't use a stopwatch. Instead, she used a clock with no second-hand during both administrations. She simply estimated 2-minutes time.

Here, we have a situation where we can not be sure whether Melissa actually grew in rhyming skill, since it may be that there was simply more time allotted to her during her second administration. Furthermore, lets say Melissa's teacher administered rhyming to all her students without timing exactly 2 minutes. In this case, we can not make accurate comparisons across students since maybe some student may have done better than others, simply because they were allotted more time.

SWI 54

GGG (IGDI)

• Picture Naming (Ages 3-5)• The format of this indicator presents a child with images of

objects commonly found in preschoolers' natural environments (i.e., home, classroom, community), one at a time, asking a child to name the pictures as fast as possible. Categories of objects used in this format include animals, food, people, household objects, games and sports materials, vehicles, tools, and clothing. Each photograph and line drawing is printed on an 8" x 5" index card. After providing a set of sample items, the examiner asks the child to look at each card and name it as quickly as possible. After exactly one minute, the examiner stops the activity and counts the total number of pictures named correctly.

SWI 55

Alliteration (Ages 3-5)

• We identified a set of words commonly known by preschoolers and obtained images of these words. Stimulus cards include one image at the top of each card (e.g., rain) and a set of three images in a row at the bottom of each card (e.g., house, rake, pig), one of which starts with the same sound as the target picture. After providing a set of sample items, the examiner asks the child to look at each card and point to one of the three pictures at the bottom of the card with the same initial sound as the fourth, target picture. The task continues for a total of two minutes. The score generated from this format is the number of pictures the child correctly identifies within two minutes.

SWI 56

Rhyming (Ages 3-5)

Stimulus cards include a target photo or line drawing at the top of each card (e.g., bees) and a set of three photos/drawings in a row at the bottom of each card (e.g., pants, gate, cheese), one of which rhymes with the target picture. After providing a set of sample items, the examiner asks the child to look at each card and point to one of the three pictures at the bottom of the card that sounds the same as (or rhymes with) the fourth, target picture. The task continues for a total of two minutes. The score generated from this format is the number of pictures the child correctly identifies within two minutes.

SWI 57

Useful Data

SWI 58

Bar Graph of Performance by Class

Class87654321

100

80

60

40

20

0

82.479.6

76

96.8

87

80.5

89.3

78.777.2

91.995

8584

RETURNING TO PRESCHOOL

GOING TO KINDERGARTEN

KINDER BOUND

Year 1 Baseline PPVT Standard Scores by Teacher by Kinder Status

SWI 59

Line Graph of Class Performance

Bailey

Blakely

Burton

CzattDarr

in Fike

Harris

Hughley

Jackso

n

Kitchen

Lurry

Mantin

McCray

Medina

Mettler

Nicol

Patters

on

Prosp

ero Ruiz

Seag

ers

24 2427

2017

2724

14

36

23

30

22

12

30

2025

22

31

22 20

26

15

24

1824

26 28

23

33

3632

23

19

30

26

29 34

32

29

21

GGG Picture NamingGGGPNCOR1 GGGPNCOR2

SWI 60

Sample Weekly Data

A Sample Graph with Aim and Performance Lines

0102030405060708090

100

w1 w2 w3 w4 w5 w6 w7 w8 w9

Week

Per

cen

tag

e o

f C

orr

ect

Res

po

nse

s Exceptionally Well

Typical (Aim Line)

At-Risk

Special Ed

SWI 61

Class 1 Class 2 Class 3Site

0.00

5.00

10.00

15.00

20.00GRTR Pretest

GRTR 2nd Test

. Get It Got It Go (N = 46, participating in both pretest and 2nd test)

SWI 62

Teacher ReportACE3 Fall 2006 Standardized Assessments

Agency: Teacher: School: Head Start

Last Name First Name PPVT PALS Letters Identified IPT

GGGPicture

Naming (#/95) GGG

Allit (#/45)GGG

Rhyme (#/52)Rapid Object

NamingGRTR

#/20

  Dray  40  6 A,B,D,L,N,Y  NE 4  0  0  6  0

  Corine 74 0   LE 20 0 0 12 10

  Alyssa  69  5 A,C,D,S,Y  NE 10  0  0  3  0

  Ulysses  58  2 L,U  NE 8  0  0  3  0

  Joey 86 6 A,B,D,E,G,Y  LE 23 0 0 28 6

  Brandon  69  2 A, B  NE 6  0  0  7  0

  Timothy 70 4 A,B,S,T  LE 24 0 0 0 4

  Anthony  72  5 A,C,G,E,M  NE 8  1  0  8  0

  Miguel 73 5 A,B,D,M,S  LE 14 0 0 4 6

  Kristofferson 75 1 X LE 21 0 0 12 10

  Dale 77 9 A,B,D,G,I,L,M,O.R  FE 13 0 0 38 13

  Déjà 82 4 A,D,E,J  LE 23 0 0 44 11

  Jaden  71 5  A,R,D,N,T  LE 15  2  0  6  0

  Kai 80 2 B,G LE 21 0 0 31 7

  Satori 74 2 X,S FE 29 0 0 42 11

  Keanu 77 21 A,B,D,O,E,X,S,P,G,L,K,R,T,F,Z,U,H,M,Y,V,J, LE 29 0 1 38 15

  Angel 81 12 A,B,D,O,E,X,S,C,G,R,U,H, LE 25 0 2 31 17

  Isabelle  73  6   NE 7  0  0  6  0

  Nina  65     NE8 9  0  0  8  0

  Tyrus 89 7 B,X,N,T,H,J,Q FE 0 0 0 20 11

PPVT:PPVT Standardized Score

HSLN:Head Start Lower Case Letter Naming

IPT:Individual Proficiency Test of English Language(NE=Non-English Speaking; LE=Limited English Speaking; FE=Fluent English Speaking)

GGG: Get it Got it Go

GRTR:Get Ready to Read

SWI 63

Plan, Plan, Plan

SWI 64

Review, Review, Review

SWI 65

Celebrate Your Success (based on your data)

For more information please contact:• Karen Burstein, PhD, Evaluator 480-222-8800 x 1

k.burstein@swifamilies.org