quality assurance: looking for quality data

Quality Assurance: Looking for Quality Data

1

I know it is in here somewhere

Presented by The Early Childhood Outcomes Center Revised January 2013

Activity

Early Childhood Outcomes Center 2

What factors work to improve the quality of your data?

What factors work to lessen the quality of your data?

How to address these factors?


Take Home Message

If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.

What do you if the data are not as good as they should be?

Answer: Continue to improve data collection through ongoing quality assurance

Many steps to ensuring quality data

BeforeGood data collection/TrainingGood data system and data entry procedures

During

Ongoing supervision of implementationFeedback to implementersRefresher training

AfterReview of COSF recordsData analyses for validity checks

4



During



5


Promoting quality data

• Training and support before and during data collection

• Analysis of the data after data collection• Data system and verification after data

collection

Promoting Quality Data


Through training and communication related to: • Assessment• Understanding the COS

process• Age expectations• Data entry



Through training materials, such as:• Video team and child examples• Written child examples• “Quizzes” for ensuring learning

Refresher trainings –Beware of Drift!!



Through data systems and verification, such as:

• Data system error checks• Good data entry procedures



During



10


Ongoing supervision

• Review of the process– Is the process high quality?– Are teams reaching the correct rating?

• Methods– Observation– Videos


Quality Review of COS Team Discussion

1. Do all team members participate in the discussion?

2. Is parent input considered in the rating? Give examples.

3. Is the team documenting the rating discussion? Give examples.

4. Does the team discuss multiple assessment sources? What are they?


Quality Review of COS Team Discussion

5. Does the team describe the child’s functioning, rather than just test scores? Give examples.

6. Does the discussion include the child’s full range of functioning, including skills and behaviors that are age appropriate, immediate foundational, and leading to immediate foundational? Give examples.


Quality review through process checks

• Provider surveys– Self assessment of competence– Knowledge checks– Process descriptions (who participates?)– Identification of barriers

• Kansas’ survey• Alaska’s survey


Questions from Alaska’s Survey

3. How would you rate your own level of proficiency with the COSF process? (please select only one)

I am confident I know how to do it, and I do it well

I know how to do it, but I need some more practice and assistance

I understand it to a point, but I need more training

I do not know how to do this yet

4. Some cases are different from others, but of the choices below, which process seems to be the most typical in your experience? (please select only one)

I gather information and determine COSF ratings on my own

I gather information and consult with another provider to determine COSF ratings

I gather information and consult with the family to determine COSF ratings

I gather information, discuss it with a team and the team determines the COSF ratings


Ongoing Supervision

• Feedback to teams is critical• Refresher training• Beware of:

– Auto pilot– Drift


Quality Review of COS Team Discussion: Activity

1.Observe team video2.Evaluate quality



During



18


Quality Review of Completed COS Forms

Is the COSF complete?

Is there adequate evidence for the basis for the rating?

Does the evidence match the appropriate outcome area?

Is the evidence based on functional behaviors?


Quality Review of Completed COS Forms

Is there evidence that the child’s functioning across settings and situations considered?

Are the ratings consistent with the evidence?


Quality Review of COS Forms: Activity

Review completed COS Form with errors


Promoting quality data through data analysis

• Examine the data for inconsistencies

• If/when you find something strange, look for other data that might help explain it.

• Is the variation caused by something other than bad data?


The validity of your data is questionable if…

The overall pattern in the data looks ‘strange’– Compared to what you expect– Compared to other data– Compared to similar states/regions/school

districts

COS Ratings - Fall

Rating Class 1 Class 2 Class 3 Class 41 3 1 1 22 4 1 2 23 5 2 3 34 6 3 2 45 1 4 5 46 1 5 5 47 0 4 2 1

24

SpringFall 1 2 3 4 5 6 7 total

1 1 4 2 72 1 1 5 6 9 3 1 263 2 15 14 27 19 6 834 4 4 21 39 28 12 1085 1 12 14 71 86 48 2326 1 3 21 48 63 1367 2 18 23 56 99Review

Total 2 13 38 60 185 207 186 691

Outcome 3: Appropriate Action

OSEP Categories

OSEP Categories Class 1 (%)

Class 2(%)

Class 3 (%)

e. Maintained Age Appro Trajectory 23 16 24

d. Changed Traj – Age Appro 15 23 13c. Changed Traj – Closer to Age

Appropriate 32 34 37

b. Same Trajectory -Progress 28 21 25a. Flat Trajectory – No Prog. 2 6 1

26


Questions to ask

• Do the data make sense?– Am I surprised? Do I believe the data?

Believe some of the data? All of the data?

• If the data are reasonable (or when they become reasonable), what might they tell us?


Validity

• Validity refers to the use of the information• Does evidence and theory support the

interpretation of the data for the proposed use?Or

• Are you justified in reaching the inference you are reaching based on the data?

Standards for Educational and Psychological Testing (1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education


The validity of your data is questionable if:

?


• ….not all providers are not knowledgeable about in the COS process

• …not all providers are careful with the COS process

• …the data look “strange”• …etc.

The validity of your data is questionable if:


31


During



quality assurance: looking for quality data

Documents

data collectionanalysis

data collectiondata

quality datatraining

process high quality

video team

team members

rating discussion

correct rating