leading indicators: evaluation for site-level improvement …...leading indicators: evaluation for...

Leading Indicators: Evaluation for Site-Level Improvement and System-Level Planning Samantha Sugar Research Associate/Analyst David P. Weikart Center for Youth Program Quality Charles Smith Vice President for Research Forum for Youth Investment Executive Director David P. Weikart Center for Youth Program Quality April 18, 2012

#readyby21

Agenda •  Welcome •  Opening Activity •  Important Questions

–  The Why –  The What –  The How

•  Site-Level Improvement •  System-Level Planning

–  The Where •  Looking Forward/Next Steps •  Questions

Objectives

•  Learn how collecting program data from staff, youth and parents help to tell the overall story about Quality for a single site.

•  See how this data can be used to identify low capacity programs and support resources targeting decisions.

Opening Activity Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:

a.Timely–Dataisavailableinreal%measeventsoccurorjusta?ercomple%on

b.Objec%ve–Dataisfocusedonbehaviorsandcondi%onsthatcanbeiden%fiedthroughobserva%onandeasilynamedinrela%ontoprac%ce

c.Reliable–Dataisseenaspreciseandfactualbyallduetostandardiza%onofmeasures/methods

d.Sensi%ve–Datadescribesbehaviorsandcondi%onsthatarelikelytochangeinresponsetointerven%onandchangecanbecapturedonthemeasures

e.Valid–Datadescribesbehaviorsandcondi%onsthoughttobealinkinacausalchainofeventsdesiredbytheactorsinvolved

f.Feasible‐Theminimumdatanecessaryarecollectedusingtypicalcommunityresources

g.Mul%‐Purpose–Astheyoccur,BOTHdatacollec%onanddatainterpreta%onprocessespromotelearningandcoordina%onamongactorsintheorganiza%on

h.Mul%‐level–Datadesignedforusebyindividualunits(staff/sites)canbeaggregatedacrossindividualunitstoassesscollec%veperformance

Op%malCharacteris%csofPerformanceData

Quality of Instruction (Point of Service Setting)

YouthVoiceandProgramGovernanceStructures

ENGAGEMENT

INTERACTION

SUPPORTIVEENVIRONMENT

SAFEENVIRONMENT

Higherorderengagementthroughchoice,planning,andreflec4on.

Peerinterac4onthroughgroupingandcoopera4velearning.

Suppor4veenvironmentthroughwelcoming,conflictresolu4on,ac4velearning,andskillbuilding.

Physicalandemo4onalsafetyisprovided.

Why Were the Leading Indicators Developed?

Continuous Improvement Practices by Site Teams (Organizational Setting)

StandardizedAssessmentofInstruc4on

Team‐basedPlanningwithData

CoachingandPerformanceFeedback

TrainingforInstruc4onalSkills


Objec4veData

MeaningfulInforma4on

Ac4on/Exper4se

ImprovedOutcomes

LowerStakesAccountabili4es

Interpre4veCommunity

• TeamSelfAssessment• Reviewexternalscores

TeamPlanningandImplemen4ng

• Improvementplanning• Performancecoaching

HigherStakesAccountabili4es



PointofServiceSeOng

Organiza4onalSeOng

PolicySeOng

Con%nuousImprovementPrac%cesforSiteTeams

QualityInstruc%on&ProximalChildOutcomes

LowStakesAccountabilityandSupports

TheoryofChange:Mul%pleLevelsofSe]ng

Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:











LeadingIndicator5.1–

FamilySa4sfac4on

LeadingIndicator4.2–AcademicEfficacy

LeadingIndicator4.1–Socioemo4onalDevelopment


EngagingInstruc4on

LeadingIndicator1.1–StaffingModel

LeadingIndicator3.4–CommunityResources


SchoolAlignment


FamilyEngagement

LeadingIndicator2.1–AcademicPress


YouthGovernance

LeadingIndicator3.1–SystemNorms

LeadingIndicator1.4–Enrollment

Policy

LeadingIndicator1.2–Con4nuousImprovement

PointofServiceSeOng

Organiza4onalSeOng

PolicySeOng

Con%nuousImprovementPrac%cesforSiteTeams

QualityInstruc%on&ProximalChild

Outcomes

LowStakesAccountabilityandSupports



•  Isn’t this just more data?

13 composite measures categorized into five different contexts:

- Organizational Context - Instructional Context - External Relationships - Youth Characteristics - Family Satisfaction

What Are the Leading Indicators? WhataretheLeadingIndicators?

Wheredidtheycomefrom?

- Grantee Director/Site Coordinator Surveys - Afterschool teacher/Youth Worker Surveys

- Youth Surveys (grades 4-12)

- Parent Surveys

- PPICS data

Howdowemeasurethem?

- Youth Program Quality Intervention (YPQI) - California Outcomes Measures (Vandell)

- PPICS data

•  Sample Report What Are the Leading Indicators?

What Are the Leading Indicators?

A

C

B

• Itemsaresimplythequestionsthatweaskonthesurveys,demographicandenrollmentdata,orYouthPQAscores.OnFigure1below,theItemscorrespondwithletterA. • ScalesaremadeupofgroupingsofdifferentItemsthatgotogetherwell.AScaleisdesignatedbyletterBinFigure1below. • LeadingIndicatorsaremadeupofgroupingsofdifferentScales,muchliketheScalesthemselvesaremadeupofItems.Intheexamplebelow,the“Accountability”Scale(alongwith“Collaboration”)makeuptheLeadingIndicator3.1–SystemNorms,whichisrepresentedbyletterCinFigure2below.[JB1] • Finally,alloftheLeadingIndicatorsaregroupedintoLiveoverarchingDomainsbasedonthecontextthattheyrepresent.TheseDomainsarecolor‐codedforeasydistinction,andinclude:OrganizationalContext(red),InstructionalContext(green),ExternalRelationships(blue),YouthCharacteristics(purple)andParentSatisfaction(brown).

Figure1

Figure2

Effec%veperformancedatadescribesbehaviorsandcondi%onsinawaythatis:










What Are the Leading Indicators?

How Have the Leading Indicators Been Used? OklahomaExemplar–SystemCharacteris4cs

•  History of the system •  Integration of QIS and required evaluation

efforts •  75 grantees in the first year, 77 this year •  Timeline •  Data Collection Methods •  Outputs

Site-Level Improvement

•  The report is… –  A tool to help you identify the

strengths of your program –  A tool to help you identify the

weaknesses of your program

•  The report is not… –  A mechanism to induce

evaluative comparisons or competitions across grants

– Something to be scared of


•  Get a feel for the layout of the report •  Study the graphs - In what areas are you

doing comparatively well? In what areas does it look like your site could improve?

•  Celebrate your strengths •  What could you work on? •  Do some thinking •  Prepare to make a plan!

Howtoreadandinterpretyourreport


Priority Assessment Form: Leading Indicators

1.  Create the story of your data (column one) –  What is the message or story of your data?

What do the numbers tell you? –  What’s missing from the data? What

important things about program quality do not come through?

–  Where are the gaps between what you want to provide and what the data says you’re providing?

2.  Brainstorm ideas for improvement (column two)


The profiles (clusters) in Figure A-8 may be interpreted as follows. •  Cluster 3: “High quality”

Thirty-three percent of grantees fall into Cluster 3, where programs show high quality in all areas.

•  Cluster 2: “High with low -- growth/mastery and family communication” This cluster represents 24% of grantees. These programs show relatively high quality in most areas, but low school alignment and parent communication.

•  Cluster 1: “Medium” Cluster 1 represents 21% of grantees. These programs have medium levels of supervision quality, high academic press & school alignment, low program quality (growth & mastery), and low family communication.

•  Cluster 4: “Low with high -- school alignment & family communication” Four percent of grantees fall into Cluster 4, where programs show low quality in supervision, growth & mastery, and academic press, but high school alignment & family communication.

•  Cluster 5: “Low quality” Eleven percent of grantees fall into Cluster 5, where programs appear to demonstrate low quality in all areas.

System-Level Planning

Intermediate & Academic Outcomes

•  Table A-8 provides means of the satisfaction variables across cluster groups. The omnibus ANOVA indicates that the clusters produce significantly different scores in the Intermediate Outcomes set (staff job satisfaction, parent satisfaction, and parent reports of the program supporting academics). The highest quality group of grantees/sites (Cluster 3) produces the highest staff satisfaction, youth engagement, and academic efficacy, whereas the lowest quality group of grantees/sites (Cluster 5) exhibits the lowest or nearly the lowest in each area.

Table A-8 Means Scores for Intermediate Outcomes by Level of Quality

System-Level Planning

Cluster Staffjobsa%sfac%on

(S) ParentSa%sfac%on

(P) YouthEngagement

(Y) Homework

comple%on(Y) Prog.Supportsacademics(P)

3:Highquality 4.5 4.7 4.0 4.4 4.2

2:Highwithlow 4.2 4.7 3.8 4.0 3.9

1:Mediumquality 4.2 4.6 3.8 4.3 3.8

4:Lowwithhigh 3.5 4.8 3.7 4.2 4.2

5:Lowquality 3.6 4.5 3.7 4.2 3.6 Omnibusdifferenceacrossclusters(ANOVAF)

8.3***

2.6*

1.4

2.3+

5.9***

Where Have the Leading Indicators Been Used?

Looking Forward/Next Steps

•  Further Validation Work – Exploration of Leading Indicators Framework –

Theoretical and Statistical

•  Expansion to New Networks •  Integration of Quality Improvement Systems

Questions?

Thank You!

leading indicators: evaluation for site-level improvement …...leading indicators: evaluation for...

Documents