bbvt_results based accountability_presentation
DESCRIPTION
Dr. Amy Carmola of the United Way of Chittenden County presents the basics of Results-Based Accountability™ to the Vermont Nonprofit Conference 2012: Benchmarks for a Better Vermont. Results-Based Accountability™ concepts utilized in this presentation are derived from the book "Trying Hard is Not Good Enough by Mark Friedman.”TRANSCRIPT
Results-Based Accountability Vermont Nonprofit ConferenceMarch 20, 2012
Goals for Today(aka How I hope you will better off)
Basic understanding of Results-Based Accountability (RBA)•Why bother?•What is it?•How it can help you plan and manage and maximize•How it can help you communicate•Where to get more information & resources
Experience working with Performance Measures
Be ready to take next steps Benchmarks for a Better VT March 2012 2
Sources & ResourcesResults-Based AccountabilityTM (RBA)
Trying Hard Is Not Good Enough: How to Produce Measurable Improvements for Customers and CommunitiesMark Friedman (2005)
Results and Performance Accountability, Decision-Making, and BudgetingFiscal Policy Studies InstituteSanta Fe, New Mexicowww.resultsaccountability.comwww.raguide.org
Benchmarks for a Better VT March 2012 3
Wait a Second…
Benchmarks for a Better VT March 2012 4
Results Accountability Performance Measures
Haven’t we heard about this before?
Everyone wants to:•Improve our programs & maximize impact•Leverage support for our programs
What gets in the way?
• “Too little time”• “Other priorities”• “Too complicated”• “Too many hard decisions”• “We don’t have expertise – or $$ to hire someone”• “They don’t get it”• “Doesn’t feel relevant”
• Overthinking…Trying too hardBenchmarks for a Better VT March 2012 5
Why Bother?Don’t I have enough to worry about already??
Benchmarks for a Better VT March 2012 6
Why Bother?
Benchmarks for a Better VT March 2012 7
What is Results Accountability?
Benchmarks for a Better VT March 2012 8
Shifting our Thinking
Results Accountability is NOT:•a product•a year-end evaluation •software•(just) a reporting tool
Results Accountability IS a :•behavior discipline •framework for planning, doing, and evaluating •process for moving from Talk to Action•cycle for continuous improvement•tool for communicating
Benchmarks for a Better VT March 2012 9
Shifting our Thinking
RBA is NOT:•fancy•new •a side track
RBA is :
•simple•the MAIN track
RBA is :•asking an answering common sense questions (that we already think about all the time)
Benchmarks for a Better VT March 2012 10
Common Sense Questions• What do we want?• How will we recognize it?• How do things look?• What will it take to get the results we want?• Who are the partners with a role to play?
• What is our role in helping achieve the result?• How good of a job are we doing?
Benchmarks for a Better VT March 2012 11
What’s the Big Deal? (1)Common Sense Questions•What do we want?•How will we recognize it?•How do things look now?
•What will it take to get the results we want?•Who are the partners with a role to play?•What is our role in helping achieve the result?•How good of a job are we doing?
Start with ENDS
Then talk about MEANS
Benchmarks for a Better VT March 2012 12
What’s the Big Deal? (2) Common Sense Questions•What do we want?•How will we recognize it?•How do things look now?
•What will it take to get the results we want?•Who are the partners with a role to play?
•What is our role in helping achieve the result?•How good of a job are we doing?
COMMUNITY FOCUS
Can look at different levels……with appropriate accountability (more on that later)
PROGRAM FOCUS
Benchmarks for a Better VT March 2012 13
What’s the Big Deal? (2 continued)Common Sense Questions•What do we want?•How will we recognize it?•How do things look now?
•What will it take to get the results we want?•Who are the partners with a role to play?
•What is our role in helping achieve the result?•How good of a job are we doing?
For each “thing” we do:
•Who do we serve?
•What do we do?
•How good of a job are we doing?
•What works to improve/strengthen?
•What do we propose to do?
Benchmarks for a Better VT March 2012 14
What’s the Big Deal? (3) Common Sense Questions•What do we want?•How will we recognize it?•How do things look now?
•What will it take to get the results we want?•Who are the partners with a role to play?
•What is our role in helping achieve the result?•How good of a job are we doing?
Common Sense
Clear Language
No Jargon
Tells a Story
Benchmarks for a Better VT March 2012 15
A few words about LANGUAGE
• Outcome, Result, Benchmark, Goal, Target
• It doesn’t matter what you call it, as long as people know what you mean.
You can call me anything you like, but just don’t call me late for dinner (or do I mean supper?).
Benchmarks for a Better VT March 2012 16
Agreeing on Clear Language• What do we want?
• How will we recognize it?• How do things look now?
• What will it take?• Who are the partners?• What is our role?• How good of a job are we doing?
RESULTA condition of well-being for children, adults, families or communities as a whole
INDICATORHow we measure this condition
Benchmarks for a Better VT March 2012 17
EXAMPLES RESULTS for All Peopleof the State, County, City or Neighborhood
A Prosperous Economy
A Clean Environment
Healthy and Safe Communities
Self-Sufficient Families
Children Ready for and Succeeding in School
Parents and Other Adults Healthy and Self-Sufficient
Elders Living with Dignity in Setting of Their Own ChoiceBenchmarks for a Better VT March 2012 18
Clear Language, continued• What do we want?• How will we recognize it?• How do things look now?• What will it take?• Who are the partners?
• What is our role? What will we do to help achieve the result?
• How good of a job are we doing?
STRATEGYAn action taken to improve the RESULT
PERFORMANCE MEASURESHow we know if strategies are workingBenchmarks for a Better VT March 2012 19
Benchmarks for a Better VT March 2012
Strategy ExamplesWhat we do to help achieve the desired result:
•Program/Service Delivery•Collaboration•System Change•Service Integration•Funding•Resource Development/Leveraging
20
Different Levels of Focus
RESULTS & INDICATORSare about the well being of populations
“All people are happy and healthy”
Benchmarks for a Better VT March 2012 21
Different Levels of Focus
STRATEGIES & PERFORMANCE MEASURESare about the well being of program participants / service recipients
“Vaccinations for school children”
Benchmarks for a Better VT March 2012 22
• Safe Community
• Crime Rate
• Average Police Department response time
• A community without graffiti
• % of Montpelier buildings without graffiti
• People have living wage jobs and income
• % of people in Windsor County with living wage jobs and income
• % of participants in job training who get living wage jobs
A quick test: RESULT, INDICATOR, or PERFORMANCE MEASURE?RESULT
INDICATOR
PERF. MEASURE
RESULT
INDICATOR
RESULT
INDICATOR
PERF. MEASURE
Different Levels of Focus = Different Levels of Accountability
1. Population Level- Focus = well being of population- Collective accountability: Many partners are responsible for achieving the desired result
2. Program (Strategy) Level- Focus = well being of program participants/service recipients- Program accountability: Program is responsible for its performance
Benchmarks for a Better VT March 2012 24
Who should be accountable?
• Children in Brattleboro are ready for school• Children in grade 3 at St. Albans City School
learn to read• Job training participants gaining new skills• Employment rate in Bennington County• Employment in Burlington’s New North End
neighborhoods
Benchmarks for a Better VT March 2012 25
What’s Appropriate Accountability?
WhoMany Partners
Program
WhatAll children are ready for school
Pre literacy skills in Head Start classroom
Benchmarks for a Better VT March 2012 26
NO!!! Not appropriate!
Makes sense
Makes sense
Bringing Two Levels Together
See Handout “Results Accountability: The WHOLE PICTURE”
Benchmarks for a Better VT March 2012 27
Benchmarks for a Better VT March 2012 28
Benchmarks for a Better VT March 2012 29
Benchmarks for a Better VT March 2012 30
The Accountability Boundary
_____________________________________________** ACCOUNTABILITY BOUNDARY **
_____________________________________________
•Why is this so important?•OK to cross –but have to be AWARE
Benchmarks for a Better VT March 2012 31
How does RBA fit with:
• Strategic planning• Outcomes evaluation• Logic Model• SSML• SWOT• PDCA
Benchmarks for a Better VT March 2012 32
Break for LunchBurning questions?
Share them with the facilitators and fax to Amy
See you again at 12:45 sharp
After lunch we will focus in on:Program Accountability & Performance Measurement
Benchmarks for a Better VT March 2012 33
REMINDER Different Levels of Focus = Different Levels of Accountability1. Population Level
- Focus = well being of population- Collective accountability: Many partners are responsible for achieving the desired result
2. Program (Strategy) Level- Focus = well being of program participants/service recipients- Program accountability: Program is responsible for its performance
Benchmarks for a Better VT March 2012 34
REMINDER What’s the Big Deal?
Common Sense Questions•What do we want?•How will we recognize it?•How do things look now?
•What will it take to get the results we want?•Who are the partners with a role to play?
•What is our role in helping achieve the result?•How good of a job are we doing?
For each “thing” we do:
•Who do we serve?
•What do we do?
•How good of a job are we doing?
•What works to improve/strengthen?
•What do we propose to do?
Benchmarks for a Better VT March 2012 35
Program AccountabilityPINK Handout
• Who do we serve?• What do we do?• How good of a job are we doing?
– How much do we do?– How well do we do it?– Is anyone better off? (Who? How so?)
• What works to improve/strengthen?• What do we propose to do?
Benchmarks for a Better VT March 2012 36
Program Accountability
• Who do we serve?• What do we do?• How good of a job are we doing?
1. How much do we do?2. How well do we do it?3. Is anyone better off? (Who? How so?)
• What works to improve/strengthen?• What do we propose to do?
Benchmarks for a Better VT March 2012 37
Performance Measures
Program Performance MeasuresCOMMON EXAMPLES
How much?•# People served•# Hours service offered•# Activities (by type)
How well?•Participant satisfaction•Use of best practice•Qualifications/training•External review
Anyone better off?•Change in skills, knowledge, behavior, circumstance, well being•Possible sources: program records, participant survey, external data
Benchmarks for a Better VT March 2012 38
Program Performance MeasuresEXAMPLE
High SchoolHow much?•# students
How well?•Student teacher ratio
Anyone better off?•NECAP scores•Graduation rates•% students entering higher education or employment after graduation
Benchmarks for a Better VT March 2012 39
Program Performance MeasuresEXAMPLE
Welfare to Work
How much?•# people trained
How well?•% child care and transportation needs met
Anyone better off?•% employed 6 months after program completion
Benchmarks for a Better VT March 2012 40
Program Performance MeasuresEXAMPLE
Raising Children
How much?•# children, # hours spent
How well?•Parent training/education; use of best practice (child satisfaction??)
Anyone better off?•Child health•Child learning achievement•Child happiness & well being
Benchmarks for a Better VT March 2012 41
Performance Measurement Challenges
How do we know if anyone is better off?
Required measuresLack of data Excess dataInappropriate dataInappropriate expectations
Benchmarks for a Better VT March 2012 42
Choosing Performance Measures (1)
• Answer all 3 questions• Choose what YOU want to know
– What do we need to know to plan? – Evaluate our work? – Make it better?
• Don’t think (too much) about others • This is about the program
Benchmarks for a Better VT March 2012 43
Choosing Performance Measures (2)
• Whole staff activity – Central part of planning, service delivery,
budgeting, communications, reporting
• Map out what you will measure & why• Look at what you already have• TRACK OVER TIME• TRY something. Did that work/help? Try
something else.
Benchmarks for a Better VT March 2012 44
GROUP ACTIVITYImaginary Program: Cavity Free RichmondWho do we serve?Residents of Richmond, all ages, who do not have access to affordable dental care (e.g., underinsured, uninsured)
What do we do?•Free dental care in clinic (preventive care & treatment)•Presentations in schools about good dental care habits•Outreach to build awareness of services
Benchmarks for a Better VT March 2012 45
GROUP ACTIVITYImaginary Program: Cavity Free RichmondYour Task: Identify possible Performance Measures
How could this program measure:1.How much it does?2.How well it does it?3.If anyone is better off?
15 minutes for group workReport out from Newport, Rutland, St. Albans
Benchmarks for a Better VT March 2012 46
More Examples
Benchmarks for a Better VT March 2012 47
Benchmarks for a Better VT March 2012
Example: Early Care & Education
1. How much ?# children served; # receiving financial assistance
2. How well?STARS ratingStaff training & qualifications, best practices
3. Is anyone better off?Old measure: developmental statusNew measure: % needs addressed
48
Benchmarks for a Better VT March 2012
Example: Youth Development
1. How much ?# and characteristics of youth served
2. How well?Licensing, external review, quality ratingStaff training & qualifications, best practices, standards
3. Is anyone better off?Youth feel safe at the program.Youth feel connected to caring adults/role models.Youth feel connected to their community.
49
Benchmarks for a Better VT March 2012
Example: Residential Treatment
Is anyone better off?How to define this?
Complexity SimplicityWhat do we really need to know?
50
WRAPPING UP PROGRAM ACCOUNTABILITY
Benchmarks for a Better VT March 2012 51
POPULATION ACCOUNTABILITYYELLOW Handout
A few key points:•Indicators = data•Don’t get lost in the data. Focus on RESULT.•How to pick good indicators? There’s a chapter for that. Let go of perfection.•When is Population Accountability useful?
– Clarifying accountability– Showing relevance– Bringing together partners.
Benchmarks for a Better VT March 201252
GROUP ACTIVITYPopulation Accountability ExerciseImaginary Problem: •Community upset about fires in Winooski•Fire Department budget questioned•Plan for community meeting
Benchmarks for a Better VT March 2012 53
GROUP ACTIVITYPopulation Accountability Exercise
What we want? RESULTAll people in Winooski are safe from fire
Your task today:Who are the partners with a role to play?10 minutesReport out from Montpelier & Bennington (if time)
Benchmarks for a Better VT March 2012 54
Population Accountability: Examples
Benchmarks for a Better VT March 2012 55
Population Accountability : Burlington Truancy Project
Started with a Problem: Burlington had highest drop out rate in VT. And it was going up.
What will it take to turn this around?Who needs to be at the table?
Benchmarks for a Better VT March 2012 56
57
Burlington Truancy Project Shared accountability & partnership• Desired Result All Burlington students graduate from
high school • Indicator High school drop out rate• Baseline Not good! Highest dropout rate in VT
Dropout Rate for Burlington School District Students
0.0%1.0%2.0%3.0%4.0%5.0%6.0%7.0%8.0%9.0%
10.0%
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
Benchmarks for a Better VT March 2012
Benchmarks for a Better VT March 2012 58
Burlington Truancy project (cont.)
• What works? Address truancy to prevent drop out• Partners Non-profit service providers, school district,
police department, school administrators, juvenile court judges, attorneys, child
protection agencies, and United Way of Chittenden County
• Strategies A coordinated response to truancy:
- School District: new policies & procedures
- Juvenile Court & SA : monthly court time
- Non profits & state agencies: services that remove barriers
Benchmarks for a Better VT March 2012 59
The Truancy Project: Key Indicator
Dropout Rate for Burlington School District Students
0.0%1.0%2.0%3.0%4.0%5.0%6.0%7.0%8.0%9.0%
10.0%
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
Example: Truancy Task Force & the EVEN BIGGER PictureWhat do we want?
Youth succeed in schoolHow will we recognize it?
School completion; CompetenciesWhat does it take?
1. Good educational opportunities2. Youth attend school3. Youth ready to learn
Benchmarks for a Better VT March 2012 60
Benchmarks for a Better VT March 2012
Example: Investing in Early Childhood (How UWCC fits in bigger picture)
ResultAll children enter school ready to learn
IndicatorKindergarten readiness
UWCC Strategies (the roles we play)1. Fund programs that provide parent support & education2. Provide access to high quality early education3. Volunteer programs supporting early literacy
RESULT
STRATEGY1
STRATEGY 2
OTHER INFLUENCES
61
RBA = Process
What about Product?
Benchmarks for a Better VT March 2012 62
64
Reminder: Goals for Today aka How I hope you will better off
Brief Intro Results-Based Accountability (RBA)•Why bother?•What is it?•How it can help you plan•How can it help you manage…and maximize…and communication•Where to get more information & resources
65
Did we cover these?
If nothing else…
Key Features of RBA
3 Performance Measure questions2 Levels of focus (& accountability)1 Simple framework to help focus
Benchmarks for a Better VT March 2012 66
Remember:• Anyone can do this, at any time. Period.
• This is a whole staff/whole organization process. All the time.
• Don’t worry about perfection. Sketch it (don’t etch it).
• Your program is not responsible for population level results.
• You have a role to play in population level results. Know what that is.
• Your program is responsible for helping its participants be better off. You have to know if and how.
• Keep RBA questions in your pocket, bring them with you. Say, “How much? How well? Anyone better off?” Say, “Who is responsible for that? For this?”
67
Feeling ready to give it a try?
• Buy the book – “There’s a chapter for that!”
• Check out the website
• Apply for the Performance Institute
• Start asking and answering the questions – Staff meeting– Board meeting– Coalition meeting
• JUST DO IT. Benchmarks for a Better VT March 2012 68