creating a culture of outcomes management (omgmt): showing what works national social services and...
Post on 13-Jan-2016
216 Views
Preview:
TRANSCRIPT
Creating a Culture of Outcomes Management (OMGmt): Showing What Works
National Social Services and Disaster Management Conference
March 26, 2014
2
Presenters• Major Lewis R. Reckline, Area Commander,
National Capital Area Command in Washington, DC
• Leslye E. Wooley, Area Command Director of Program Services, Washington, DC
3
Overview of theNational Capital Area Command
• Area Command was created in 2006• Encompasses the Army’s programs in the
following areas:– Two suburban counties in Maryland –
Montgomery and Prince George’s County– Washington, DC– Northern Virginia – City of Alexandria, Arlington,
Fairfax, and Prince William Counties
4
Creation of the Area CommandThe Strengths
• Increased visibility of The Salvation Army across the region
• Increased opportunities to expand programs, provide additional services across the entire region
• Increased the viability of our fundraising• Increased the influence of a regional Advisory
Board
5
Creation of the Area CommandThe Challenges
• Increased scrutiny of the effectiveness of our services
• Increased need for funding, especially new funding streams
• Increased emphasis on reporting our program outcomes
6
Have you or your staff ever looked like this at the mention of
outcomes?
7
Tell the truth…
Have you or your staff ever…1.) broken out into a cold sweat when a grant application asked for outcome information?
2.) laughed manically when a donor asked, “what is your success rate?”
8
Tell the truth…continued
3.) looked blankly at your Development Department colleagues when they ask you for a logic model for your program?
4.) cursed an Advisory Board member who asked “tell us about a ‘typical’ client?”
9
Tell the truth, continued
• If you answered yes, to any of these questions, you are not alone.
• This presentation is the Area Command’s story of how it came to see data as a friend, not an enemy.
10
Recognizing the Importance of Outcome Measurement
“Why did we need to create a culture of data collection and
outcome measurement?”AND
“Who cares about outcomes anyway?!”
11
Why did the Area Command choose to focus on measuring outcomes?
1. Increased accountability/stewardship – are our programs the best that they can be?
a. Are we Doing the Most Good for clients? With donor dollars?
b. To answer, “what is your success rate?”
2. What gets measured, gets improved.• Data backs up our anecdotal evidence.• The data validates the work that we do.• The data shows us our areas of weakness.
12
Why did the Area Command choose to focus on measuring outcomes?
3. Change in direction/emphasis by funders (example: homeless services).
4. To make informed policy decisions and program changes.
a. If the program is not working, what changes need to be made? Can it be fixed or… b. Does the program need to end?
13
Why did the Area Command choose to focus on measuring outcomes?
5. To inform stakeholders of our successes.a. Donors – leads to increased donations;b. Funders – leads to increased funding streams;c. Staff – what they are doing is having a positive impact on program participants;d. Program Participants – increased buy in because “this a good program that can really help me.”
14
Who cares about outcomes anyway?
1. Funders – nearly every grant requires information about outcomes from the programs that are applying. Then funders award funding to programs with proven results, “more bang for the buck.”
2. Donors – more major and smaller donors also want to know the outcomes of our programs. They want to know The Salvation Army really is “doing the most good” with their donation.
15
Who cares about outcomes anyway? Continued
3. Policy makers – want to know which interventions work best in order to replicate successful programs, end unsuccessful programs, and (hopefully) increase the funding for those with the strongest outcomes.
4. Stakeholders – including program participants themselves, Advisory Board members, etc.
5. Ourselves/Staff – does our program do what we want/need it to do? Did our intervention produce change? (And, if yes, was it the change we intended?)
16
Outcomes Measurement 101
• Pre-Test – what was our existing culture around data collection
and outcomes measurement at the outset?• Intervention – Creating a dedicated staff position for outcomes
measurement and outcomes management.• Post-Test – reviewing the outcomes, disseminating the outcomes,
and making program changes to improve the outcomes.
17
The Pre-Test: Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof)
Strengths Challenges
1. “Extra” temporary staff to compile the data.
2. Staff commitment to starting the process.
3. Increased Advisory Board involvement through Program Services Committee.
1. Lack of staff resources2. Lack of staff time3. Lack of staff expertise
18
The Pre-Test
Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof)
Early Attempts – 2010-111. Emergency Rental and Utility Assistance2. Angel Tree3. Turning Point Center
19
The Pre-Test: Emergency Assistance
20
The Pre-Test: Emergency Assistance
• Providing emergency financial assistance for rent, mortgage, and utility arrearages to prevent greater crises—e.g. homelessness prevention.
• What happened to the client after we provided financial assistance?
WE HAD NO IDEA.
21
The Pre-Test: Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof), cont.
22
The Pre-Test: Angel Tree
What did we want to know?1. What are the characteristics of the “typical” family who is registering for Christmas assistance?
2. How can we use information about the families to market the program and increase the number of (corporate) donors who sponsor angels at Christmas?
23
The Pre-Test: Angel Tree, cont.
What data and reports did we have already?Using a common database, we could pull reports for:
1. Number of clients per zip code2. Number of children by age and gender
Doesn’t paint much of a picture…
24
The Pre-Test: Turning Point Center
25
The Pre-Test: Turning Point Center
The Turning Point Center is a two year transitional housing program for young women, between the ages of 18 and 30 years old, with up to four children who are homeless. Families may stay at TP for up to twenty-four months.
26
The Pre-Test: Turning Point Center
• Changes in emphasis within homeless services forced our hand– Reduction/elimination of funding for transitional
housing programs– Increased funding/emphasis on rapid re-housing
and permanent supportive housing programsHow could we use outcomes to strengthen our
position as a TH provider in a PSH/RRH world?
27
The Pre-Test: Turning Point Center
• Needed to be able to show why our transitional program was relevant and necessary– Serve a population (young women with children)
that needs transitional housing b/c many barriers to maintaining permanent housing
– Show success in reducing lengths of stay, increasing numbers served, and better exit destinations, and better outcomes.
28
The Pre-Test: Turning Point Center
THE DREAM (2010)Proposed outcomes (based on former HUD SHP
outcomes) e.g. what we were putting in our grant proposals:1. Increased education and income
2. Exit to permanent housing3. Increased self-determination
29
The Pre-Test: Turning Point CenterTHE REALITY
First Attempt: Using the data we knew we could get quickly, we asked the following:
a. Type of Exit - how did our clients leave?
b. Exit Destination - where did they go?
30
Turning Point Outcomes Data – First Attempt 2010
Client ID Number Outcome
Placement @ Exit
001 T ?002 G TH003 T ?004 G PH005 T ?006 T Family007 T ?008 ASA+ Family009 G ?010 ASA+ Family011 G PH012 G ?013 G ?
31
The Pre-Test – Turning Point CenterThe Results (reported to TP staff and the newly
created Advisory Board Program Committee):• 50% left for positive reasons• 50% left for negative reasons• Lengths of stay ranged from 46 months to 1 month. Average stay - 14 months. • Exit destinations were unclear; no
uniform definition for the exit destination.Honesty time…these were not good.
32
The Pre-Test – Turning Point Center
Additional Questions We Wanted Answered:1. How long were families staying (supposed to be a twenty-four month program)?
2. Were HoHs employed before, during, after program?3. Were household incomes increasing?
4. Which ILS classes were most effective? 5. Were education levels changing?
33
Conclusion of the Pre-Test
We had the will to measure outcomes, but not the means.
34
The Intervention: Creating the Culture
What did we need to do to create a culture of data collection and outcome measurement?
35
The Intervention: Creating the Culture
Step 1
•Don’t reinvent the wheel—who else has a position like this? Turns out, no one.
Step 2
•Dedicate a staff position to tracking outcomes – Program Outcomes Coordinator
Step 3
•Market the position externally - secure funding for the position.
36
The Intervention: Creating the CultureContinued
Step 4
•Market the position internally - get buy in from program staff.
Step 5
•Hiring – don’t hire someone like yourself.
Step 6
•Rally support and get to work. Air the dirty laundry: show our early attempts at data collection and outcomes management.
37
Intervention: Our First 18 Months
What was our Program Outcomes Coordinator able to accomplish? (WHAT?)
What did the data tell us/what outcomes did we have? (SO WHAT?)
What did we do with the information once we had it? (NOW WHAT?)
38
Intervention: What did our Program Outcomes Coordinator accomplish?1. Benchmarking
39
Intervention: What did our Program Outcomes Coordinator accomplish?
2. Defined who the “typical” client is using client data (v. anecdotal stories).
Example: Typical Angel Tree Family• Average Number of Children - 2.5• Majority of Head of Household are Single
Parents – 63.41%• Average Monthly Income - $1,135
($13,620/yr)
40
Intervention: What did our Program Outcomes Coordinator accomplish?
3. Developed logic models for each program.
The Salvation Army Turning Point Center Logic Model
ProblemSingle mothers
with children are struggling with
homelessness and need assistance.
Our ProgramThe Turning Point Center for Women and Children is a transitional living
program. We offer mothers and their
children an environment that will prepare them and their families for independent living and help them achieve greater self-sufficiency.
Inputs
Funding
Staff
Equipment
Facility
Partnerships
Program Development
Program Evaluation
ActivityChild CareTemporary
HousingIndividual and Group Therapy
Financial PlanningIndependent Life
Skills ProgramOut of Poverty
ProgramCase ManagementReferral Services
Child Development Assessments
Job Placement Assistance
Job Readiness Training
OutputsParticipation in Out of Poverty Program
Change in Self-Sufficiency Matrix
Completion of Independent Life
Skills ProgramChange in IncomeChange in Credit
ScoreChange in
EmploymentObtain Permanent
HousingChange in
Education/Job Training
Disabilities for Children Identified
No return to the homelessness
continuum for at least 12 months
Outcomes
Families are able to obtain
permanent housing.
Families who complete our
program achieve greater self-sufficiency.
Families who complete our
program experience an
increase in Quality of Life
41
Intervention: What did our Program Outcomes Coordinator accomplish?
4. Established outcome measures per program.• Emergency Assistance– 30, 60, 90 day follow up asked two questions:
1. Are you still housed or is utility still on?
2. Is your rental or utility balance current?
42
Intervention: What did our Program Outcomes Coordinator accomplish?
4. Established outcome measures per program.• Turning Point Center – Reason for Exit– Destination at Exit– Changes in:• income • education• employment • housing barrier
assessment • Self-Sufficiency Matrix score• Independent Living Skills mastery
43
Intervention: What did our Program Outcomes Coordinator accomplish?
5. Developed tools to measure those outcomes.
A. Emergency Assistance30/60/90 day follow up – online tool,
staff clicks on answers as they speak with clients, then Coordinator can view results immediately
44
Intervention: What did our Program Outcomes Coordinator accomplish?
5. Developed tools to measure those outcomes.
B. Angel TreeZip code analysis overlay with percentage of residents living at or below FPL in that zip code.
45
Zip Code Analysis Overlay with FPL
Angel Tree by
Zip Code
# of Fami-lies
Percent of
Cases
% of Residents with Income
Below the Poverty Line
% of Residents with Income
below 50% of the Poverty
Line.20020 699 25.35% 36.9%* 16.6%*20019 434 15.74% 29.9%* 11.8%*20032 397 14.40% 39.8%* 14.8%*
46
Zip Code Analysis Overlay with FPL
20009 266 9.65% 13.70% 6.40%
20011 231 8.38% 15.50% 6.50%
20002 193 7.00% 21.5%* 10.2%*
20010 171 6.20% 15.30% 8.5%*
20001 115 4.17% 26.8%* 11.6%*
20003 57 2.07% 20.4%* 7.00%20018 55 1.99% 17.40% 10.4%*20024 55 1.99% 21.3%* 7.90%20017 42 1.52% 19.5%* 8.7%*
47
Intervention: What did our Program Outcomes Coordinator accomplish?
5. Developed tools to measure those outcomes.
C.Turning Point Center• Developed entry/exit tools for Turning Point to track changes in income, employment, Self-Sufficiency matrix scores, ILS scores, Out of Poverty pre- and post-test, etc.• Developed standardized definitions for reasons for exit and exit destinations.
48
The Tools
• Examples of– Entry/Exit Form – Reason for Exit/Exit Destination Codes
49
Post-Test: Now What?
1. Reviewing the outcomes2. Disseminating the outcomes3. Making program changes based
on the outcomes in order to improve the outcomes
Reviewing OutcomesEmergency Assistance
Days after Assistance Utilities Connected Utilities Current30 Days 95.60% 68.80%60 Days 100% 46.80%90 Days 97.60% 41.80%
“Is there any other type of assistance or services your family needs at this time?”
• Most Common Responses– Need Legal Assistance • Landlord that won’t repair broken items
– Medical Bills/Affordable Health Services• Usually also resulted from job loss
Post-Test: Reviewing OutcomesAngel Tree
Who is the typical Angel Tree family?• Average Family Size - 5.5• Average Number of Children - 2.5• Majority of Heads of Household are Single
Parents – 63.41%• 48.35% African American, 45.18% Caucasian
with Hispanic Heritage
52
Post-Test: Reviewing OutcomesAngel Tree
“Typical” Angel Tree Family INCOME
Average Income – Angel Tree Family
$1,135/month ($13,620/year)
Average Income – DC Metro Area
$74,665/year
Median Income – Angel Tree Family
$1,022/month ($12,264/year)
Median Income – DC Metro Area
$107,000/year
Post-Test: Angel Tree, cont.
• Our Angel Tree clients live well below the poverty line.
• Our clients average income is 81.76% less than the average income for the DC Metro area.
• Our clients median income is 88.54% less than the median income for the DC Metro area.
Post-Test: Reviewing OutcomesAngel Tree
Typical Angel Tree Family in the DC Metro Area:Average Monthly Income for AT Family
$1,135/monthly
Average FMR in DC Metro Area$1,239 1BDR/$1,469 2BDR
Angel Tree Client Income by Household Size Compared to FMR in DC Region
Household Size
Avg. Monthly Income
% of Income Spent on Housing – 1BR*
% of Income Spent on Housing – 2BR*
2 $680 182% 216%3 $948 130% 154%4 $1220 102% 120%5 $1428 86% 103%6 $1357 91% 108%7 $1528 81% 96%8 $1317 94% 111%9 $1392 89% 106%
10 $1380 89% 106%
*Based on Fair Market Value - $1,239 for 1 Bedroom, $1,469 for 2 Bedroom
56
Post-Test: Reviewing Outcomes Turning Point Center
1. Families at entry to program2. Families at exit3. Reasons for exit4. Exit destinations
“Typical” Turning Point Client at Entry
• Average Age – 22• Average Number of Children – 1.6• Unemployed at Entry to Program– 76.35%• Education Levels at Entry– 43.54% No H.S. Diploma or GED– 36.73% H.S. Diploma– 10.20% GED
• Average Monthly Income of $701 ($8714/yr)
What impact did we have on our Turning Point Clients?
Turning Point Family at Exit 1. Decreased Lengths of Stay
a. Average length of stay in 2011 = 14 monthsb. Average length of stay in 2014 = 11 months
2. Increased Household Income a. Families increased their household income
by 16% from entry to exit.b. Number of families with savings accounts
increased – Only 2 families had savings accounts at entry, now all families savings accounts.
What impact did we have on our Turning Point Clients?
3. Increased Education Levels77% of clients increased their education levels from entry to exit
a. 80% with No HS Diploma Left with HS Diploma or Better
b. 77% with HS Diploma Left with Trade/Voc or Better
c. 70% with a GED Left with Trade/Voc or Better
60
What impact did we have on our Turning Point Clients?
4. Increased Rates of Employment • 31% of women who went through the Turning
Point Program were employed at exit from the program.
• 80% of women who entered the program employed were still employed at exit.
• 16.22% of Mothers who did not have a job upon entering Turning Point, left the program employed.
Reason Percentage
Obtained Housing (Name on Lease) 34.68%
Obtained Housing (Name not on Lease) 22.58%
Voluntarily Leave Program 8.87%
Terminated from Program 33.88%
Left for Positive Reasons 57.26%
Left for Negative Reasons 42.75%
Turning Point – Reason for Exit
Outcome Percentage
Permanent Housing (Name on Lease) 45.04%
Permanent Housing (Name not on Lease) 44.75%
Back into the Homeless System/Continuum of Care 6.11%
Unknown 6.11%
Turning Point - Exit Destinations
Overall Housing Outcomes• 89.79% of clients who left/completed the program went into permanent
housing.– 29.01% went to Subsidized Housing.– 16.03% went to Un-Subsidized Housing.– 32.06% moved in with a Family Member.– 6.11% moved in with a Significant Other.– 4.58% moved in with a Friend.
29%
16%
32%
5%6%
Un-Subsidized Housing
Subsidized Housing
Family
Friend
Significant Other
64
Post-Test: Program Changes Made in Response to Data
• Emergency Assistance– Better knowledge of local resources in order to
respond to the questions about need for:• Legal Assistance –Landlord that won’t repair broken items
• Medical Bills/Affordable Health Services–Usually also resulted from job loss
65
Post-Test: Program Changes Made in Response to Data
• Angel Tree– Publicize better in areas with higher poverty rates– Relocate registration to areas convenient to higher
poverty areas– Use client data to better market the program to
potential donors
66
Post-Test: Program Changes Made in Response to Data
• Turning Point 1. Focus on employment and housing search now
starts immediately upon entry.• Once a family is ready to transition, they exit. They
don’t have to stay the full 24 months if they don’t need to.• Results in decreased lengths of stay which then results
in more families served each year.
67
Post-Test: Program Changes Made in Response to the Data
• Turning Point cont.2. Tweaked referral and application process to
get a better match between families and the program.• Resulted in more families exiting for positive reasons
(50/50 in 2011; 57.26% for positive reasons in 2014).
68
Post-Test: Program Changes Made in Response to Data
• Turning Point cont.3. Greater emphasis on reality testing with families
regarding finances.• Resulted in increased rates of saving, earlier
employment, and creative solutions to housing.
69
Post-Test: Disseminating the Outcomes Information
Telling The Story of the DataTo Whom?– Internally– Externally
70
Post-Test: Telling the Story
• Internally–Program Staff• Held mini-presentations with the various
program staff.–Development Staff • Grants Manager• Major Gifts Manager• PR Manager
71
Post-Test: Telling the Story
• Externally– Advisory Board– Monthly Newsletter to Donors– Impact Statements to Prospective Donors, Annual
Reports, Online – In grant proposals, meetings with other
stakeholders and potential non-profit partners
72
Creating An Outcomes Culture
On a budget and can’t afford to hire an employee just to do outcomes?
You can still get started:1. Don’t make it more difficult than it needs to
be. When in doubt, compare pre-test and post-test results.
73
Creating An Outcomes Culture
2. Assess where you are now – what data do you already collect? How can you use the data to measure change?
3. What gets measured gets improved.Choose what you want to know about and to improve.
74
Creating the Culture
4. Identify people who are interested and willing to help. Utilize the resources around you:
a. Advisory Board membersb. local universitiesc. CoC staffd. United Way, etc.
75
Creating the Culture
5. As you gather results, be sure to communicate them widely.
6. Make changes to the program as needed for improvement.
7. Find an accountability partner(s) to keep you on task.
8. Make a date with data, don’t be scared of it.
76
Thank You• Questions and Answers• Link to presentation: http://bit.ly/omgmt• Contact Information
Major Lewis R. Reckline Leslye E. Wooley, JD/MSW
Area Commander Director of Program Services
Lewis_Reckline@uss.salvationarmy.org
Leslye_Wooley@uss.salvationarmy.org
(202) 756-3913 (202) 756-2649
top related