creating a culture of outcomes management (omgmt): showing what works national social services and...

Creating a Culture of Outcomes Management (OMGmt): Showing What Works

National Social Services and Disaster Management Conference

March 26, 2014

2

Presenters• Major Lewis R. Reckline, Area Commander,

National Capital Area Command in Washington, DC

• Leslye E. Wooley, Area Command Director of Program Services, Washington, DC

3

Overview of theNational Capital Area Command

• Area Command was created in 2006• Encompasses the Army’s programs in the

following areas:– Two suburban counties in Maryland –

Montgomery and Prince George’s County– Washington, DC– Northern Virginia – City of Alexandria, Arlington,

Fairfax, and Prince William Counties

4

Creation of the Area CommandThe Strengths

• Increased visibility of The Salvation Army across the region

• Increased opportunities to expand programs, provide additional services across the entire region

• Increased the viability of our fundraising• Increased the influence of a regional Advisory

Board

5

Creation of the Area CommandThe Challenges

• Increased scrutiny of the effectiveness of our services

• Increased need for funding, especially new funding streams

• Increased emphasis on reporting our program outcomes

6

Have you or your staff ever looked like this at the mention of

outcomes?

7

Tell the truth…

Have you or your staff ever…1.) broken out into a cold sweat when a grant application asked for outcome information?

2.) laughed manically when a donor asked, “what is your success rate?”

8

Tell the truth…continued

3.) looked blankly at your Development Department colleagues when they ask you for a logic model for your program?

4.) cursed an Advisory Board member who asked “tell us about a ‘typical’ client?”

9

Tell the truth, continued

• If you answered yes, to any of these questions, you are not alone.

• This presentation is the Area Command’s story of how it came to see data as a friend, not an enemy.

10

Recognizing the Importance of Outcome Measurement

“Why did we need to create a culture of data collection and

outcome measurement?”AND

“Who cares about outcomes anyway?!”

11

Why did the Area Command choose to focus on measuring outcomes?

1. Increased accountability/stewardship – are our programs the best that they can be?

a. Are we Doing the Most Good for clients? With donor dollars?

b. To answer, “what is your success rate?”

2. What gets measured, gets improved.• Data backs up our anecdotal evidence.• The data validates the work that we do.• The data shows us our areas of weakness.

12


3. Change in direction/emphasis by funders (example: homeless services).

4. To make informed policy decisions and program changes.

a. If the program is not working, what changes need to be made? Can it be fixed or… b. Does the program need to end?

13


5. To inform stakeholders of our successes.a. Donors – leads to increased donations;b. Funders – leads to increased funding streams;c. Staff – what they are doing is having a positive impact on program participants;d. Program Participants – increased buy in because “this a good program that can really help me.”

14

Who cares about outcomes anyway?

1. Funders – nearly every grant requires information about outcomes from the programs that are applying. Then funders award funding to programs with proven results, “more bang for the buck.”

2. Donors – more major and smaller donors also want to know the outcomes of our programs. They want to know The Salvation Army really is “doing the most good” with their donation.

15

Who cares about outcomes anyway? Continued

3. Policy makers – want to know which interventions work best in order to replicate successful programs, end unsuccessful programs, and (hopefully) increase the funding for those with the strongest outcomes.

4. Stakeholders – including program participants themselves, Advisory Board members, etc.

5. Ourselves/Staff – does our program do what we want/need it to do? Did our intervention produce change? (And, if yes, was it the change we intended?)

16

Outcomes Measurement 101

• Pre-Test – what was our existing culture around data collection

and outcomes measurement at the outset?• Intervention – Creating a dedicated staff position for outcomes

measurement and outcomes management.• Post-Test – reviewing the outcomes, disseminating the outcomes,

and making program changes to improve the outcomes.

17

The Pre-Test: Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof)

Strengths Challenges

1. “Extra” temporary staff to compile the data.

2. Staff commitment to starting the process.

3. Increased Advisory Board involvement through Program Services Committee.

1. Lack of staff resources2. Lack of staff time3. Lack of staff expertise

18

The Pre-Test

Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof)

Early Attempts – 2010-111. Emergency Rental and Utility Assistance2. Angel Tree3. Turning Point Center

19

The Pre-Test: Emergency Assistance

20

The Pre-Test: Emergency Assistance

• Providing emergency financial assistance for rent, mortgage, and utility arrearages to prevent greater crises—e.g. homelessness prevention.

• What happened to the client after we provided financial assistance?

WE HAD NO IDEA.

21

The Pre-Test: Assessing Our Existing Culture Around Outcome Measurement (or Lack Thereof), cont.

22

The Pre-Test: Angel Tree

What did we want to know?1. What are the characteristics of the “typical” family who is registering for Christmas assistance?

2. How can we use information about the families to market the program and increase the number of (corporate) donors who sponsor angels at Christmas?

23

The Pre-Test: Angel Tree, cont.

What data and reports did we have already?Using a common database, we could pull reports for:

1. Number of clients per zip code2. Number of children by age and gender

Doesn’t paint much of a picture…

24

The Pre-Test: Turning Point Center

25


The Turning Point Center is a two year transitional housing program for young women, between the ages of 18 and 30 years old, with up to four children who are homeless. Families may stay at TP for up to twenty-four months.

26


• Changes in emphasis within homeless services forced our hand– Reduction/elimination of funding for transitional

housing programs– Increased funding/emphasis on rapid re-housing

and permanent supportive housing programsHow could we use outcomes to strengthen our

position as a TH provider in a PSH/RRH world?

27


• Needed to be able to show why our transitional program was relevant and necessary– Serve a population (young women with children)

that needs transitional housing b/c many barriers to maintaining permanent housing

– Show success in reducing lengths of stay, increasing numbers served, and better exit destinations, and better outcomes.

28


THE DREAM (2010)Proposed outcomes (based on former HUD SHP

outcomes) e.g. what we were putting in our grant proposals:1. Increased education and income

2. Exit to permanent housing3. Increased self-determination

29

The Pre-Test: Turning Point CenterTHE REALITY

First Attempt: Using the data we knew we could get quickly, we asked the following:

a. Type of Exit - how did our clients leave?

b. Exit Destination - where did they go?

30

Turning Point Outcomes Data – First Attempt 2010

Client ID Number Outcome

Placement @ Exit

001 T ?002 G TH003 T ?004 G PH005 T ?006 T Family007 T ?008 ASA+ Family009 G ?010 ASA+ Family011 G PH012 G ?013 G ?

31

The Pre-Test – Turning Point CenterThe Results (reported to TP staff and the newly

created Advisory Board Program Committee):• 50% left for positive reasons• 50% left for negative reasons• Lengths of stay ranged from 46 months to 1 month. Average stay - 14 months. • Exit destinations were unclear; no

uniform definition for the exit destination.Honesty time…these were not good.

32

The Pre-Test – Turning Point Center

Additional Questions We Wanted Answered:1. How long were families staying (supposed to be a twenty-four month program)?

2. Were HoHs employed before, during, after program?3. Were household incomes increasing?

4. Which ILS classes were most effective? 5. Were education levels changing?

33

Conclusion of the Pre-Test

We had the will to measure outcomes, but not the means.

34

The Intervention: Creating the Culture

What did we need to do to create a culture of data collection and outcome measurement?

35

The Intervention: Creating the Culture

Step 1

•Don’t reinvent the wheel—who else has a position like this? Turns out, no one.

Step 2

•Dedicate a staff position to tracking outcomes – Program Outcomes Coordinator

Step 3

•Market the position externally - secure funding for the position.

36

The Intervention: Creating the CultureContinued

Step 4

•Market the position internally - get buy in from program staff.

Step 5

•Hiring – don’t hire someone like yourself.

Step 6

•Rally support and get to work. Air the dirty laundry: show our early attempts at data collection and outcomes management.

37

Intervention: Our First 18 Months

What was our Program Outcomes Coordinator able to accomplish? (WHAT?)

What did the data tell us/what outcomes did we have? (SO WHAT?)

What did we do with the information once we had it? (NOW WHAT?)

38

Intervention: What did our Program Outcomes Coordinator accomplish?1. Benchmarking

39

Intervention: What did our Program Outcomes Coordinator accomplish?

2. Defined who the “typical” client is using client data (v. anecdotal stories).

Example: Typical Angel Tree Family• Average Number of Children - 2.5• Majority of Head of Household are Single

Parents – 63.41%• Average Monthly Income - $1,135

($13,620/yr)

40


3. Developed logic models for each program.

The Salvation Army Turning Point Center Logic Model

ProblemSingle mothers

with children are struggling with

homelessness and need assistance.

Our ProgramThe Turning Point Center for Women and Children is a transitional living

program. We offer mothers and their

children an environment that will prepare them and their families for independent living and help them achieve greater self-sufficiency.

Inputs

Funding

Staff

Equipment

Facility

Partnerships

Program Development

Program Evaluation

ActivityChild CareTemporary

HousingIndividual and Group Therapy

Financial PlanningIndependent Life

Skills ProgramOut of Poverty

ProgramCase ManagementReferral Services

Child Development Assessments

Job Placement Assistance

Job Readiness Training

OutputsParticipation in Out of Poverty Program

Change in Self-Sufficiency Matrix

Completion of Independent Life

Skills ProgramChange in IncomeChange in Credit

ScoreChange in

EmploymentObtain Permanent

HousingChange in

Education/Job Training

Disabilities for Children Identified

No return to the homelessness

continuum for at least 12 months

Outcomes

Families are able to obtain

permanent housing.

Families who complete our

program achieve greater self-sufficiency.

Families who complete our

program experience an

increase in Quality of Life

41


4. Established outcome measures per program.• Emergency Assistance– 30, 60, 90 day follow up asked two questions:

1. Are you still housed or is utility still on?

2. Is your rental or utility balance current?

42


4. Established outcome measures per program.• Turning Point Center – Reason for Exit– Destination at Exit– Changes in:• income • education• employment • housing barrier

assessment • Self-Sufficiency Matrix score• Independent Living Skills mastery

43


5. Developed tools to measure those outcomes.

A. Emergency Assistance30/60/90 day follow up – online tool,

staff clicks on answers as they speak with clients, then Coordinator can view results immediately

44



B. Angel TreeZip code analysis overlay with percentage of residents living at or below FPL in that zip code.

45

Zip Code Analysis Overlay with FPL

Angel Tree by

Zip Code

# of Fami-lies

Percent of

Cases

% of Residents with Income

Below the Poverty Line

% of Residents with Income

below 50% of the Poverty

Line.20020 699 25.35% 36.9%* 16.6%*20019 434 15.74% 29.9%* 11.8%*20032 397 14.40% 39.8%* 14.8%*

46

Zip Code Analysis Overlay with FPL

20009 266 9.65% 13.70% 6.40%

20011 231 8.38% 15.50% 6.50%

20002 193 7.00% 21.5%* 10.2%*

20010 171 6.20% 15.30% 8.5%*

20001 115 4.17% 26.8%* 11.6%*

20003 57 2.07% 20.4%* 7.00%20018 55 1.99% 17.40% 10.4%*20024 55 1.99% 21.3%* 7.90%20017 42 1.52% 19.5%* 8.7%*

47



C.Turning Point Center• Developed entry/exit tools for Turning Point to track changes in income, employment, Self-Sufficiency matrix scores, ILS scores, Out of Poverty pre- and post-test, etc.• Developed standardized definitions for reasons for exit and exit destinations.

48

The Tools

• Examples of– Entry/Exit Form – Reason for Exit/Exit Destination Codes

49

Post-Test: Now What?

1. Reviewing the outcomes2. Disseminating the outcomes3. Making program changes based

on the outcomes in order to improve the outcomes

Reviewing OutcomesEmergency Assistance

Days after Assistance Utilities Connected Utilities Current30 Days 95.60% 68.80%60 Days 100% 46.80%90 Days 97.60% 41.80%

“Is there any other type of assistance or services your family needs at this time?”

• Most Common Responses– Need Legal Assistance • Landlord that won’t repair broken items

– Medical Bills/Affordable Health Services• Usually also resulted from job loss

Post-Test: Reviewing OutcomesAngel Tree

Who is the typical Angel Tree family?• Average Family Size - 5.5• Average Number of Children - 2.5• Majority of Heads of Household are Single

Parents – 63.41%• 48.35% African American, 45.18% Caucasian

with Hispanic Heritage

52


“Typical” Angel Tree Family INCOME

Average Income – Angel Tree Family

$1,135/month ($13,620/year)

Average Income – DC Metro Area

$74,665/year

Median Income – Angel Tree Family

$1,022/month ($12,264/year)

Median Income – DC Metro Area

$107,000/year

Post-Test: Angel Tree, cont.

• Our Angel Tree clients live well below the poverty line.

• Our clients average income is 81.76% less than the average income for the DC Metro area.

• Our clients median income is 88.54% less than the median income for the DC Metro area.


Typical Angel Tree Family in the DC Metro Area:Average Monthly Income for AT Family

$1,135/monthly

Average FMR in DC Metro Area$1,239 1BDR/$1,469 2BDR

Angel Tree Client Income by Household Size Compared to FMR in DC Region

Household Size

Avg. Monthly Income

% of Income Spent on Housing – 1BR*

% of Income Spent on Housing – 2BR*

2 $680 182% 216%3 $948 130% 154%4 $1220 102% 120%5 $1428 86% 103%6 $1357 91% 108%7 $1528 81% 96%8 $1317 94% 111%9 $1392 89% 106%

10 $1380 89% 106%

*Based on Fair Market Value - $1,239 for 1 Bedroom, $1,469 for 2 Bedroom

56

Post-Test: Reviewing Outcomes Turning Point Center

1. Families at entry to program2. Families at exit3. Reasons for exit4. Exit destinations

“Typical” Turning Point Client at Entry

• Average Age – 22• Average Number of Children – 1.6• Unemployed at Entry to Program– 76.35%• Education Levels at Entry– 43.54% No H.S. Diploma or GED– 36.73% H.S. Diploma– 10.20% GED

• Average Monthly Income of $701 ($8714/yr)

What impact did we have on our Turning Point Clients?

Turning Point Family at Exit 1. Decreased Lengths of Stay

a. Average length of stay in 2011 = 14 monthsb. Average length of stay in 2014 = 11 months

2. Increased Household Income a. Families increased their household income

by 16% from entry to exit.b. Number of families with savings accounts

increased – Only 2 families had savings accounts at entry, now all families savings accounts.


3. Increased Education Levels77% of clients increased their education levels from entry to exit

a. 80% with No HS Diploma Left with HS Diploma or Better

b. 77% with HS Diploma Left with Trade/Voc or Better

c. 70% with a GED Left with Trade/Voc or Better

60


4. Increased Rates of Employment • 31% of women who went through the Turning

Point Program were employed at exit from the program.

• 80% of women who entered the program employed were still employed at exit.

• 16.22% of Mothers who did not have a job upon entering Turning Point, left the program employed.

Reason Percentage

Obtained Housing (Name on Lease) 34.68%

Obtained Housing (Name not on Lease) 22.58%

Voluntarily Leave Program 8.87%

Terminated from Program 33.88%

Left for Positive Reasons 57.26%

Left for Negative Reasons 42.75%

Turning Point – Reason for Exit

Outcome Percentage

Permanent Housing (Name on Lease) 45.04%

Permanent Housing (Name not on Lease) 44.75%

Back into the Homeless System/Continuum of Care 6.11%

Unknown 6.11%

Turning Point - Exit Destinations

Overall Housing Outcomes• 89.79% of clients who left/completed the program went into permanent

housing.– 29.01% went to Subsidized Housing.– 16.03% went to Un-Subsidized Housing.– 32.06% moved in with a Family Member.– 6.11% moved in with a Significant Other.– 4.58% moved in with a Friend.

29%

16%

32%

5%6%

Un-Subsidized Housing

Subsidized Housing

Family

Friend

Significant Other

64

Post-Test: Program Changes Made in Response to Data

• Emergency Assistance– Better knowledge of local resources in order to

respond to the questions about need for:• Legal Assistance –Landlord that won’t repair broken items

• Medical Bills/Affordable Health Services–Usually also resulted from job loss

65


• Angel Tree– Publicize better in areas with higher poverty rates– Relocate registration to areas convenient to higher

poverty areas– Use client data to better market the program to

potential donors

66


• Turning Point 1. Focus on employment and housing search now

starts immediately upon entry.• Once a family is ready to transition, they exit. They

don’t have to stay the full 24 months if they don’t need to.• Results in decreased lengths of stay which then results

in more families served each year.

67

Post-Test: Program Changes Made in Response to the Data

• Turning Point cont.2. Tweaked referral and application process to

get a better match between families and the program.• Resulted in more families exiting for positive reasons

(50/50 in 2011; 57.26% for positive reasons in 2014).

68


• Turning Point cont.3. Greater emphasis on reality testing with families

regarding finances.• Resulted in increased rates of saving, earlier

employment, and creative solutions to housing.

69

Post-Test: Disseminating the Outcomes Information

Telling The Story of the DataTo Whom?– Internally– Externally

70

Post-Test: Telling the Story

• Internally–Program Staff• Held mini-presentations with the various

program staff.–Development Staff • Grants Manager• Major Gifts Manager• PR Manager

71

Post-Test: Telling the Story

• Externally– Advisory Board– Monthly Newsletter to Donors– Impact Statements to Prospective Donors, Annual

Reports, Online – In grant proposals, meetings with other

stakeholders and potential non-profit partners

72

Creating An Outcomes Culture

On a budget and can’t afford to hire an employee just to do outcomes?

You can still get started:1. Don’t make it more difficult than it needs to

be. When in doubt, compare pre-test and post-test results.

73

Creating An Outcomes Culture

2. Assess where you are now – what data do you already collect? How can you use the data to measure change?

3. What gets measured gets improved.Choose what you want to know about and to improve.

74

Creating the Culture

4. Identify people who are interested and willing to help. Utilize the resources around you:

a. Advisory Board membersb. local universitiesc. CoC staffd. United Way, etc.

75

Creating the Culture

5. As you gather results, be sure to communicate them widely.

6. Make changes to the program as needed for improvement.

7. Find an accountability partner(s) to keep you on task.

8. Make a date with data, don’t be scared of it.

76

Thank You• Questions and Answers• Link to presentation: http://bit.ly/omgmt• Contact Information

Major Lewis R. Reckline Leslye E. Wooley, JD/MSW

Area Commander Director of Program Services

[email protected]

[email protected]

(202) 756-3913 (202) 756-2649

creating a culture of outcomes management (omgmt): showing what works national social services and...

Documents

area commander

program outcomes 5major5have

measuring outcomes

mention of outcomes

program changes

program participants

area commands story

staff position