Performance Measurement Series:Analyzing, Using and Reporting Data
June 6, 2012
YWCA Great Lakes Alliance Region
Shelly M. SchnuppCapacity Building Consultant
Three Part Series:
1. Establishing the Basis for Program Performance Tracking and Measurement
April 17, 2012 9am CT
2. Defining Performance Indicators and Measures May 1, 2012 9am CT
3. Analyzing, Using and Reporting DataJune 6, 2012 9am CT
SESSION 1 Review Laying the groundwork for
measurement….
SESSION 1 Review Laying the groundwork for
measurement…. Organizational measurement systems and the
role of program performance measurement The Importance of program logic in establishing
the basis for measurement Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples
Organizational measurement systems and the role of program performance measurement
The Importance of program logic in establishing the basis for measurement
Outputs vs. outcomes Mission-driven chain of results Selecting outcomes to measure Resources and examples
SESSION 2 ReviewMaking measurement real….
SESSION 2 ReviewMaking measurement real….
Framing indicators Sources of data Validity, reliability and other criteria for
establishing effective measures Measurement problems and challenges Resources and examples
Framing indicators Sources of data Validity, reliability and other criteria for
establishing effective measures Measurement problems and challenges Resources and examples
5
SESSION 3 Making Outcomes Data Useful
Analyzing, using and reporting outcomes dataBenchmarkingDisaggregating dataTrendsData PresentationData Management Tools
Steps for Analyzing Program Outcomes Data
1. Begin with the Basics
2. Delve Deeper into Client and Service Characteristics
3. Make Sense of the Numbers
Analyzing Outcomes Data
Calculations + Human Judgment
Analysis is as much about thinking as it is about calculating.
Urban Institute, 2004
1. Beginning with the Basics
Calculate overall outcomes for all clients Compare the latest overall outcomes with
outcomes from previous time periods Compare the latest overall outcomes with
pre-established targets Compare the latest overall outcomes with
outcomes for other, similar program and/or to outside standards
Calculate overall outcomes for all clients
Provides a summary overview of program’s results.
The number and percent of people served who achieved the outcome, often specified by the outcome indicator.
25 or 60% of the women who completed the financial literacy program established savings accounts and saved at least $500.
Basic Calculation Issues Determining the Relevant Clients
Who should be included in the denominator? 75 or 60% of the 125 women who completed the
financial literacy program established savings accounts and saved at least $500.
75
125 = .60
OR 75 or 25% of the 300 women who participated in the
financial literacy program established savings accounts and saved at least $500.
75
300 = .25
Benchmarking: Compare the latest overall outcomes with outcomes from previous time periods Comparisons provide context for interpreting outcomes
information. Trends over time can help identify influencing factors. % of girls who increased technology skills
2006 2007 2010 2011
Compare the latest overall outcomes with pre-established targets
Sufficient program delivery allows for establishing performance targets.
35% of women in transitional housing, who complete the program, are expected to move into permanent housing within one year.
Are we meeting expectations? Exceeding or falling short?
Compare the latest overall outcomes with those from similar programs
From programs in same or other communities.
Be cautious about how and when outcomes are measured by others, quality of data.
2. Delving Deeper
Break out and compare client outcomes by demographics, characteristics Age, gender, race/ethnicity, income Program entry status
White women vs. women of color
Women with vs. without high school diploma
Delving Deeper
Break out and compare outcomes by service characteristics
Different service locations Different service providers, caseworkers Different specific interventions Various levels of service engagement
Delving Deeper Compare breakouts with previous reporting
periods or targets.
% of girls who increased technology skills
Location 1
Location 2
2006 2007 2010 2011
3. Making Sense of the Numbers
Identify which numbers should be highlighted
Seek explanations Use and communicate the findings
Explaining the Data
Find factors that explain data-- changes and trends
EXTERNAL Changes in external conditions Social or political changes Changes in client needs, challenges
INTERNAL Unexpected changes in staff, funding leadership Problems with program design
Using the Data
Identify special issues needing attention – missed targets, trends, anomalies
Identify client groups that need attention – disparities based on gender? race? age group?
Identify services procedures that are working– or need improvement
Identify training and technical assistance needs Celebrate good outcomes! Hold regular “how are we doing” meetings
Presenting and Reporting Data
Provide clear context Include numbers and percentages Reports data trends when possible Be cautious about assigning cause Use charts and graphics
Percent of Youth Reaching Performance Standard on Outcome Objectives
56
6760.6
50
0
10
20
30
40
50
60
70
80
ReduceFighting
Improve Self-Esteem
Reduce DrugUse
ImproveGrades
49.7
58.1
51.4
44.2
48.5
62.8
70.6
62.1
55.3
56.7
0 20 40 60 80
Physical
Self-Help
Social
Academic
Communication
Pretest Posttest
Pre and Post Test Scores on Outcome Pre and Post Test Scores on Outcome MeasuresMeasures
Presenting and Reporting Data Tailor to various audiences:
Board of Directors and Staff Funders of the program Potential funders, supporters Program clients, stakeholders
Use to promote organization Prospective Board members Supporters Volunteers Prospective program clients
Managing Data Bruce Wantuch,
Management Consultant
Identify obstacles to effective service delivery
Identify an internal resource to manage the data collection process
Assess computer hardware and network Decide what software package will be used
to house the data collected How to pay for the additional cost of data
collection and management
YWCA of XXXX
Mission
Eliminating racism. Empowering women.
ProgramsCenter for Racial Justice and ActivismEconomic EmpowermentSexual Violence and Support ServicesEarly Childhood Services
YWCA of XXXX
Why ETO?
Obstacles to effective service delivery:Tracking methods inconsistent between programsOutcomes depended on funding source tracking requirementsAnecdotal information about customer makeupService integration challenges between programsLacked the ability to evaluate programs
Needs:Determining who customers are and how they are being servedDetermining whether services are effective and appropriate
Critical Capabilities in ETO
Efforts – used to depict individual progress towards program and funder outcomes
Efforts – used to depict individual progress towards program and funder outcomes
Assessments – help show program results by providing a before and after snaphotAssessments – help show program results by providing a before and after snaphot
Barriers to Outcomes Measurement— typical excuses
But some things can’t be measured. Pre-and post tests are too cumbersome. Our clients will simply tell us what we want to
hear. We never had to do it before now. There’s not enough time.
Pitfalls to Avoid
1. Don't balk at data collection because it seems too "scientific."
2. There is no "perfect" design. It's far more important to do something, than to wait until every last detail has been tested.
3. Consider including interviews. Surveys don't capture "the story," and the story is a powerful depiction of the benefits of your services.
4. Don't interview just the successes. You'll learn a great deal about the program by understanding its failures, dropouts, etc.
5. Don't throw away data once a report has been generated. Results can provide precious information later when trying to understand changes in the program.
Resources to Assist with Measuring Outcomes
Urban Institute www.urban.org Has developed “outcomes taxonomy” for fourteen service
areas Published series on outcome management for nonprofit
organizations Performance Measurement: Getting Results
by Harry Hatry, 2nd ed. 2006 Offers tips on indicator development, sources of data,
customer survey design, trained observer ratings design, etc.
Measuring Performance in Public and Nonprofit Organizations by Theodore Poister, 2003
Resources to Assist with Measuring Outcomes
The Grantsmanship Center www.tgci.com Offers guides to program evaluation
United Way www.unitedway.org/outcomes Offers outcome measurement resources, guides
W.K. Kellog Foundation www.wkkf.org Publishes Logic Model Development Guide and Evaluation
Handbook
Performance Management Network, inc. www.pmn.net search: logic models Instruction on logic models framework and development