data, impact, & resources

TECHNICAL ASSISTANCE TECHNICAL ASSISTANCE WORKSHOP:WORKSHOP:

DATA, IMPACT, & DATA, IMPACT, & RESOURCESRESOURCES

City of QuincyCity of Quincy

Dept of Planning & Community Development Dept of Planning & Community Development

December 8December 8thth, 2011, 2011

Realizing Success Without Increased Funding

United Way of Mass Bay and Merrimack Valley

MYTH:MYTH:

Outcomes-Outcomes-based based

evaluation is a evaluation is a waste of waste of

valuable staff valuable staff time!time!

AGENDAAGENDA::►Understand & utilize DATA Understand & utilize DATA

MANAGEMENT; MANAGEMENT; ►Develop quality OUTCOMES & Develop quality OUTCOMES &

INDICATORS; and INDICATORS; and►Advocate effectively for Advocate effectively for

FUNDING from various FUNDING from various sources for programs with sources for programs with social impact.social impact.

LOGIC & DECISION LOGIC & DECISION MAKINGMAKING

INPUTS

ACTIVITIES / OUTPUTS

OUTCOME TARGETS / OUTCOME INDICATORS

FUNDING SOURCES / POLICY

OUTCOMES /IMPACT

LOGIC & EXTERNAL LOGIC & EXTERNAL FORCESFORCES

EXTERNAL FACTORS

ASSUMPTIONS

SITUATION / PRIORITIES

DECISION MAKING

CYCLE Theory of

Change

Intended Impact

CDBG PROGRAM EXAMPLE:CDBG PROGRAM EXAMPLE:Asian Liaison ProgramAsian Liaison Program

Inputs Activities Outputs Outcomes Impacts

Strategy

Grant

Initiative

Outreach to increase the diversity of beneficiaries for City of Quincy CDBG programs.

Offer both direct and indirect service at sites where CDBG programs are implemented.Assist and track clients through data collection, spoken and written materials translation, and ESL classes.

Activities

Inputs Outcomes

ImpactsOutputs

Strategy

Initiative

Grant

CDBG PROGRAM EXAMPLE:CDBG PROGRAM EXAMPLE:SSYMCA - Germantown SSYMCA - Germantown Neighborhood CenterNeighborhood Center

Inputs Activities Outputs Outcomes Impacts

Strategy

Grant

Initiative

Transform neighborhood into a healthy & sustainable community of choice & opportunity.

Offer programs to strengthen the community for women, youth, and family and offer a food pantry to supply basic needs. Provide match funding for services and activities aligned with the initiative while tracking program participants, and success stories.

Activities

Inputs Outcomes

ImpactsOutputs

Strategy

Initiative

Grant

STEPS TO SUCCESSFUL STEPS TO SUCCESSFUL “ACTIONABLE” “ACTIONABLE” MEASUREMENTMEASUREMENT

1.1. Prepare OrganizationPrepare Organization

2.2. Define Outcomes Define Outcomes

3.3. Define IndicatorsDefine Indicators

4.4. Collect DataCollect Data

5.5. Pilot SystemPilot System

6.6. Analyze / Report ResultsAnalyze / Report Results

ORGANIZATIONAL ORGANIZATIONAL READINESSREADINESS

►MissionMission: : For what For what purpose does the purpose does the organization exist?organization exist?►VisionVision: : What would the What would the community look like if the community look like if the organization was no organization was no longer needed?longer needed?


►Decision-MakingDecision-Making:: Does Does the organization’s the organization’s leadershipleadership use DATA to make decisions?use DATA to make decisions?

►ImprovementsImprovements:: Are there Are there opportunities for opportunities for staffstaff to use to use DATA to drive improvements?DATA to drive improvements?


Marguerite Casey Foundation Marguerite Casey Foundation Capacity Assessment Tool, 2001Capacity Assessment Tool, 2001 http://www.caseygrants.org/pages/reshttp://www.caseygrants.org/pages/resources/resources_downloadassessmenources/resources_downloadassessment.asp t.asp Well written, well formatted Well written, well formatted FREE excel downloadFREE excel download Capacity Measured: Leadership, Capacity Measured: Leadership,

Adaptive, Management, OperationalAdaptive, Management, Operational Offers Scores & AnalysisOffers Scores & Analysis

OUTCOMESOUTCOMES

►Difficult:Difficult: Preventative (Health)Preventative (Health) Developmental (Education)Developmental (Education) ““One-time” / Anonymous (Food Shelves)One-time” / Anonymous (Food Shelves)

►Easier:Easier: Counteractive programsCounteractive programs Designed to address observable problemsDesigned to address observable problems Example: Chronic Homelessness Example: Chronic Homelessness

OUTCOMESOUTCOMES►Early ChildhoodEarly Childhood►EducationEducation►Jobs & Economic Jobs & Economic Security Security ►SurvivalSurvival

Citation:Citation: The Robin Hood Foundation The Robin Hood Foundation http://www.robinhood.orghttp://www.robinhood.org

►ImmediateImmediate:: Currently in OR just after Currently in OR just after leaving the programleaving the program

►Intermediate:Intermediate: 3-6 months after program 3-6 months after program►Ultimate:Ultimate: 6-12 months 6-12 months & beyond& beyond program program

OUTCOMESOUTCOMES

INDICATORSINDICATORS►What would I see, hear, or read What would I see, hear, or read

about clients that would mean about clients that would mean progressprogress toward the outcome? toward the outcome?

►What numbers and/or What numbers and/or percentagepercentage would indicate significance? would indicate significance?

►Example:Example: “2,000 or 50% of the “2,000 or 50% of the participants will quit smoking by the participants will quit smoking by the end of the program.”end of the program.”

INDICATORSINDICATORS

Making Connections: Making Connections: The National Survey Indicators The National Survey Indicators DatabaseDatabase Annie E. Casey Foundation Annie E. Casey Foundation http://tarc.aecf.org/initiatives/mc/mcid/index.phphttp://tarc.aecf.org/initiatives/mc/mcid/index.php

FREE download, easy to adaptFREE download, easy to adapt Offers 150 indicators in 8 areas of impact, Offers 150 indicators in 8 areas of impact,

including: including: ►Alliances, Advocacy, & Collective ActionAlliances, Advocacy, & Collective Action►Economic OpportunityEconomic Opportunity►Child and Family well-beingChild and Family well-being

INDICATORSINDICATORSNational Neighborhood National Neighborhood IndicatorsIndicators Partnership: Urban Partnership: Urban Institute, 2008-2012 Institute, 2008-2012 http://neighborhoodindicators.orghttp://neighborhoodindicators.org Community databases of Community databases of

indicators, handbooks, curricula & indicators, handbooks, curricula & technical assistancetechnical assistance

Local Partnerships, (The Boston Local Partnerships, (The Boston Foundation)Foundation)

Data compiled from gov’t, census, Data compiled from gov’t, census, research, & survey sourcesresearch, & survey sources

COLLECTING DATACOLLECTING DATA►Current program recordsCurrent program records►Cost/Time restraints Cost/Time restraints ►Follow-up data collection Follow-up data collection ►Utilize Existing Databases: Utilize Existing Databases:

EXAMPLE: Harvard Family Research EXAMPLE: Harvard Family Research Project OST Evaluation Database:Project OST Evaluation Database: http://gseweb.harvard.edu/hfrp/projehttp://gseweb.harvard.edu/hfrp/projects/afterschool/evaldatabase.htmlcts/afterschool/evaldatabase.html

useful

DATA COLLECTION TIPSDATA COLLECTION TIPS ►Don’t ask for unnecessary dataDon’t ask for unnecessary data►Consider time your survey will takeConsider time your survey will take►Format survey to meet your needsFormat survey to meet your needs►Consider the language you are usingConsider the language you are using►Be sensitive to culture, assumptions, Be sensitive to culture, assumptions,

bias, and loaded questionsbias, and loaded questions

Citation: Innovation Network Citation: Innovation Network http://www.innonet.org/client_docs/File/Survey_Dev_Tips.pdfhttp://www.innonet.org/client_docs/File/Survey_Dev_Tips.pdf

ONLINE SURVEY TOOLSONLINE SURVEY TOOLSSurvey MonkeySurvey Monkey www.surveymonkey.comwww.surveymonkey.com Free version may be useful for small, informal Free version may be useful for small, informal

surveys – 10 questions, 100 responses;surveys – 10 questions, 100 responses; Select version $16.99/mo, or $199/yr – unlimited Select version $16.99/mo, or $199/yr – unlimited

questions & responses, customization, ability to questions & responses, customization, ability to export Excel & PDF filesexport Excel & PDF files

Other Options: Zoomerang.com; SurveyGizmo.com; PollDaddy.com; ConstantContact.com; FormSite.com; Moodle.com

SOFTWARE OPTIONSSOFTWARE OPTIONS► Community Tech Knowledge: Community Tech Knowledge:

http://www.communitytech.nethttp://www.communitytech.net► Vista Share: Vista Share: http://www.vistashare.comhttp://www.vistashare.com►Design Data: Design Data: http://www.ddco.comhttp://www.ddco.com; ;

http://www.outcomeresults.comhttp://www.outcomeresults.com► City Span (Provider): City Span (Provider): http://www.cityspan.comhttp://www.cityspan.com► Social Solutions: Social Solutions:

http://www.socialsolutions.comhttp://www.socialsolutions.com► Athena Case Management (Penelope): Athena Case Management (Penelope): http://http://

www.athenasoftware.netwww.athenasoftware.net

PILOTINGPILOTING►Impact is a long term goal; Impact is a long term goal;

therefore, measure outcomes more therefore, measure outcomes more frequently frequently

►How do joint efforts lead to How do joint efforts lead to solutions? solutions?

►Measure for contribution (part), not Measure for contribution (part), not attribution (credit) attribution (credit)

►Harmonize and CollaborateHarmonize and Collaborate

OUTCOMES & OUTCOMES & COLLABORATIONCOLLABORATION

► Connect & Coordinate the various Connect & Coordinate the various partnerspartners – –non-profits, schools, & other support non-profits, schools, & other support organizationsorganizations

► Identify key indicatorsIdentify key indicators across the across the continuum; continuously make progress continuum; continuously make progress toward improvement toward improvement

► TransformTransform education by education by using data to using data to determine which strategies yield the best determine which strategies yield the best resultsresults and driving resources toward those and driving resources toward those strategiesstrategies

Strive Partnership, Cincinnati – Strive Partnership, Cincinnati – Coalition for Cradle to Career Coalition for Cradle to Career

SuccessSuccess

OUTCOMES & OUTCOMES & COLLABORTATIONCOLLABORTATION

YouTube Video:YouTube Video: http://www.youtube.com/watch?http://www.youtube.com/watch?v=FLqc_9VxfCEv=FLqc_9VxfCE

Citation: Strive Together Partnership, http://www.strivetogether.org

Jeff Edmondson, Executive Director of Strive Partnership

ANALYZE & ANALYZE & REPORTREPORT

►State & Federal Policy Makers State & Federal Policy Makers ►Foundation LeadershipFoundation Leadership►Grantees & Ultimate Grantees & Ultimate

BeneficiariesBeneficiaries►Private DonorsPrivate Donors►PractitionersPractitioners

EVALUATION EVALUATION DASHBOARDSDASHBOARDS

Example: Idashboards:Example: Idashboards: http://examples2.idashboards.com/idashboahttp://examples2.idashboards.com/idashboards/?guestuser=wpgv1rds/?guestuser=wpgv1

CDBG PUBLIC SERVICES CDBG PUBLIC SERVICES EVALUATION DASHBOARD EVALUATION DASHBOARD

FY 06-07 - FY 10-11FY 06-07 - FY 10-11

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

2006-2007

2008-2009

2010-2011

Beneficiaries

$0$200,000$400,000$600,000$800,000

$1,000,000$1,200,000$1,400,000$1,600,000$1,800,000

2006-2007

2008-2009

2010-2011

LeveragingExpenditures

RECOMMENDED RECOMMENDED FOLLOW-UPFOLLOW-UP

Evaluation Plan Workbook, Evaluation Plan Workbook, Innovation Network,Innovation Network,

http://www.innonet.org/client_docs/File/ehttp://www.innonet.org/client_docs/File/evaluation_plan_workbook.pdfvaluation_plan_workbook.pdf

FACTFACT““(With outcomes-based (With outcomes-based

measurement system in place), measurement system in place), We’ve been able to We’ve been able to learn very learn very quickly what is working and quickly what is working and where our participants need where our participants need

more helpmore help.”.”Marcele Carneiro Gama VianaMarcele Carneiro Gama Viana

10,000 Women Project Manager10,000 Women Project ManagerBrazilian University FDCBrazilian University FDC

Measurement as LearningMeasurement as LearningBridgespan, 2011Bridgespan, 2011

QUESTIONS & QUESTIONS & COMMENTSCOMMENTS

REFERENCESREFERENCES► Actionable Measurement Guidelines, Bill & Actionable Measurement Guidelines, Bill &

Melinda Gates Foundation 2010, Melinda Gates Foundation 2010, http://www.gatesfoundation.org/learning/Dohttp://www.gatesfoundation.org/learning/Documents/guide-to-actionable-measurement.cuments/guide-to-actionable-measurement.pdfpdf

► Basic Guide to Outcomes-Based Evaluation Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very for Nonprofit Organizations with Very Limited Resources, Authenticity Consulting, Limited Resources, Authenticity Consulting, LLC., LLC., http://managementhelp.org/evaluation/outchttp://managementhelp.org/evaluation/outcomes-evaluation-guide.htmomes-evaluation-guide.htm

► Community Development Evaluation Community Development Evaluation Storymap and Legend, NeighborWorks Storymap and Legend, NeighborWorks America 2006, America 2006, http://www.nw.org/network/training/documehttp://www.nw.org/network/training/documents/CommunityDevEvalStorymap.pdfnts/CommunityDevEvalStorymap.pdf

REFERENCESREFERENCES► Evaluation Dashboards: Practical Solutions Evaluation Dashboards: Practical Solutions

for Reporting Results, Innovation Network for Reporting Results, Innovation Network 2008, 2008, http://www.innonet.org/resources/files/AEA_http://www.innonet.org/resources/files/AEA_Dashboard_Presentation_Slides.pdfDashboard_Presentation_Slides.pdf

► Key Steps in Outcome Management, Series Key Steps in Outcome Management, Series on Outcome Management for Nonprofit on Outcome Management for Nonprofit Organizations, The Urban Institute 2003, Organizations, The Urban Institute 2003, http://www.urban.org/publications/310776.hhttp://www.urban.org/publications/310776.htmltml

► Low-Cost Online Survey Tools Keep Low-Cost Online Survey Tools Keep Nonprofits on Budget, MassNonProfit 2011, Nonprofits on Budget, MassNonProfit 2011, http://www.massnonprofit.org/expert.php?http://www.massnonprofit.org/expert.php?artid=2593&catid=67artid=2593&catid=67

REFERENCESREFERENCES►Measurement as Learning, BridgeSpan Measurement as Learning, BridgeSpan

2011,2011, http://www.bridgespan.org/measuremhttp://www.bridgespan.org/measurement-as-learning.aspxent-as-learning.aspx

►Program Planning & Development – Program Planning & Development – Program Logic Model, University of Program Logic Model, University of Missouri Extension, Missouri Extension, http://http://extension.missouri.edu/staff/programdextension.missouri.edu/staff/programdev/plmev/plm

►Software for Nonprofit Evaluation and Software for Nonprofit Evaluation and Case Management, Innovation Case Management, Innovation Network 2010, Network 2010, http://www.innonet.org/resources/files/http://www.innonet.org/resources/files/2010-01_eval_software_matrix.pdf2010-01_eval_software_matrix.pdf