Download - Data, Impact, & Resources
TECHNICAL ASSISTANCE TECHNICAL ASSISTANCE WORKSHOP:WORKSHOP:
DATA, IMPACT, & DATA, IMPACT, & RESOURCESRESOURCES
City of QuincyCity of Quincy
Dept of Planning & Community Development Dept of Planning & Community Development
December 8December 8thth, 2011, 2011
Realizing Success Without Increased Funding
United Way of Mass Bay and Merrimack Valley
MYTH:MYTH:
Outcomes-Outcomes-based based
evaluation is a evaluation is a waste of waste of
valuable staff valuable staff time!time!
AGENDAAGENDA::►Understand & utilize DATA Understand & utilize DATA
MANAGEMENT; MANAGEMENT; ►Develop quality OUTCOMES & Develop quality OUTCOMES &
INDICATORS; and INDICATORS; and►Advocate effectively for Advocate effectively for
FUNDING from various FUNDING from various sources for programs with sources for programs with social impact.social impact.
LOGIC & DECISION LOGIC & DECISION MAKINGMAKING
INPUTS
ACTIVITIES / OUTPUTS
OUTCOME TARGETS / OUTCOME INDICATORS
FUNDING SOURCES / POLICY
OUTCOMES /IMPACT
LOGIC & EXTERNAL LOGIC & EXTERNAL FORCESFORCES
EXTERNAL FACTORS
ASSUMPTIONS
SITUATION / PRIORITIES
DECISION MAKING
CYCLE Theory of
Change
Intended Impact
CDBG PROGRAM EXAMPLE:CDBG PROGRAM EXAMPLE:Asian Liaison ProgramAsian Liaison Program
Inputs Activities Outputs Outcomes Impacts
Strategy
Grant
Initiative
Outreach to increase the diversity of beneficiaries for City of Quincy CDBG programs.
Offer both direct and indirect service at sites where CDBG programs are implemented.Assist and track clients through data collection, spoken and written materials translation, and ESL classes.
Activities
Inputs Outcomes
ImpactsOutputs
Strategy
Initiative
Grant
CDBG PROGRAM EXAMPLE:CDBG PROGRAM EXAMPLE:SSYMCA - Germantown SSYMCA - Germantown Neighborhood CenterNeighborhood Center
Inputs Activities Outputs Outcomes Impacts
Strategy
Grant
Initiative
Transform neighborhood into a healthy & sustainable community of choice & opportunity.
Offer programs to strengthen the community for women, youth, and family and offer a food pantry to supply basic needs. Provide match funding for services and activities aligned with the initiative while tracking program participants, and success stories.
Activities
Inputs Outcomes
ImpactsOutputs
Strategy
Initiative
Grant
STEPS TO SUCCESSFUL STEPS TO SUCCESSFUL “ACTIONABLE” “ACTIONABLE” MEASUREMENTMEASUREMENT
1.1. Prepare OrganizationPrepare Organization
2.2. Define Outcomes Define Outcomes
3.3. Define IndicatorsDefine Indicators
4.4. Collect DataCollect Data
5.5. Pilot SystemPilot System
6.6. Analyze / Report ResultsAnalyze / Report Results
ORGANIZATIONAL ORGANIZATIONAL READINESSREADINESS
►MissionMission: : For what For what purpose does the purpose does the organization exist?organization exist?►VisionVision: : What would the What would the community look like if the community look like if the organization was no organization was no longer needed?longer needed?
ORGANIZATIONAL ORGANIZATIONAL READINESSREADINESS
►Decision-MakingDecision-Making:: Does Does the organization’s the organization’s leadershipleadership use DATA to make decisions?use DATA to make decisions?
►ImprovementsImprovements:: Are there Are there opportunities for opportunities for staffstaff to use to use DATA to drive improvements?DATA to drive improvements?
ORGANIZATIONAL ORGANIZATIONAL READINESSREADINESS
Marguerite Casey Foundation Marguerite Casey Foundation Capacity Assessment Tool, 2001Capacity Assessment Tool, 2001 http://www.caseygrants.org/pages/reshttp://www.caseygrants.org/pages/resources/resources_downloadassessmenources/resources_downloadassessment.asp t.asp Well written, well formatted Well written, well formatted FREE excel downloadFREE excel download Capacity Measured: Leadership, Capacity Measured: Leadership,
Adaptive, Management, OperationalAdaptive, Management, Operational Offers Scores & AnalysisOffers Scores & Analysis
OUTCOMESOUTCOMES
►Difficult:Difficult: Preventative (Health)Preventative (Health) Developmental (Education)Developmental (Education) ““One-time” / Anonymous (Food Shelves)One-time” / Anonymous (Food Shelves)
►Easier:Easier: Counteractive programsCounteractive programs Designed to address observable problemsDesigned to address observable problems Example: Chronic Homelessness Example: Chronic Homelessness
OUTCOMESOUTCOMES►Early ChildhoodEarly Childhood►EducationEducation►Jobs & Economic Jobs & Economic Security Security ►SurvivalSurvival
Citation:Citation: The Robin Hood Foundation The Robin Hood Foundation http://www.robinhood.orghttp://www.robinhood.org
►ImmediateImmediate:: Currently in OR just after Currently in OR just after leaving the programleaving the program
►Intermediate:Intermediate: 3-6 months after program 3-6 months after program►Ultimate:Ultimate: 6-12 months 6-12 months & beyond& beyond program program
OUTCOMESOUTCOMES
INDICATORSINDICATORS►What would I see, hear, or read What would I see, hear, or read
about clients that would mean about clients that would mean progressprogress toward the outcome? toward the outcome?
►What numbers and/or What numbers and/or percentagepercentage would indicate significance? would indicate significance?
►Example:Example: “2,000 or 50% of the “2,000 or 50% of the participants will quit smoking by the participants will quit smoking by the end of the program.”end of the program.”
INDICATORSINDICATORS
Making Connections: Making Connections: The National Survey Indicators The National Survey Indicators DatabaseDatabase Annie E. Casey Foundation Annie E. Casey Foundation http://tarc.aecf.org/initiatives/mc/mcid/index.phphttp://tarc.aecf.org/initiatives/mc/mcid/index.php
FREE download, easy to adaptFREE download, easy to adapt Offers 150 indicators in 8 areas of impact, Offers 150 indicators in 8 areas of impact,
including: including: ►Alliances, Advocacy, & Collective ActionAlliances, Advocacy, & Collective Action►Economic OpportunityEconomic Opportunity►Child and Family well-beingChild and Family well-being
INDICATORSINDICATORSNational Neighborhood National Neighborhood IndicatorsIndicators Partnership: Urban Partnership: Urban Institute, 2008-2012 Institute, 2008-2012 http://neighborhoodindicators.orghttp://neighborhoodindicators.org Community databases of Community databases of
indicators, handbooks, curricula & indicators, handbooks, curricula & technical assistancetechnical assistance
Local Partnerships, (The Boston Local Partnerships, (The Boston Foundation)Foundation)
Data compiled from gov’t, census, Data compiled from gov’t, census, research, & survey sourcesresearch, & survey sources
COLLECTING DATACOLLECTING DATA►Current program recordsCurrent program records►Cost/Time restraints Cost/Time restraints ►Follow-up data collection Follow-up data collection ►Utilize Existing Databases: Utilize Existing Databases:
EXAMPLE: Harvard Family Research EXAMPLE: Harvard Family Research Project OST Evaluation Database:Project OST Evaluation Database: http://gseweb.harvard.edu/hfrp/projehttp://gseweb.harvard.edu/hfrp/projects/afterschool/evaldatabase.htmlcts/afterschool/evaldatabase.html
useful
DATA COLLECTION TIPSDATA COLLECTION TIPS ►Don’t ask for unnecessary dataDon’t ask for unnecessary data►Consider time your survey will takeConsider time your survey will take►Format survey to meet your needsFormat survey to meet your needs►Consider the language you are usingConsider the language you are using►Be sensitive to culture, assumptions, Be sensitive to culture, assumptions,
bias, and loaded questionsbias, and loaded questions
Citation: Innovation Network Citation: Innovation Network http://www.innonet.org/client_docs/File/Survey_Dev_Tips.pdfhttp://www.innonet.org/client_docs/File/Survey_Dev_Tips.pdf
ONLINE SURVEY TOOLSONLINE SURVEY TOOLSSurvey MonkeySurvey Monkey www.surveymonkey.comwww.surveymonkey.com Free version may be useful for small, informal Free version may be useful for small, informal
surveys – 10 questions, 100 responses;surveys – 10 questions, 100 responses; Select version $16.99/mo, or $199/yr – unlimited Select version $16.99/mo, or $199/yr – unlimited
questions & responses, customization, ability to questions & responses, customization, ability to export Excel & PDF filesexport Excel & PDF files
Other Options: Zoomerang.com; SurveyGizmo.com; PollDaddy.com; ConstantContact.com; FormSite.com; Moodle.com
SOFTWARE OPTIONSSOFTWARE OPTIONS► Community Tech Knowledge: Community Tech Knowledge:
http://www.communitytech.nethttp://www.communitytech.net► Vista Share: Vista Share: http://www.vistashare.comhttp://www.vistashare.com►Design Data: Design Data: http://www.ddco.comhttp://www.ddco.com; ;
http://www.outcomeresults.comhttp://www.outcomeresults.com► City Span (Provider): City Span (Provider): http://www.cityspan.comhttp://www.cityspan.com► Social Solutions: Social Solutions:
http://www.socialsolutions.comhttp://www.socialsolutions.com► Athena Case Management (Penelope): Athena Case Management (Penelope): http://http://
www.athenasoftware.netwww.athenasoftware.net
PILOTINGPILOTING►Impact is a long term goal; Impact is a long term goal;
therefore, measure outcomes more therefore, measure outcomes more frequently frequently
►How do joint efforts lead to How do joint efforts lead to solutions? solutions?
►Measure for contribution (part), not Measure for contribution (part), not attribution (credit) attribution (credit)
►Harmonize and CollaborateHarmonize and Collaborate
OUTCOMES & OUTCOMES & COLLABORATIONCOLLABORATION
► Connect & Coordinate the various Connect & Coordinate the various partnerspartners – –non-profits, schools, & other support non-profits, schools, & other support organizationsorganizations
► Identify key indicatorsIdentify key indicators across the across the continuum; continuously make progress continuum; continuously make progress toward improvement toward improvement
► TransformTransform education by education by using data to using data to determine which strategies yield the best determine which strategies yield the best resultsresults and driving resources toward those and driving resources toward those strategiesstrategies
Strive Partnership, Cincinnati – Strive Partnership, Cincinnati – Coalition for Cradle to Career Coalition for Cradle to Career
SuccessSuccess
OUTCOMES & OUTCOMES & COLLABORTATIONCOLLABORTATION
YouTube Video:YouTube Video: http://www.youtube.com/watch?http://www.youtube.com/watch?v=FLqc_9VxfCEv=FLqc_9VxfCE
Citation: Strive Together Partnership, http://www.strivetogether.org
Jeff Edmondson, Executive Director of Strive Partnership
ANALYZE & ANALYZE & REPORTREPORT
►State & Federal Policy Makers State & Federal Policy Makers ►Foundation LeadershipFoundation Leadership►Grantees & Ultimate Grantees & Ultimate
BeneficiariesBeneficiaries►Private DonorsPrivate Donors►PractitionersPractitioners
EVALUATION EVALUATION DASHBOARDSDASHBOARDS
Example: Idashboards:Example: Idashboards: http://examples2.idashboards.com/idashboahttp://examples2.idashboards.com/idashboards/?guestuser=wpgv1rds/?guestuser=wpgv1
CDBG PUBLIC SERVICES CDBG PUBLIC SERVICES EVALUATION DASHBOARD EVALUATION DASHBOARD
FY 06-07 - FY 10-11FY 06-07 - FY 10-11
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
2006-2007
2008-2009
2010-2011
Beneficiaries
$0$200,000$400,000$600,000$800,000
$1,000,000$1,200,000$1,400,000$1,600,000$1,800,000
2006-2007
2008-2009
2010-2011
LeveragingExpenditures
RECOMMENDED RECOMMENDED FOLLOW-UPFOLLOW-UP
Evaluation Plan Workbook, Evaluation Plan Workbook, Innovation Network,Innovation Network,
http://www.innonet.org/client_docs/File/ehttp://www.innonet.org/client_docs/File/evaluation_plan_workbook.pdfvaluation_plan_workbook.pdf
FACTFACT““(With outcomes-based (With outcomes-based
measurement system in place), measurement system in place), We’ve been able to We’ve been able to learn very learn very quickly what is working and quickly what is working and where our participants need where our participants need
more helpmore help.”.”Marcele Carneiro Gama VianaMarcele Carneiro Gama Viana
10,000 Women Project Manager10,000 Women Project ManagerBrazilian University FDCBrazilian University FDC
Measurement as LearningMeasurement as LearningBridgespan, 2011Bridgespan, 2011
QUESTIONS & QUESTIONS & COMMENTSCOMMENTS
REFERENCESREFERENCES► Actionable Measurement Guidelines, Bill & Actionable Measurement Guidelines, Bill &
Melinda Gates Foundation 2010, Melinda Gates Foundation 2010, http://www.gatesfoundation.org/learning/Dohttp://www.gatesfoundation.org/learning/Documents/guide-to-actionable-measurement.cuments/guide-to-actionable-measurement.pdfpdf
► Basic Guide to Outcomes-Based Evaluation Basic Guide to Outcomes-Based Evaluation for Nonprofit Organizations with Very for Nonprofit Organizations with Very Limited Resources, Authenticity Consulting, Limited Resources, Authenticity Consulting, LLC., LLC., http://managementhelp.org/evaluation/outchttp://managementhelp.org/evaluation/outcomes-evaluation-guide.htmomes-evaluation-guide.htm
► Community Development Evaluation Community Development Evaluation Storymap and Legend, NeighborWorks Storymap and Legend, NeighborWorks America 2006, America 2006, http://www.nw.org/network/training/documehttp://www.nw.org/network/training/documents/CommunityDevEvalStorymap.pdfnts/CommunityDevEvalStorymap.pdf
REFERENCESREFERENCES► Evaluation Dashboards: Practical Solutions Evaluation Dashboards: Practical Solutions
for Reporting Results, Innovation Network for Reporting Results, Innovation Network 2008, 2008, http://www.innonet.org/resources/files/AEA_http://www.innonet.org/resources/files/AEA_Dashboard_Presentation_Slides.pdfDashboard_Presentation_Slides.pdf
► Key Steps in Outcome Management, Series Key Steps in Outcome Management, Series on Outcome Management for Nonprofit on Outcome Management for Nonprofit Organizations, The Urban Institute 2003, Organizations, The Urban Institute 2003, http://www.urban.org/publications/310776.hhttp://www.urban.org/publications/310776.htmltml
► Low-Cost Online Survey Tools Keep Low-Cost Online Survey Tools Keep Nonprofits on Budget, MassNonProfit 2011, Nonprofits on Budget, MassNonProfit 2011, http://www.massnonprofit.org/expert.php?http://www.massnonprofit.org/expert.php?artid=2593&catid=67artid=2593&catid=67
REFERENCESREFERENCES►Measurement as Learning, BridgeSpan Measurement as Learning, BridgeSpan
2011,2011, http://www.bridgespan.org/measuremhttp://www.bridgespan.org/measurement-as-learning.aspxent-as-learning.aspx
►Program Planning & Development – Program Planning & Development – Program Logic Model, University of Program Logic Model, University of Missouri Extension, Missouri Extension, http://http://extension.missouri.edu/staff/programdextension.missouri.edu/staff/programdev/plmev/plm
►Software for Nonprofit Evaluation and Software for Nonprofit Evaluation and Case Management, Innovation Case Management, Innovation Network 2010, Network 2010, http://www.innonet.org/resources/files/http://www.innonet.org/resources/files/2010-01_eval_software_matrix.pdf2010-01_eval_software_matrix.pdf