roi committee 2 nd meeting november 1, 2013. committee purpose key decision points timeline...
TRANSCRIPT
ROI Committee 2nd MeetingNovember 1, 2013
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Preliminaries
AgendaMeeting Objectives:
•Review and understand committee process and key decision points
•Review and approve evaluation design developed by the advisory committee, staff, and evaluator
•Make initial decisions on ROI reporting/dashboards
9:00 Welcome and Introductions
9:10 Committee Process and Decision Points
9:20 Evaluation Design
10:00 Audiences and Dashboards
10:50 Wrap-Up and Next Steps
11:00 Adjourn
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Preliminaries
Review and understand work of the ROI Advisory Committee, staff, and
evaluator
Receive and consider recommendations from the
advisory group and staff
Make final recommendations to the full council for approval
Preliminaries
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Evaluation Design
Policy RecommendationsData Sharing, Funding, Incentives, etc.
Management FunctionsHow is ROI used and reported?
GovernanceWho administers/audits/certifies/
reports/interprets ROI?
Preliminaries
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
OctoberIntroduction to ROI
November – DecemberDiscussion and Decisions
JanuaryFinalize Findings / Recommendations
FebruaryPresent to Full Council, Discuss, and Vote
Preliminaries
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Inez Wildwood, ChairMo AmundsonRep. Karen ClarkKatie Clark Sieben (alternate: Cynthia Bauerly)Steve DitschlerFrank Forsberg (alternate: Brian Paulson)Sen. John HoffmanEric Kaler (alternate: Mary Nichols)Mary Russell
Nicholas Maryns, Staff
Preliminaries
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Attend all meetings; listen/watch webinars if a meeting is missed
Actively participate, do any reading/assignments between meetings
Express your opinions; engage in thoughtful discussion and debate
Disagree respectfully
Act as an “ambassador” for the ROI Initiative; share what you’ve learned
Preliminaries
Committee Purpose
Key Decision Points
Timeline
Membership
Responsibilities
Procedure
Meeting AttendanceNo quorum needed to hold a meeting
VotingQuorum needed to hold an in-person/conference call vote;
Majority of quorum needed for approval;Votes can be held over email over an established timeframe
Others?
Overview and Basic Parameters
Values
Objectives of ROI
Scope of Programs Manageable, easy to administer
Useful, relevant and timely
Understandable, transparent
Credible, trusted
Adaptable
Sensitive to change
Overview and Basic Parameters
Smarter Investments
Improving Services
Strengthening Accountability/Transparency
Communicating Value
Standardizing Results
Values
Objectives of ROI
Scope of Programs
Overview and Basic Parameters
Publicly-administered and funded workforce programs
DEED and other state agencies
Non-profit passthroughs
Public education (elements of K-12 and PS)
State and federal competitive grants and special initiatives
Programs serving targeted populations (e.g. people with disabilities, veterans)
Long-term VisionIndependent nonprofits and education
providers
Other service areas
Values
Objectives of ROI
Scope of Programs
Evaluation Design
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
TransparencyPolicy Framework
Management Framework
Cost-Benefit Analysis
Regression Analysis
Evaluation Design
Staff Recommendation:
Approve of the evaluation design developed by the Advisory Group,
evaluator, and staff.
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Directly Observed Benefits(no assumptions needed)
EmploymentIncome / Employment Status
Public Assistance SavingsMFIP / SNAP / UI
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Partially Imputed Benefits(requiring some assumptions)
EmploymentFringe Benefits (healthcare, retirement, vacation): 20% of gross wages
TaxesIncome Tax: use estimators like TaxSIM and those developed by the MN Dept. of Revenue
Payroll Tax: change in earnings * statutory rate
Sales Tax: change in earnings * average marginal sales tax rate as provided by DOR Tax Incidence Studies
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Partially Imputed Benefits(requiring some assumptions)
Healthcare SavingsMinnesotaCare and Medical Assistance: observe enrollment, use average monthly costs at different eligibility levels provided by DHS
Incarceration AvoidanceChange in costs derived from differences in incarceration and recidivism rates. Cost per inmate equals length of stay (using averages as needed) * marginal per diem cost per inmate.
Note: We currently track felony-level offenses; misdemeanors should be added in a future iteration.
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Program CostsTime-weighted / Service-weighted
Cost to Participant
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Savings associated with misdemeanor offenses
Some public benefits•Subsidized housing costs•Prescription Drug Program costs•Child Support payments
Other important but difficult-to-quantify effects•Change in mental and physical health•Change in worker productivity•Reduction in criminal activity•Economic multipliers
An important corollary: additional types of benefits should be considered for inclusion in future iterations.
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
No economic multiplier
No projection of impacts beyond timeframes we observe
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Benefits and costs are estimated from two distinct perspectives:
The Participant
The Taxpayer / Government
Together, the two add up to total social return on investment
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Benefits and costs are estimated from two distinct perspectives:
The Participant
The Taxpayer / Government
Together, the two add up to total social return on investment
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
All results are based on a net-impact methodology, i.e. using control groups
Difference-in-Difference Kernel Density Propensity Score Matching
(“as good as random”)
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Net Impact
Comparison Group
Treatment Group
Earnings
Time
Not the relevant comparison
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Treatment CohortsDefined by Time of Entrance
Control GroupsUnemployment Insurance ApplicantsMnWorks.net and WFC Registrants
In Future IterationsMFIP Recipients
ABE Enrollees (not participants)
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Contextualized Performance Targets Adjusted for population served, local conditions
Leading IndicatorsFor near-term relevance; based on statistical
relationships between near-term and long-term indicators
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Near TermAverage of quarters 2-3 after exit
Medium TermAverage of quarters 5-6 after exit
Long-TermAverage of quarters 9-12 after exit
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
Complete methodology and data analysis process should be published
with as much transparency as possible, up to current standards in the
econometrics literature.
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
Evaluation Design
All analyses should be accompanied with a full set of caveats with regard to
methodology and interpretation
TOPICS
Cost-Benefit Analysis
Benefits
Costs
What’s Not Included
B/C Perspectives
Regression Analysis
Net Impact
Cohorts
Other Features
Timeframes
Transparency
ROI in Practice
Policy Framework
Management Framework
Cost-Benefit Analysis
Regression Analysis
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
How to make information useful?
Which questions & decision points should the dashboard inform? For
which audiences?
Frequency of reporting/analysis/review
Simplicity vs. Nuance
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
Audiences
More FormativeMore Summative
Providers“What drives value?”
Investors*“What are we getting for
our money?”
* Broadly defined; can include legislators, taxpayers, and philanthropy
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
Object of Evaluation
More SpecificMore Broad
Services
Funding Streams
* Broadly defined; can include legislators, taxpayers, and philanthropy
Programs
Providers
Target Populations
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
• Contextual Questions
• General Performance Questions
• Questions about Disparities
• Explanatory Questions
• Current and Future Performance
• Methodological Questions
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
Program Over Time
Contextualized Targets
Similar Programs•How direct of a comparison?•How is “similar” defined?
• Population Served• Programmatic Goals• Intensity of Service
(cost, duration, treatment)
•Training vs. Non-Training•High- vs. Low-Intensity Programs
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
ROI in Practice
Dynamic, Online Dashboard
Print ReportsPrimary Report1.Performance Across Whole System, Including Major Funding Streams
2.Performance by Service Type (“What works?”)
3.Performance Across Target Populations (“How well are we serving _____?”)
Individual Program Reports
Technical Appendices
DASHBOARD TOPICS
Big Questions
Audiences
Objects of Evaluation
Categories of Questions
Comparisons
Reporting
Discussion and Next Steps
Upcoming Meetings
Thoughts and Questions?
Discussion and Next Steps
Upcoming Meetings
ROI Committee Webinar #3Friday, November 159:00 am – 11:00 am
ROI Advisory Group MeetingWednesday, November 61:00 pm – 3:00 pmDEED
Motivations, History, Partners
Overview and Basic Parameters
Pilot Project ROI in PracticeEvaluation Design
Preliminaries Next Steps
The Return on Investment Initiative