mobility mangement evaluation: the long and short of it
DESCRIPTION
MOBILITY MANGEMENT EVALUATION: The Long and Short of It. CIVITAS Training 7 May 2014 Florence, Italy. Eric N. Schreffler Transport Consultancy ESTC San Diego, CA USA. OUTLINE. My experience Why evaluate? and for whom? Forecasting vs. measurement - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/1.jpg)
CIV ITAS TRAINING7 MAY 2014
FLORENCE, ITALY
MOBILITY MANGEMENT EVALUATION:
The Long and Short of It
Eric N. SchrefflerTransport ConsultancyESTC San Diego, CA USA
![Page 2: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/2.jpg)
OUTLINE
1. My experience
2. Why evaluate? and for whom?
3. Forecasting vs. measurement
4. Principles of good evaluation
5. Overall framework for MM evaluation
6. U.S. examples – California and Washington D.C.
7. Transitioning from short-term to long-term evaluation
8. The politics of MM evaluation
9. Common mistakes
10. Top ten tips for effective evaluation
![Page 3: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/3.jpg)
Quick Bio
MM/TDM evaluation for over 30 years Written/contributed to several seminar guidebooks Involved in several EC-funded projects
MOST – MOST MET MAX – MaxEVA CIVITAS GUARD CIVITAS II –final brochure CIVITAS PLUS training
Currently leading evaluation of TDM component of 6 U.S.
Urban Partnership Agreement projects
![Page 4: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/4.jpg)
Why Evaluate? And for Whom?
Good management practice! Measure progress against objectives
Output or outcome objectives Program or public policy objectives Integral to performance-based planning
![Page 5: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/5.jpg)
Why Evaluate? And for Whom?
Benchmark program against peers Compare cost effectiveness to other solutions
Marginal cost to accommodate a car vs. cost to reduce a car trip
e.g., LA Metro evaluation performance-based planning Satisfy funding entities Satisfy policy or oversight boards
![Page 6: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/6.jpg)
Don’t be afraid of evaluation!!!
Know your evaluation,
Love your evaluation,
One day your evaluation may save your (life) program.
![Page 7: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/7.jpg)
Forecasting vs. Results
Evaluation can be defined as: A priori estimation of expected impacts (forecasting) Ex post measurement of outcomes (results)
Forecasting Predicting what could/should occur Using comparative case studies, models,
sketch planning, SWAG Measurement
Quantifying what has actually occurred Using counts, surveys, etc.
Forecast
Actual
![Page 8: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/8.jpg)
PROST! Principles for Sound Evaluation
Practical
Rigorous
Objective
Standardized
Timely
![Page 9: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/9.jpg)
FRAMEWORK FOR EVALUATION
Important to plan for evaluation Have plan to frame approach and tasks Should include:
Purpose and objectives Data collection methods Analysis methods Budget Schedule Reporting
Use conceptual framework - MaxEva
![Page 10: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/10.jpg)
MaxEva Assessment Levels
Assessment of Services Provided
A Project activities and outputsB Awareness of mobility services provided
C Usage of mobility services provided
D Satisfaction with mobility services provided
Assessment of Mobility Options Offered
E Acceptance of mobility option offered
F Take up of mobility option offered
G Satisfaction with the mobility option offered
Overall Effects H Long-term attitudes and behavior
I System impacts
MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva
![Page 11: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/11.jpg)
European Example – SwedenEmployee Public Transit “Test Rider” Pass Pilot
Assessment of Services Provided
A Outputs: posters, intranet info, meetings, 1,000 employees
B Awareness: 48% of public transit ticket offer (480 emp)
C Usage: 28% participated in meeting, sought info
D Satisfaction: 80% satisfied with info and pilot concept
Assessment of Mobility Options Offered
E Acceptance: 6% signed a contract to participate (54 emp)
F Take up: 85% of these used PT; 5% used before
G Satisfaction: 90% of participants satisfied with PT
Overall Effects
H Long-term: after one year, 40% still using PT
I System impacts: car use 110K km; CO2 20 tons/year
MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva
![Page 12: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/12.jpg)
TDM and Highway Reconstruction US101 Cuesta Grade CA
Reconstruction mitigation$730,000/year for TDMThree elements:
More commuter express buses Vanpool promotion/subsidies Carpool fuel incentive
Surveyed all participantsRemoved 310 cars dailyCarpool incentive most cost
effective ($3.36/VTR/day)
U.S. Examples
![Page 13: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/13.jpg)
Regional TDM Program Commute ConnectionsWashington D.C.
Triennial evaluation since 1997 Consistent approach (MaxEva) Evaluate total impacts = travel,
emissions, energy (e.g., 118K fewer car trips per day)
Evaluate separate impacts Ridematching Employer outreach Mass marketing Guaranteed ride home Bike to work Carshare Telework Incentives
U.S. Examples
![Page 14: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/14.jpg)
SHORT-TERM TO LONG-TERM
Short-run generally covers one year or year or duration of project funding duration.
Uses before and after data
Long-term can include projecting impacts into future (BCA, lifecycle)
Or can involve time series data for program over years
Key is planning and consistency
![Page 15: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/15.jpg)
The Politics of Evaluation
There may be pressure to:
- subvert findings
- document desired outcomes
- spin the results
RESEARCH
![Page 16: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/16.jpg)
Common Evaluation Mistakes
1. Putting off evaluation
2. Underfunding evaluation
3. Assuming “projected” = “actual” results
4. Incorrectly comparing projected to actual
5. Projecting target group findings to entire population
6. Ignoring causality and externalities
![Page 17: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/17.jpg)
Common Evaluation Mistakes (con’t)
7. Confusing outputs with outcomes
8. Changing methods/assumption mid-evaluation
9. Assuming all mode shift from drive alone
10. Ignoring access mode to new travel options
11. Ignoring non-response in surveys
12. Giving into pressure to change findings
![Page 18: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/18.jpg)
Top Ten Evaluation Tips
1. Get help, at least the first time
2. Plan, plan, plan Plan for evaluation Performance-based plan Stick to the plan
3. Budget for evaluation
![Page 19: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/19.jpg)
Top Ten Evaluation Tips
4. Keep findings simple!
5. Don’t be afraid of what you may find!
6. Be confident! No whining!
![Page 20: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/20.jpg)
Top Ten Evaluation Tips
7. Seek local default factors (e.g. trip length)
8. Use standardized methods/tools/guidance
9. Learn to talk like an engineer
![Page 21: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/21.jpg)
Top Ten Evaluation Tips
10. JUST DO IT!
![Page 22: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/22.jpg)
Key Resources
MaxEva – The EPOMM Evaluation Toolwww.epomm.eu/maxeva
Evaluation Matters: A Practitioners’ Guide to Sound Evaluation for Urban Mobility Measures
www.civitas.eu/content/evaluation-matters
Integrating Demand Management into the Transportation Planning Process: A Desk Reference (Chapter 9)
www.ops.fhwa.dot.gov/publications/fhwahop12035/index.htm
Canadian TDM Measurement Toolboxwww.tc.gc.ca/media/documents/programs/tdm-toolbox.pdf
![Page 23: MOBILITY MANGEMENT EVALUATION: The Long and Short of It](https://reader034.vdocuments.us/reader034/viewer/2022052414/56816928550346895de06488/html5/thumbnails/23.jpg)
Grazie per la vostra attenzione!
Eric N. Schreffler
Transportation Consultant
ESTC
San Diego, CA, USA