building the knowledge base how to successfully evaluate a tcsp project presented by william m....
TRANSCRIPT
Building the Knowledge BaseBuilding the Knowledge BaseHow to Successfully Evaluate A TCSP ProjectHow to Successfully Evaluate A TCSP Project
presented bypresented by
William M. LyonsWilliam M. LyonsU.S. Department of Transportation/Volpe CenterU.S. Department of Transportation/Volpe CenterandandJohn H. SuhrbierJohn H. SuhrbierChristopher D. PorterChristopher D. PorterCambridge Systematics, Inc.Cambridge Systematics, Inc.
September 14/15, 2000September 14/15, 2000
AgendaAgenda
Role of evaluation in the TCSP programRole of evaluation in the TCSP program
• William LyonsWilliam Lyons
Responsibilities for evaluationResponsibilities for evaluation
• William LyonsWilliam Lyons
Guidelines for project evaluationsGuidelines for project evaluations
• John Suhrbier and Chris PorterJohn Suhrbier and Chris Porter
Examples of approaches to project evaluation Examples of approaches to project evaluation
• Washington, DCWashington, DC
• Hartford, CTHartford, CT
Questions and discussionsQuestions and discussions
Why Evaluation?Why Evaluation?
Legislative intent of Section 1604 of TEA-21 (TCSP)Legislative intent of Section 1604 of TEA-21 (TCSP)
• Emphasis on Emphasis on learninglearning
Improve knowledgeImprove knowledge of the costs and benefits of the costs and benefitsof innovative approaches to integrating:of innovative approaches to integrating:
• Transportation investments and strategiesTransportation investments and strategies
• Community preservation, Community preservation,
• Land development patterns, andLand development patterns, and
• Environmental qualityEnvironmental quality
InvestigateInvestigate relationships between transportation and relationships between transportation and community and system preservationcommunity and system preservation
Project and Program EvaluationProject and Program EvaluationCrucial and ComplementaryCrucial and Complementary
Project evaluationProject evaluation
• Results of individual projectsResults of individual projects
• Responsibility of grantee, with technical assistanceResponsibility of grantee, with technical assistance
• Direct source of learningDirect source of learning
• Challenge Challenge objectivity, quality, consistency, credibility objectivity, quality, consistency, credibility
- Results useful locally as well as nationallyResults useful locally as well as nationally
Program evaluationProgram evaluation
• Project evaluations, taken togetherProject evaluations, taken together
• Focus on program goals, analysis, synthesisFocus on program goals, analysis, synthesis
• Assessment of success of overall programAssessment of success of overall program
Desired Results Desired Results What We Are Trying to DoWhat We Are Trying to Do
Build national knowledge base, leverage resultsBuild national knowledge base, leverage results
Provide insights into what is and what is not workingProvide insights into what is and what is not working
Evaluate innovations and results through systematic analysis Evaluate innovations and results through systematic analysis
• Document learningDocument learning
- Problems as well as successesProblems as well as successes
- Falling short of expected goals is acceptableFalling short of expected goals is acceptable
Foster innovation, encourage experimentationFoster innovation, encourage experimentation
Demonstrate success of programDemonstrate success of program
Context Context Major ConsiderationsMajor Considerations
Evaluation plan Evaluation plan how comprehensive? how comprehensive?
Timing Timing early planning and action are critical early planning and action are critical
Focus on key innovationsFocus on key innovations
Process versus outcome results Process versus outcome results
Description versus documentation and credible analysisDescription versus documentation and credible analysis
Importance of flexibilityImportance of flexibility
Effort should be interesting and worthwhile for all involvedEffort should be interesting and worthwhile for all involved
Session focus Session focus practical strategies to improve evaluation practical strategies to improve evaluation
Roles and ResponsibilitiesRoles and ResponsibilitiesHow to Improve Effectiveness?How to Improve Effectiveness?
GranteesGrantees evaluation of individual TCSP projects evaluation of individual TCSP projects
FHWA HQFHWA HQ evaluation of overall TCSP pilot program evaluation of overall TCSP pilot program
• Availability of technical assistanceAvailability of technical assistance• Resource centersResource centers• U.S. DOT/Volpe Center and Cambridge Systematics, Inc. U.S. DOT/Volpe Center and Cambridge Systematics, Inc.
Division OfficesDivision Offices • Encourage and support evaluationEncourage and support evaluation• Point of contact for technical assistancePoint of contact for technical assistance• Need for flexibilityNeed for flexibility• What’s realistic?What’s realistic?
TCSP EvaluationTCSP EvaluationMost Simply IsMost Simply Is
Identifying the set of messages to be used inIdentifying the set of messages to be used in
Telling the TCSP storyTelling the TCSP story
What is Evaluation?What is Evaluation?
TCSP grants are resulting in things (planning and projects) TCSP grants are resulting in things (planning and projects) being done differentlybeing done differently
Most basically, evaluation addressesMost basically, evaluation addresses
• What are the What are the benefitsbenefits of these changes? of these changes?
• What actions are important in achieving these benefits?What actions are important in achieving these benefits?
• What do these changes cost in terms of time and dollars?What do these changes cost in terms of time and dollars?
• Are longer term cost savings being achieved?Are longer term cost savings being achieved?
• Are the benefits worth the extra effort?Are the benefits worth the extra effort?
• How can these successes be repeated, or How can these successes be repeated, or “institutionalized”, within other projects?“institutionalized”, within other projects?
Target Audiences For Project Evaluation ResultsTarget Audiences For Project Evaluation Results
TCSP project staffTCSP project staff
• Help refine project work planHelp refine project work plan- What is working?What is working?- What is not working?What is not working?
Local elected and appointed officialsLocal elected and appointed officials
• What is the project accomplishing that is different?What is the project accomplishing that is different?
Other TCSP granteesOther TCSP grantees
• Learning from othersLearning from others
Input to overall TCSP program evaluationInput to overall TCSP program evaluation
EvaluationEvaluationAssessment of Experience to DateAssessment of Experience to Date
Most people (grantees, FHWA, and other program participants)Most people (grantees, FHWA, and other program participants)
• Recognize that TCSP is resulting in lots of interesting Recognize that TCSP is resulting in lots of interesting initiativesinitiatives
• Acknowledge importance of assessing benefits of Acknowledge importance of assessing benefits of these initiativesthese initiatives
• Want to do a good jobWant to do a good job
• They have only a limited understanding of evaluation They have only a limited understanding of evaluation approaches and methodologiesapproaches and methodologies
• Not really sure how to get startedNot really sure how to get started
Most also admit Most also admit
• Anecdotal evidence is helpful but not sufficientAnecdotal evidence is helpful but not sufficient
• Evaluation is more than public relations or marketingEvaluation is more than public relations or marketing
Available Evaluation ResourcesAvailable Evaluation Resources
Guidance for the preparation of TCSP evaluation plansGuidance for the preparation of TCSP evaluation plans
Selected references evaluating the relationships between Selected references evaluating the relationships between travel and land use, including contact informationtravel and land use, including contact information• Literative reviews (9)Literative reviews (9)• Individual studies (16)Individual studies (16)
Evaluation examplesEvaluation examples• Upgrade of a rural highway corridorUpgrade of a rural highway corridor• Transit supportive land use developmentTransit supportive land use development• Implementation of urban design, land use, and TDM, Implementation of urban design, land use, and TDM,
strategies for an urban office complexstrategies for an urban office complex
TCSP web siteTCSP web site
Technical assistance Technical assistance
Components of an Evaluation ProcessComponents of an Evaluation Process
DefinitionDefinition
ProjectProject
InitiationInitiation
On-GoingOn-Going
CompletionCompletion
Assess Implementation Assess Implementation ExperienceExperience
Compare “After” to Compare “After” to “Before” Conditions“Before” Conditions
DiscretionaryDiscretionaryGrantsGrants
Preliminary Preliminary Evaluation PlanEvaluation Plan
ReportingReporting
Refined Refined Evaluation PlanEvaluation Plan
EarmarkedEarmarkedGrantsGrants
Evaluation PlanEvaluation Plan
ReportingReporting
Assess Implementation Assess Implementation ExperienceExperience
Compare “After” to Compare “After” to “Before” Conditions“Before” Conditions
Steps in Developing an Evaluation Plan Steps in Developing an Evaluation Plan
Identify Data Identify Data Sources and Sources and Evaluation Evaluation
MethodsMethods
Identify Identify Performance Performance
MeasuresMeasures
Define Project Define Project Goals and Goals and ObjectivesObjectives
How Much Effort is Required For Evaluation?How Much Effort is Required For Evaluation?
For a planning grantFor a planning grant
• Not muchNot much
• More a willingness to step back and ask the right questionsMore a willingness to step back and ask the right questions
For an implementation grantFor an implementation grant
• More effort requiredMore effort required
• Longer term monitoring possible withLonger term monitoring possible with
- Before/after dataBefore/after data
- Statistical controlStatistical control
TCSP Web SiteTCSP Web SiteHome PageHome Page
Web PageWeb PageProject DescriptionsProject Descriptions