1 he says vs. she says model validation and calibration kevin chang hntb corporation
DESCRIPTION
3 Model Validation and Calibration CALIBRATION – an iterative procedure to fine tune model parameters and settings so that the model can achieve what the modeler wants it to perform. VALIDATION – an analytical process to verify if the model’s behavior and output statistics can truly represent actual traffic system operations. PURPOSE – to have a valid simulation model that is able to generate representative numerical results that replicate traffic operations in the modeled network for analyses.TRANSCRIPT
2
• Model Validation and Calibration• Keys to a Successful Simulation Model• Model Validation Concept• Stories on Model Validation• Lessons Learned
Contents
3
Model Validation
and Calibration
• CALIBRATION – an iterative procedure to fine tune model parameters and settings so that the model can achieve what the modeler wants it to perform.
• VALIDATION – an analytical process to verify if the model’s behavior and output statistics can truly represent actual traffic system operations.
• PURPOSE – to have a valid simulation model that is able to generate representative numerical results that replicate traffic operations in the modeled network for analyses.
4
Facts
• Model Calibration– Results may be limited by the tool used– Modeler’s knowledge of the simulation tool – Usually the most time consuming process
• Model and Model Validation– Law and Kelton (1991) : “a simulation model of a complex
system can only be an approximation to the actual system.” – Pegden et al. (1995) : “no model can ever be absolutely
correct”. “A model is created for a specific purpose, and its adequacy or validity can only be evaluated in terms of that purpose.”
– Fu, M : “Model validation is more an art work than science.”
5
KEYS TO A SUCCESSFUL
MODEL
• Use the Right Tool• Modeler’s knowledge on the
– System: traffic environment, operations, controls, management, etc.
– Tools used– Issues to be addressed
• Data availability – Usually the most critical element : availability and accuracy
• Model Validation – Right people to review validation results– Selection of validation objects and focus on objectives– Art of work– Sometimes good luck
6
Model Validation
ConceptTraffic Flow Traffic Operations
Actual System
Highway/Freeway Network
?=
Modeled System
Output Statistics
Model Logic- Vehicle movements and interactions- Traffic assignment, routing decision- Weaving, merging, lane change- Queuing and delay- Traffic controls and managements- etc.
Input Data
7
STORIES ON MODEL
VALIDATION
• Inaccurate traffic demand for model inputs– Demand vs. flows (throughputs)
• Incompatible performance measures– Delays: definition and collection– LOS criteria– Average, maximum, 90th percentile, etc.
• Inconsistent data– Data collected at different times, locations, methods, etc.
• Validating MOE against MEMORY– Usually best or worst scenario will be memorized– Always talk to the right person with field experience
• MOE selection– Quantifiable and collectable with a clear definition: queue
length, delay vs. speed, link density
8
LESSONS LEARNED
• Know your tools • Clear about project issues, system environment, scope
of work• Always budget for data collection and analysis• Select “right” MOEs for model validation and
presentation• Talk to the right person• Upgrade hardware and update software – be sensitive to
the time required to run your models and to get output stats
• Art of work and good luck