uncertainty in transport modelling; the four-stage næstved

Uncertainty in transport modelling; the four-stage Næstved model case study

Stefano Manzo Otto A. Nielsen Carlo G. Prato

DTU Transport, Technical University of Denmark

Rationale

• An extensive literature has demonstrated that there is a considerable and almost systematic inaccuracy between forecasted and observed traffic flows

• One of the reason of this inaccuracy can be traced in the complexity of systems generating traffic. Complex systems generate unpredictable output (emerging behaviour)

• Whenever a model is created to reproduce a complex system the output it generates will invariably be unpredictable because affected by uncertainty

• If not properly understood and quantified, the uncertainty inherent in transport models makes analyses based on their output highly unreliable

2 25-09-2012


Outline

• Introduction

• Case study and methodology • Results

• Conclusions

3 25-09-2012


Introduction • Ideally we would like to build deterministic models; in this way modelled

output would be perfectly accurate. Uncertainty and ignorance prevent from making this possible, originating the modeller inability to model a complex system with a deterministic approach

• Ignorance refers to a complete absence of awareness of some model components or their dynamics

• Uncertainty instead refers to limited knowledge (epistemic) or stochasticity (ontological) of some model components and the way they interact

– “Everything we do not know to a full extent” (Van Zuylen 1999)

– “Any departure from the unachievable ideal of complete deterministic

knowledge of the relevant system” (Walker 2003)

4 25-09-2012


Introduction

5 25-09-2012


Introduction • Transport model point output only represents one of the possible output

generated by the model

• “(...) modelled output is better expressed as a central estimate and an overall range of uncertainty margins articulated in terms of values and likelihood of occurrence” (Boyce 1999)

• Uncertainty analysis pertains to:

– how uncertainty in each model component affects the model output – how to express the model output as a distribution, so as to reflect the

overall uncertainty in the model

6 25-09-2012


Case study and methodology

7

• Næstved model (Traffic Analyst)

– 106 zones – 315 links – low congestion – four-stage model (3 overall iterations) – 2 modes: private and public transport – 2 categories: home/work and

business trips – traffic is modelled on 24H time interval

25-09-2012



8 25-09-2012



9 25-09-2012

Model (Parameters) Type Category 1 Category 2

Assignment

Length 0.350 0.350

Free TT 0.520 1.300

Congested TT 0.820 1.300

Error Term 0.000 0.000

Price 1.000 1.000

Type Name Value

Small road Alpha 0.800

Large road Alpha 0.500

Highway Alpha 0.450

Small road Beta 1.500

Large road Beta 2.500

Highway Beta 4.000

Small road Gamma 0.150

Large road Gamma 0.100

Highway Gamma 0.000

Model (Parameters) Name Category 1 Category 2

Trip generation

WP_PRIM 1.061 0.118 WP_SEC 1.432 0.159

Workplaces 0.005 0.001 Workers 1.342 0.149

Gravity model GravAlpha 0.052 0.052 GravBeta 0.043 0.043

Mode Split Beta 0.060 0.060

Model (Input) Name

Trip Generation

Production Workers

Workplaces

Attraction WP_PRIM

WP_SEC

Assignment

FreeSpeed

QueueSpeed

LaneHCFor

LaneHCBack



• This study investigates uncertainty in model input and parameters through stochastic sampling technique combined with sensitivity analysis

– Latin hypercube sampling simulation (100 draws) – Input (Triangular) Parameters (Lognormal) – Model sensitivity analysis (100 runs)

• Uncertainty is quantified in terms of Coefficient of Variation (CV)

10 25-09-2012


Case study and methodology • Different scenarios are analysed

– three uncertainty locations (input, parameters and total) – two levels of uncertainty (CV 0.1 and 0.3) – three levels of congestion (1,1.5 and 3 times the base traffic)

• Analysis focuses on:

– uncertainty propagation pattern – effects on output uncertainty of uncertainty location – effects on output uncertainty of network congestion

11 25-09-2012


Results

12 25-09-2012

Correlation Model Pearson Spearman

Workers (Input) Generation 0.24021 0.21905

Workers (Parameter) Generation 0.86464 0.85929

Beta (Parameter) Mode split 0.22432 0.20462


Results

13 25-09-2012

Generation Distribution Mode Assignment

Total 0.366 0.487 0.511 0.366

Parameters 0.282 0.348 0.455 0.224

Input 0.241 0.323 0.364 0.300


Results

14 25-09-2012


CV 0.3 0.366 0.487 0.511 0.366

CV 0.1 0.119 0.162 0.294 0.136


Results

15 25-09-2012


SUE 0.366 0.487 0.511 0.366

UE 0.366 0.490 0.510 0.422


Results

16 25-09-2012

Not ready yet


Cong 3 0.365 0.594 0.668 0.371

Cong 1.5 0.366 0.487 0.511 0.366

Cong 1 0.371 0.500 0.436 0.361


Results

17 25-09-2012


Results

18 25-09-2012


Results

19 25-09-2012


Conclusions • In terms of propagation pattern and uncertainty location

– uncertainty increases throughout the first three model stages to then

reduce in the assignment model; this pattern is irrespective to uncertainty location, level of network congestion and base CV

– trip generation stage seems to define the level of final model uncertainty

– SUE reduces final uncertainty more than UE

– input uncertainty appears to have an higher influence on final model

uncertainty than parameters uncertainty

20 25-09-2012


Conclusions • In terms of network congestion

– different levels of congestion do not seem to result in significant

differences in terms of final model uncertainty – with regard to traffic volume the dispersion around the CV mean

value reduces at increasing level of veh/cap ratio and traffic volumes – with regard to travel time, the level of dispersion around the CV

mean value appear to be stable, irrespective to travel time length

21 25-09-2012


Perspective • The method illustrated requires many repeated simulations. Is it possible

to reduce the number of model runs without losing information?

• Correlation between parameters in the sampling procedure? In case of forecasts, which values should be used for the parameters?

• How big is the cost of inaccurate forecasts and how time-consuming and effective is the implementation of uncertainty analyses?

• How does transport model uncertainty affect new infrastructure?

22 25-09-2012


Thanks for the attention

23 25-09-2012


Break in case of emergency 1

24 25-09-2012

Uncertainty matrix

Location

Context (system modeled)

Model (structure/methodology)

Input (from the system)

Parameters (calibrated/assumed)

Model output (propagated)

Level

Stat

istic

al

unce

rtain

ty

Scen

ario

un

certa

inty

Rec

ogni

zed

igno

ranc

e

Stat

istic

al

unce

rtain

ty

Scen

ario

un

certa

inty

Rec

ogni

zed

igno

ranc

e

Stat

istic

al

unce

rtain

ty

Scen

ario

un

certa

inty

Rec

ogni

zed

igno

ranc

e

Stat

istic

al

unce

rtain

ty

Scen

ario

un

certa

inty

Rec

ogni

zed

igno

ranc

e

Stat

istic

al

unce

rtain

ty

Scen

ario

un

certa

inty

Rec

ogni

zed

igno

ranc

e

Nat

ure Epistemic

Ontological

Modified after Walker et al. (2003)


Break in case of emergency 2

25 25-09-2012

Socio-economic analyses in perspective: Uncertainties and bias in decision support

Associate Professor, PhD Kim Bang Salling DTU Transport Traffic days in Aalborg 2012 – Special session: “Uncertainties in Transport Project Evaluation (UNITE)”

DTU Transport, Technical University of Denmark 2

Project Plan of UNITE

Uncertainties in Transport Project Evaluation (UNITE): the five Work-Packages

(5) Evaluation methodologyWP5 project leader: Steen Leleur (DMG)

(4) Uncertainty calculation in transport modelsWP4 project leader: Otto Anker Nielsen (TMG)

(2) Organizational context of Modelling, an empirical study

WP2 project leader: Petter Næss (AAU)

(3) Uncertainty calculation of cost estimates

WP3 project leader: Bo Friis Nielsen (DTU Informatics)

(1) Systematic biases in transport models (recognized ignorance), an empirical studyWP1 project leader: Petter Næss (AAU)


How do we evaluate transport projects?

• Various existing guideline report: –Denmark, Sweden, UK, European Union, ....

• Socio-economic analysis by the use of Cost-Benefit

Analysis (CBA)

• Produces single point estimates such as Net Present Values (NPV), Benefit Cost Ratios (BCR), etc

• However, no common rule have been set in order to acommodate the uncertainties in CBA!

–Recent conducted PhD dissertation proved this point


Background & Motivation

• The Manual for socio-economic analysis in the transport sector (2003)

–Unique guidelines for evaluating transport infrastructure projects

–Lack of uncertainty handling –Expected revision 2012-2013


How do we evaluate transport projects?

• However, no common rule have been set in order to acommodate the uncertainties in CBA!

–Recent conducted PhD dissertation proved this point


The Case Study: HH-Connection • Connecting Denmark with Sweden: Scandinavian link

–Currently, close to the capacity limit on Oresund

HH-Connection (alternatives*)

Description (Alignment of connection)

Cost (million DKK)

Alternative 1 Tunnel for rail (2 tracks) person traffic only 7,700

Alternative 2 Tunnel for rail (1 track) goods traffic only 5,500

Alternative 3 Bridge for road and rail (2x2 lanes & 2 tracks) 11,500

Alternative 4 Bridge for road (2x2 lanes) 6,000

* Larsen, L.A. & Skougaard, B.Z. (2010). Vurdering af alternativer for en fast forbindelse Helsingør-Helsingborg, M.Sc. thesis, Department of Transport, Technical University of Denmark (in Danish)


The UNITE-DSS Modelling Framework Todays Outline


Results: Cost-Benefit Analysis

• Construction costs – by far the largest contributor of costs

• User Benefits – by far the largest contributor of benefits – Consists of Ticket revenue and time savings – Relies on the prognosis of future number of passengers i.e.

demand forecasts

HH-Connection (alternatives)

Cost (million DKK)

BCR NPV (million DKK)

Alternative 1 7,700 1.50 5,530

Alternative 2 5,500 0.16 -6,640

Alternative 3 11,500 2.71 28,240

Alternative 4 6,000 3.08 17,860


Are we telling the truth?!?!

Construction cost overruns

0%

200%

400%

600%

800%

1000%

1200%

1400%

1600%

1800%

2000%

Suez

Can

al

Sydn

eyO

pera

Hou

se

Con

cord

eSu

pers

onic

Aero

plan

eBo

ston

'sAr

tery

/Tun

nel

Proj

ect,

USA

Hum

ber

Brid

ge, U

K

Bost

on-

Was

hing

ton-

New

Yor

kG

reat

Bel

tR

ail T

unne

l,D

KA6

Mot

orw

ayC

hape

l-en-

le-

Frith

/Wha

ley

Shin

kans

enJo

etsu

Rai

llin

e, J

apan

Was

hing

ton

met

ro, U

SA

Cha

nnel

Tunn

el, U

K &

Fran

ceKa

rlsru

he-

Bret

ten

light

rail,

Ger

man

yØ

resu

ndAc

cess

link

s,D

K &

Swed

enM

exic

o ci

tym

etro

line

,M

exic

oPa

ris-A

uber

-N

ante

rre

rail

line,

Fra

nce

Cos

t Ove

rrun

s (%

)

Q: Have we learned anything from history?

”Chunnel” in 1987 £2,600 million (’85 prices) Completion 1994 £4,650 million (’85 prices) Total cost overrun of approx. 80%

”Øresund access link” in 1991 3.2 billion DKK (’90 prices) Completion 1998 5.4 billion DKK (’90 prices) Total cost overrun of approx. 68%


Theoretical anchoring The Transport Planning Phase: Adapted from the British Department for Transport (DfT) (2004)

Reference Class Forecasting: Optimism Bias

Inside View Outside View

”Uniqueness” of Project

”The Planning Fallacy”

Reference Class Forecasting

Forecasting of particular projects

Forecasting from a group of projects

(1) Identification of relevant reference

classes

(2) Establishing probability distribution

(3) Placing and comparing the

project

Optimism Bias UpliftsCurrent Situation


Optimism Bias and uplifts

• Deriving uplifts is highly dependet on large data-sets –Flyvbjerg from (AAU) has since 2003 developed a large

database –Unfortunately, it looks upon mega-projects

• The basis is Reference Class Forecasting i.e. statistical measurements on various project pools

Source: Flyvbjerg and COWI (2004)


Results : Optimism Bias Uplifts

• The BCR are lower, however, still point estimates towards DM –Moreover an advanced form of sensitivity analysis

• Imply to introduce risk analysis and Monte Carlo simulation

HH-Connection (alternatives)

Cost (uplifted) (million DKK)

BCR (orig.) (from slide 8)

BCR (uplifts): 80% uplift

Alternative 1 12,090 1.50 0.97

Alternative 2 8,640 0.16 0.10

Alternative 3 15,180 2.71 1.75

Alternative 4 7,920 3.08 1.98


The UNITE Project Database (UPD)

• The convention used is as follows: ( )( )

forecasted

forecastedactual

XXX

U100×−

=

Over estimation of Demand


• Demand forecasts (user benefits) are derived: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build

• Combination of two database samples

0

5

10

15

20

25

30

(-12

0;-1

00)

(-10

0;-8

0)

(-80

;-60)

(-60

;-40)

(-40

;-20)

(-20

;0)

(0;2

0)

(20;

40)

(40;

60)

(60;

80)

(80;

100)

(100

;120

)

(120

;140

)

(140

;160

)

(160

;180

)

(180

;200

)

(200

;220

)

(220

;240

)

Freq

uenc

y of

occ

uren

ce (%

)

Inaccuracies in demand forecasts (%)

Inaccuracies in demand forecasts (road projects)

Salling et al. (2012)

Flyvbjerg et al. (2003)

Nicolaisen et al. (2012)


The UNITE Project Database (UPD)

• The convention used is as follows: ( )( )

forecasted

forecastedactual

XXX

U100×−

=

Under estimation of costs


• Construction costs bias derived similarly: – U is percent inaccuracy, – Xa is the actual traffic after the project is opened – Xf is the forecasted traffic on the decision to build

• Combination of two database samples

0

10

20

30

40

50

(-10

0;-8

0)

(-80

;-60)

(-60

;-40)

(-40

;-20)

(-20

;0)

(0;2

0)

(20;

40)

(40;

60)

(60;

80)

(80;

100)

(100

;120

)

(120

;140

)

(140

;160

)

(160

;180

)

(180

;200

)

(200

;220

)

(220

;240

)

Freq

uenc

y of

occ

uren

ce (%

)

Inaccuracies in construction costs (%)

Inaccuracies in construction cost (road projects)

Salling et al (2012)

Flyvbjerg et al. (2003)

Nicolaisen et al. (2012)


Results (RCF): Monte Carlo simulation


Conclusions

• Feasibility risk assessment can be carried out by using historical experience stemming from RCF in order to obtain interval results

• An important aspect in RCF and UNITE is to set and validate input parameters. Hence, empirical data enter the assessment.

• The RCF approach has been illustrated on a case example concerning the construction of a new fixed link, the HH-Connection, between Denmark and Sweden.

• Clearly vital to include uncertainties within socio-economic analyses in order to validate results


Perspectives

• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)

• More info on UNITE can be found: (www.transport.dtu.dk/unite)

• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.

http://www.transport.dtu.dk/unite


SIMSIGHT: Decision Conferencing (DC)

20

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model

• Enables to include Stakeholders and Decision-makers in an early stage, i.e. to include experts opinion on MIN and MAX values as entries to the Monte Carlo simulation


Results from DC and RSF


SIMSIGHT: Overconfidence

22


Perspectives

• Recovering of further data (UPD) with regard to both the demand forecast uncertainty as well as the construction costs through large-scale research study

• Producing so-called decision conferences in order to achieve better input parameters to the UNITE-DSS Model combined with overconfidence theory allows for expert opinions (SIMSIGHT)

• More info on UNITE can be found: (www.transport.dtu.dk/unite)

• An international conference on the topic is scheduled in September 2013 – a specific call will be posted in the upcoming month.

http://www.transport.dtu.dk/unite


Thank you for your attention!

Affiliation:

Associate Professor, PhD Kim Bang Salling

Department of Transport Technical University of Denmark

[email protected]

Introduktion af beslutningskonferencer - Vurderinger af usikkerheder i beslutningsgrundlaget for samfundsøkonomiske analyser Trafikdage, 28-08-2012 Michael Bruhn Barfod Adjunkt, DTU Transport


Introduktion

• Beslutningstagning inden for transportplanlægning – Ofte kompleks proces – Mange interessenter involveret

• Fokus på mangler og usikkerheder ifm. samfundsøkonomiske analyser

• UNITE

– Fastlæggelse af trafikprognoser – Fastlæggelse af anlægsoverslag – Evalueringskriterier som punktestimater

• Hvordan kan beslutningstagerne/interessenterne involveres i analysen af

usikkerhederne?

28-08-2012 Trafikdage, Aalborg Universitet 2


SIMDEC beslutningsstøttesystemet • SIMDEC tilgangen har til formål at udføre omfattende vurderinger af

infrastrukturprojekter, hvor der kan tages højde for en lang række af kriterier og synspunkter

Vurdering

Beslutningskonference

SIMDEC tilgangen

28-08-2012 3 Trafikdage, Aalborg Universitet


SIMDEC tilgangen

• Flere forskellige metoder kombineres:

– Feasibility risk assessment: • Cost-benefit analyse • Monte Carlo simulering

– Multi-kriterie analyse: • REMBRANDT • SMARTER

• Formålet er at benytte SIMDEC til vurderingen og allokere passende scorer til alternativerne og vægte til kriterierne



Overblik over SIMDEC tilgangen

(Leleur og Ambrasaite, 2012) 28-08-2012 5 Trafikdage, Aalborg Universitet


Beslutningskonference • En beslutningskonference kan ses som et strategisk værktøj, der kan

benyttes til interaktion med beslutningstagere/interessenter

• Set-up’et afhænger af diverse faktorer (type, tid, deltagere, etc.)

• 3 elementer interagerer

Gruppe processer

IT

Beslutnings-analyse



Beslutningsprocessen


The preliminary problem structuring phase

The five steps of the decision conference

Step 2-Definition of the

criteria/impacts that are to be assessed

(possibility to include more or omit some)

Step 3-Scoring of

alternatives within each criterion/impact

Step 4-Weighting of criteria

-Trade-off considerations

Step 5-Results

-Sensitivity analysis-Validation

Step 1-Introduction to the

concepts and techniques to be used

in the process

Possibility for revising assessments in order to accomplish shared understanding

Identify the problem Select an appropriate analytic approach

Develop a detailed analytic structure

Possible decision

Focus on the problem

Focus on the alternatives

Problem formulation

Generate relevant criteria (workshops etc.)

Select one or more assessment techniques

Division of problem into simple judgments

Division of criteria if defined too broad


Case studie

• Forbindelse mellem Helsingør og Helsingborg (HH) – 3 alternative løsningsmuligheder

• Samfundsøkonomi

– CBA alle 3 alternativer rentable – Riskoanalyse rentabilitet ikke sikker


B/C rate

NPV (mio. DKK)

Certainty Value

Alt 1 1.23 2657 27 %

Alt 2 2.38 40506 77 %

Alt 3 1.99 38518 69 %

CV = 77%


Strategiske kriterier

• Kriterier identificeret og fundet relevante for valget af alternativ

– Samfundsøkonomisk robusthed – Forbedringer for passagerbiler og offentlig transport – Effekt på byer og arealanvendelse – Effekt på regional økonomi – Effekt på logistisk fleksibilitet – Bidrag til EU’s grønne korridorer



Scorer for alternativer Intensitet Definition

0 Ens

2 Svag

4 Klar

6 Stærk

8 Meget stærk

1, 3, 5, 7 Kompromis

Kriterium: Samfundsøkonomisk robusthed

A1 A2 A3 Score

A1 0 -6 -4 0,10

A2 6 0 3 8,00

A3 4 -3 0 1,26

Samfundsøkonomisk robusthed



Kriterievægte

Kriterier Rangorden Bæredygtighed

Bussines-as-usual

Samfundsøkonomisk robusthed 2 (0.24)

Forbedringer for passagerbiler og offentlig transport 3 (0.19)

Effekt på byer og arealanvendelse 6 (0.04)

Effekt på regional økonomi 1 (0.30)

Effekt på logistisk fleksibilitet 5 (0.09)

Bidrag til EU’s grønne korridorer 4 (0.14)

Deltagerne kan opdeles / prædefineret rangorden til diskussion



Resultater



Fordele ved beslutningskonferencer

• Er en beslutning, som tages på baggrund af konsensus ved en beslutningskonference, mere eller mindre valid end vurderinger og løsninger, som findes uden hjælpemidler?

• En beslutningskonference giver følgende fordele:

– Bedre kommunikation imellem grupper – Fælles forståelse af strategiske mål – Dedikeret indsats for at nå målet/målene – Forbedret teamwork – Større viden om og relation til diverse usikkerheder – Beslutninger som kan forsvares



Konklusion og perspektiv

• Ved at benytte SIMDEC tilgangen i en planlægningsproces kombineret med en beslutningskonference får man mulighed for at vurdere et beslutningsproblem på en helhedsorienteret måde

• Det fulde potentiale vil først blive tydeligt når tilgangen benyttes i en fuld beslutningskonference med deltagelse af rigtige interessenter

• Det er nødvendig at undersøge potentialet af denne tilgang videre i rigtige beslutningssituationer og teste dens brugbarhed

• Applikationer af beslutningskonferencer

– Lokalisering i Øresundsregionen (STMØ, 2008) – Udvælgelse af jernbanekorridor – Ostlänken (Vinnova, 2009) – Udvælgelse af cykelprojekter (Cykelpuljen, 2009-2010) – Vurdering af alternativer til HH-forbindelse (EcoMobility, 2011) – UNITE, 2012 (oktober)



Tak for opmærksomheden!


uncertainty in transport modelling; the four-stage næstved

Documents