l. mentaschi, g. besio, f. cassola, a. mazzino dipartimento di ingegneria civile, chimica ed...
Post on 27-Mar-2015
232 Views
Preview:
TRANSCRIPT
L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),
Università degli studi di Genova
Validation of wave model in the Mediterranean sea
Model Wavewatch IIIArdhuin et al. (2008) source terms.Tolman and Chalikov (1996) source terms, not tuned
to the Mediterranean sea conditions.
O
OSNBI
)( Normalized Bias (NBI)
Statistical indicators used for validation
Normalized Root MeanSquare Error (NRMSE) 2
2)(
O
OSNRMSE
Correlation coeff. (ρ)OS
ii
N
OOSS
)()(
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Statistics on 17 storms and 23 buoys
ACC350 T & C
NBI 2.1% -11.2%
ρ 0.889 0.883
NRMSE 0.2864 0.2798 (-2.3%)
February 1990 storm
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
In what sense NRMSE is not completely reliable?A numerical example
ρ=0.614 for both of the simulations
NBI = -12% for the red simulation
One would say the best simulation is the blue one.
But …
NRMSE(blue) = 0.384
NRMSE(red) = 0.356 (~ -7.2%)
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Geometrical decomposition of NRMSE
2
22
22
2
2)]()[(NBI
O
ONRMSE
O
OOSSSI
Oi
ii
NBIO
OBC
O
NRMSE 22
2
222NRMSEBCSINRMSE
Bias component
Scatter component (sometimes called Scatter Index)
O
OSNBI
where
Are SI and BC independent?
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Example of set of simulations with constant ratio :
In general and are not independent.
.~ constSS
SS /
iiNBI SNBIS 0)1(
where is the unbiased simulation.iS0
S S
Let’s assume this relation:
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Holds for amplifications
Amplification factor: NBI1
Relation SI – NBI
22
22
2
2
2 2)]()[(
O
OSOS
i
ii
OO
OOSSSI
)1( NBIOS
)1(./ 0 NBIconstS SSS •
22
2200
20
2202 )(2
O
OSOSSS
OSI
•
OS ~0 0~)1(2
~ 2022
2
0
2
SIONBI
SI
O
O
NBI
SI grows linearly in NBI around NBI=0
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
NRMSE
O BCO
OSISI
2
22
0 2
11~
Fits well real world data
Significant wave height Mean period
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
NBISISI
2
11~ 0
The simulation with the minimum value of NRMSE underestimates the average value
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
How to overcome this problem?
Hanna and Heinold (1985) indicator:
SO
OS
OS
OSHH
ii
ii22 )()(
Demonstration that with ρ constantHH is minimum when bias is null
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
2)( 22222
2
OS
OS
ii
ii
OS
OS
OS
OSHH
OS ~0•
0~0
2
NBINBI
HH
02~22
22
0
2
22
O
O
NBIO
O
NBI
HH
HH has a minimumfor null bias
)1( NBIOS
)1(./ 0 NBIconstS SSS •
•
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Wavewatch III validation on the Mediterranean sea.
ACC350 T & C
NBI 0.021 -0.112
ρ 0.889 0.883
NRMSE 0.2864 0.2798 (-2.3%)
HH 0.3459 0.3634 (+4.8%)
HH has a minimum for null bias
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Future development
• NRMSE tends to assume smaller values for simulations that underestimate observations, and this is mostly evident when relation holds.
• HH indicator overcomes this problem introducing a different normalization of the root mean square error.
.~/ constSS
Conclusions
• Usage of HH indicator to validate Wavewatch III model on the Mediterranean sea.
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
Thank you!
Mentaschi et al. 2013 - Why NRMSE is not completely reliable
L. Mentaschi, G. Besio, F. Cassola, A. MazzinoDipartimento di Ingegneria Civile, Chimica ed Ambientale (DICCA),
Università degli studi di Genova
top related