forecasting using - rob j. hyndmanforecasting residuals residuals in forecasting: difference between...
TRANSCRIPT
![Page 1: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/1.jpg)
Forecasting using
2. The forecaster’s toolbox
OTexts.com/fpp/2/
Forecasting using R 1
Rob J Hyndman
![Page 2: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/2.jpg)
Outline
1 Some simple forecasting methods
2 Forecast residuals
3 Evaluating forecast accuracy
Forecasting using R Some simple forecasting methods 2
![Page 3: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/3.jpg)
Some simple forecasting methods
Average method
Forecast of all future values is equal to mean ofhistorical data {y1, . . . , yT}.Forecasts: yT+h|T = y = (y1 + · · ·+ yT)/T
Naïve method
Forecasts equal to last observed value.Forecasts: yT+h|T = yT.Consequence of efficient market hypothesis.
Seasonal naïve method
Forecasts equal to last value from same season.Forecasts: yT+h|T = yT+h−km where m =seasonal period and k = b(h− 1)/mc+1.
Forecasting using R Some simple forecasting methods 3
![Page 4: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/4.jpg)
Some simple forecasting methods
Average method
Forecast of all future values is equal to mean ofhistorical data {y1, . . . , yT}.Forecasts: yT+h|T = y = (y1 + · · ·+ yT)/T
Naïve method
Forecasts equal to last observed value.Forecasts: yT+h|T = yT.Consequence of efficient market hypothesis.
Seasonal naïve method
Forecasts equal to last value from same season.Forecasts: yT+h|T = yT+h−km where m =seasonal period and k = b(h− 1)/mc+1.
Forecasting using R Some simple forecasting methods 3
![Page 5: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/5.jpg)
Some simple forecasting methods
Average method
Forecast of all future values is equal to mean ofhistorical data {y1, . . . , yT}.Forecasts: yT+h|T = y = (y1 + · · ·+ yT)/T
Naïve method
Forecasts equal to last observed value.Forecasts: yT+h|T = yT.Consequence of efficient market hypothesis.
Seasonal naïve method
Forecasts equal to last value from same season.Forecasts: yT+h|T = yT+h−km where m =seasonal period and k = b(h− 1)/mc+1.
Forecasting using R Some simple forecasting methods 3
![Page 6: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/6.jpg)
Some simple forecasting methods
Forecasting using R Some simple forecasting methods 4
Forecasts for quarterly beer production
1995 2000 2005
400
450
500
Mean methodNaive methodSeasonal naive method
![Page 7: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/7.jpg)
Drift method
Forecasts equal to last value plus averagechange.
Forecasts:
yT+h|T = yT +h
T − 1
T∑t=2
(yt − yt−1)
= yT +h
T − 1(yT − y1).
Equivalent to extrapolating a line drawnbetween first and last observations.
Forecasting using R Some simple forecasting methods 5
![Page 8: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/8.jpg)
Drift method
Forecasts equal to last value plus averagechange.
Forecasts:
yT+h|T = yT +h
T − 1
T∑t=2
(yt − yt−1)
= yT +h
T − 1(yT − y1).
Equivalent to extrapolating a line drawnbetween first and last observations.
Forecasting using R Some simple forecasting methods 5
![Page 9: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/9.jpg)
Drift method
Forecasts equal to last value plus averagechange.
Forecasts:
yT+h|T = yT +h
T − 1
T∑t=2
(yt − yt−1)
= yT +h
T − 1(yT − y1).
Equivalent to extrapolating a line drawnbetween first and last observations.
Forecasting using R Some simple forecasting methods 5
![Page 10: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/10.jpg)
Some simple forecasting methods
Forecasting using R Some simple forecasting methods 6
Dow Jones Index (daily ending 15 Jul 94)
Day
0 50 100 150 200 250 300
3600
3700
3800
3900
Mean methodNaive methodDrift model
![Page 11: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/11.jpg)
Some simple forecasting methods
Mean: meanf(x, h=20)
Naive: naive(x, h=20) or rwf(x, h=20)
Seasonal naive: snaive(x, h=20)
Drift: rwf(x, drift=TRUE, h=20)
Forecasting using R Some simple forecasting methods 7
![Page 12: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/12.jpg)
Some simple forecasting methods
Mean: meanf(x, h=20)
Naive: naive(x, h=20) or rwf(x, h=20)
Seasonal naive: snaive(x, h=20)
Drift: rwf(x, drift=TRUE, h=20)
Forecasting using R Some simple forecasting methods 7
![Page 13: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/13.jpg)
Some simple forecasting methods
Mean: meanf(x, h=20)
Naive: naive(x, h=20) or rwf(x, h=20)
Seasonal naive: snaive(x, h=20)
Drift: rwf(x, drift=TRUE, h=20)
Forecasting using R Some simple forecasting methods 7
![Page 14: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/14.jpg)
Some simple forecasting methods
Mean: meanf(x, h=20)
Naive: naive(x, h=20) or rwf(x, h=20)
Seasonal naive: snaive(x, h=20)
Drift: rwf(x, drift=TRUE, h=20)
Forecasting using R Some simple forecasting methods 7
![Page 15: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/15.jpg)
Homework 1
Forecasting using R Some simple forecasting methods 8
Any questions?
![Page 16: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/16.jpg)
Outline
1 Some simple forecasting methods
2 Forecast residuals
3 Evaluating forecast accuracy
Forecasting using R Forecast residuals 9
![Page 17: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/17.jpg)
Forecasting residuals
Residuals in forecasting: difference betweenobserved value and its forecast based on allprevious observations: et = yt − yt|t−1.
Assumptions1 {et} uncorrelated. If they aren’t, then
information left in residuals that should be usedin computing forecasts.
2 {et} have mean zero. If they don’t, thenforecasts are biased.
Useful properties (for prediction intervals)3 {et} have constant variance.4 {et} are normally distributed.
Forecasting using R Forecast residuals 10
![Page 18: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/18.jpg)
Forecasting residuals
Residuals in forecasting: difference betweenobserved value and its forecast based on allprevious observations: et = yt − yt|t−1.
Assumptions1 {et} uncorrelated. If they aren’t, then
information left in residuals that should be usedin computing forecasts.
2 {et} have mean zero. If they don’t, thenforecasts are biased.
Useful properties (for prediction intervals)3 {et} have constant variance.4 {et} are normally distributed.
Forecasting using R Forecast residuals 10
![Page 19: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/19.jpg)
Forecasting residuals
Residuals in forecasting: difference betweenobserved value and its forecast based on allprevious observations: et = yt − yt|t−1.
Assumptions1 {et} uncorrelated. If they aren’t, then
information left in residuals that should be usedin computing forecasts.
2 {et} have mean zero. If they don’t, thenforecasts are biased.
Useful properties (for prediction intervals)3 {et} have constant variance.4 {et} are normally distributed.
Forecasting using R Forecast residuals 10
![Page 20: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/20.jpg)
Forecasting Dow-Jones index
Forecasting using R Forecast residuals 11
Day
Dow
−Jo
nes
inde
x
0 50 100 150 200 250 300
3600
3700
3800
3900
![Page 21: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/21.jpg)
Forecasting Dow-Jones index
Naïve forecast:
yt|t−1 = yt−1
et = yt − yt−1
Note: et are one-step-forecast residuals
Forecasting using R Forecast residuals 12
![Page 22: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/22.jpg)
Forecasting Dow-Jones index
Naïve forecast:
yt|t−1 = yt−1
et = yt − yt−1
Note: et are one-step-forecast residuals
Forecasting using R Forecast residuals 12
![Page 23: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/23.jpg)
Forecasting Dow-Jones index
Naïve forecast:
yt|t−1 = yt−1
et = yt − yt−1
Note: et are one-step-forecast residuals
Forecasting using R Forecast residuals 12
![Page 24: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/24.jpg)
Forecasting Dow-Jones index
Forecasting using R Forecast residuals 13
Day
Dow
−Jo
nes
inde
x
0 50 100 150 200 250 300
3600
3700
3800
3900
![Page 25: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/25.jpg)
Forecasting Dow-Jones index
Forecasting using R Forecast residuals 14
Day
Cha
nge
in D
ow−
Jone
s in
dex
0 50 100 150 200 250 300
−10
0−
500
50
![Page 26: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/26.jpg)
Forecasting Dow-Jones index
Forecasting using R Forecast residuals 15
Histogram of residuals
Change in Dow−Jones index
Fre
quen
cy
−100 −50 0 50
010
2030
4050
60
Normal?
![Page 27: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/27.jpg)
Outline
1 Some simple forecasting methods
2 Forecast residuals
3 Evaluating forecast accuracy
Forecasting using R Evaluating forecast accuracy 16
![Page 28: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/28.jpg)
Measures of forecast accuracy
Let yt denote the tth observation and yt|t−1 denote its forecastbased on all previous data, where t = 1, . . . , T. Then thefollowing measures are useful.
MAE = T−1T∑
t=1
|yt − yt|t−1|
MSE = T−1T∑
t=1
(yt − yt|t−1)2 RMSE =
√√√√T−1
T∑t=1
(yt − yt|t−1)2
MAPE = 100T−1T∑
t=1
|yt − yt|t−1|/|yt|
MAE, MSE, RMSE are all scale dependent.
MAPE is scale independent but is only sensible if yt � 0for all t, and y has a natural zero.
Forecasting using R Evaluating forecast accuracy 17
![Page 29: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/29.jpg)
Measures of forecast accuracy
Let yt denote the tth observation and yt|t−1 denote its forecastbased on all previous data, where t = 1, . . . , T. Then thefollowing measures are useful.
MAE = T−1T∑
t=1
|yt − yt|t−1|
MSE = T−1T∑
t=1
(yt − yt|t−1)2 RMSE =
√√√√T−1
T∑t=1
(yt − yt|t−1)2
MAPE = 100T−1T∑
t=1
|yt − yt|t−1|/|yt|
MAE, MSE, RMSE are all scale dependent.
MAPE is scale independent but is only sensible if yt � 0for all t, and y has a natural zero.
Forecasting using R Evaluating forecast accuracy 17
![Page 30: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/30.jpg)
Measures of forecast accuracy
Let yt denote the tth observation and yt|t−1 denote its forecastbased on all previous data, where t = 1, . . . , T. Then thefollowing measures are useful.
MAE = T−1T∑
t=1
|yt − yt|t−1|
MSE = T−1T∑
t=1
(yt − yt|t−1)2 RMSE =
√√√√T−1
T∑t=1
(yt − yt|t−1)2
MAPE = 100T−1T∑
t=1
|yt − yt|t−1|/|yt|
MAE, MSE, RMSE are all scale dependent.
MAPE is scale independent but is only sensible if yt � 0for all t, and y has a natural zero.
Forecasting using R Evaluating forecast accuracy 17
![Page 31: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/31.jpg)
Measures of forecast accuracy
Mean Absolute Scaled Error
MASE = T−1T∑
t=1
|yt − yt|t−1|/Q
where Q is a stable measure of the scale of the timeseries {yt}.
Forecasting using R Evaluating forecast accuracy 18
![Page 32: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/32.jpg)
Measures of forecast accuracy
Mean Absolute Scaled Error
MASE = T−1T∑
t=1
|yt − yt|t−1|/Q
where Q is a stable measure of the scale of the timeseries {yt}.
Proposed by Hyndman and Koehler (IJF, 2006)
Forecasting using R Evaluating forecast accuracy 18
![Page 33: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/33.jpg)
Measures of forecast accuracy
Mean Absolute Scaled Error
MASE = T−1T∑
t=1
|yt − yt|t−1|/Q
where Q is a stable measure of the scale of the timeseries {yt}.
For non-seasonal time series,
Q = (T − 1)−1T∑
t=2
|yt − yt−1|
works well. Then MASE is equivalent to MAE relativeto a naive method.
Forecasting using R Evaluating forecast accuracy 18
![Page 34: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/34.jpg)
Measures of forecast accuracy
Mean Absolute Scaled Error
MASE = T−1T∑
t=1
|yt − yt|t−1|/Q
where Q is a stable measure of the scale of the timeseries {yt}.
For seasonal time series,
Q = (T −m)−1T∑
t=m+1
|yt − yt−m|
works well. Then MASE is equivalent to MAE relativeto a seasonal naive method.
Forecasting using R Evaluating forecast accuracy 19
![Page 35: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/35.jpg)
Measures of forecast accuracy
Forecasting using R Evaluating forecast accuracy 20
Forecasts for quarterly beer production
1995 2000 2005
400
450
500
Mean methodNaive methodSeasonal naive method
![Page 36: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/36.jpg)
Measures of forecast accuracy
Forecasting using R Evaluating forecast accuracy 20
Forecasts for quarterly beer production
1995 2000 2005
400
450
500
Mean methodNaive methodSeasonal naive method
![Page 37: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/37.jpg)
Measures of forecast accuracy
Mean method
RMSE MAE MAPE MASE38.0145 33.7776 8.1700 2.2990
Naïve method
RMSE MAE MAPE MASE70.9065 63.9091 15.8765 4.3498
Seasonal naïve method
RMSE MAE MAPE MASE12.9685 11.2727 2.7298 0.7673
Forecasting using R Evaluating forecast accuracy 21
![Page 38: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/38.jpg)
Measures of forecast accuracy
Forecasting using R Evaluating forecast accuracy 22
Dow Jones Index (daily ending 15 Jul 94)
Day
0 50 100 150 200 250 300
3600
3700
3800
3900
Mean methodNaive methodDrift model
![Page 39: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/39.jpg)
Measures of forecast accuracy
Forecasting using R Evaluating forecast accuracy 22
Dow Jones Index (daily ending 15 Jul 94)
Day
0 50 100 150 200 250 300
3600
3700
3800
3900
Mean methodNaive methodDrift model
![Page 40: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/40.jpg)
Measures of forecast accuracy
Mean method
RMSE MAE MAPE MASE148.2357 142.4185 3.6630 8.6981
Naïve method
RMSE MAE MAPE MASE62.0285 54.4405 1.3979 3.3249
Drift model
RMSE MAE MAPE MASE53.6977 45.7274 1.1758 2.7928
Forecasting using R Evaluating forecast accuracy 23
![Page 41: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/41.jpg)
Training and test sets
Available data
Training set Test set(e.g., 80%) (e.g., 20%)
The test set must not be used for any aspect ofmodel development or calculation of forecasts.
Forecast accuracy is based only on the test set.
Forecasting using R Evaluating forecast accuracy 24
![Page 42: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/42.jpg)
Training and test sets
Available data
Training set Test set(e.g., 80%) (e.g., 20%)
The test set must not be used for any aspect ofmodel development or calculation of forecasts.
Forecast accuracy is based only on the test set.
Forecasting using R Evaluating forecast accuracy 24
![Page 43: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/43.jpg)
Training and test sets
beer3 <- window(ausbeer,start=1992,end=2005.99)beer4 <- window(ausbeer,start=2006)
fit1 <- meanf(beer3,h=20)fit2 <- rwf(beer3,h=20)
accuracy(fit1,beer4)accuracy(fit2,beer4)
In-sample accuracy (one-step forecasts)accuracy(fit1)accuracy(fit2)
Forecasting using R Evaluating forecast accuracy 25
![Page 44: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/44.jpg)
Training and test sets
beer3 <- window(ausbeer,start=1992,end=2005.99)beer4 <- window(ausbeer,start=2006)
fit1 <- meanf(beer3,h=20)fit2 <- rwf(beer3,h=20)
accuracy(fit1,beer4)accuracy(fit2,beer4)
In-sample accuracy (one-step forecasts)accuracy(fit1)accuracy(fit2)
Forecasting using R Evaluating forecast accuracy 25
![Page 45: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/45.jpg)
Beware of over-fitting
A model which fits the data well does notnecessarily forecast well.A perfect fit can always be obtained by using amodel with enough parameters. (Compare R2)Over-fitting a model to data is as bad as failingto identify the systematic pattern in the data.Problems can be overcome by measuring trueout-of-sample forecast accuracy. That is, totaldata divided into “training” set and “test” set.Training set used to estimate parameters.Forecasts are made for test set.Accuracy measures computed for errors in testset only.
Forecasting using R Evaluating forecast accuracy 26
![Page 46: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/46.jpg)
Beware of over-fitting
A model which fits the data well does notnecessarily forecast well.A perfect fit can always be obtained by using amodel with enough parameters. (Compare R2)Over-fitting a model to data is as bad as failingto identify the systematic pattern in the data.Problems can be overcome by measuring trueout-of-sample forecast accuracy. That is, totaldata divided into “training” set and “test” set.Training set used to estimate parameters.Forecasts are made for test set.Accuracy measures computed for errors in testset only.
Forecasting using R Evaluating forecast accuracy 26
![Page 47: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/47.jpg)
Beware of over-fitting
A model which fits the data well does notnecessarily forecast well.A perfect fit can always be obtained by using amodel with enough parameters. (Compare R2)Over-fitting a model to data is as bad as failingto identify the systematic pattern in the data.Problems can be overcome by measuring trueout-of-sample forecast accuracy. That is, totaldata divided into “training” set and “test” set.Training set used to estimate parameters.Forecasts are made for test set.Accuracy measures computed for errors in testset only.
Forecasting using R Evaluating forecast accuracy 26
![Page 48: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/48.jpg)
Beware of over-fitting
A model which fits the data well does notnecessarily forecast well.A perfect fit can always be obtained by using amodel with enough parameters. (Compare R2)Over-fitting a model to data is as bad as failingto identify the systematic pattern in the data.Problems can be overcome by measuring trueout-of-sample forecast accuracy. That is, totaldata divided into “training” set and “test” set.Training set used to estimate parameters.Forecasts are made for test set.Accuracy measures computed for errors in testset only.
Forecasting using R Evaluating forecast accuracy 26
![Page 49: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/49.jpg)
Beware of over-fitting
A model which fits the data well does notnecessarily forecast well.A perfect fit can always be obtained by using amodel with enough parameters. (Compare R2)Over-fitting a model to data is as bad as failingto identify the systematic pattern in the data.Problems can be overcome by measuring trueout-of-sample forecast accuracy. That is, totaldata divided into “training” set and “test” set.Training set used to estimate parameters.Forecasts are made for test set.Accuracy measures computed for errors in testset only.
Forecasting using R Evaluating forecast accuracy 26
![Page 50: Forecasting using - Rob J. HyndmanForecasting residuals Residuals in forecasting: difference between observed value and its forecast based on all previous observations: e t = y t ^y](https://reader030.vdocuments.us/reader030/viewer/2022040220/5e23f765c003766596341479/html5/thumbnails/50.jpg)
Poll: true or false?
1 Good forecast methods should have normallydistributed residuals.
2 A model with small residuals will give goodforecasts.
3 The best measure of forecast accuracy is MAPE.
4 If your model doesn’t forecast well, you shouldmake it more complicated.
5 Always choose the model with the best forecastaccuracy as measured on the test set.
Forecasting using R Evaluating forecast accuracy 27