Models for Non-Stationary Time Series

The ARIMA(p,d,q) time series

The ARIMA(p,d,q) time series

Many non-stationary time series can be converted to a stationary time series by taking dth order differences.

Let {xt|t T} denote a time series such that

{wt|t T} is an ARMA(p,q) time series where

wt = dxt

= (I – B)dxt

= the dth order differences of the series xt.

Then {xt|t T} is called an ARIMA(p,d,q) time series (an integrated auto-regressive moving average time series.)

The equation for the time series {wt|t T} is:

(B)wt = + (B)ut

or

(B) xt = + (B)ut..

Where

(B) = (B)d = (B)(I - B) d

The equation for the time series {xt|t T} is:

(B)dxt = + (B)ut

Suppose that d roots of the polynomial (x) are equal to unity then (x) can be written:

(B) = (1 - 1x - 2x2 -... - pxp)(1-x)d.

and (B) could be written:

(B) = (I - 1B - 2B2 -... - pBp)(I-B)d= (B)d.

In this case the equation for the time series becomes:

(B)xt = + (B)ut

or

(B)d xt = + (B)ut..

Comments:

1. The operator

(B) =(B)d = 1 - 1x - 2x2 -... - p+dxp+d

is called the generalized autoregressive operator. (d roots are equal to 1, the remaining p roots have |ri| > 1)

2. The operator (B) is called the autoregressive operator. (p roots with |ri| > 1)

3. The operator (B) is called moving average operator.

Example – ARIMA(1,1,1)

The equation:

(I – 1B)(I – B)xt = + (I + 1)ut

(I – (1 + 1) B + 1B2)xt = + ut + 1 ut - 1

xt – (1 + 1) xt-1 + 1xt-2 = + ut + 1 ut – 1

or

xt = (1 + 1) xt-1 – 1xt-2 + + ut + 1 ut – 1

Modeling of Seasonal Time Series

If a time series, {xt: t T},that is seasonal we would expect observations in the same season in adjacent years to have a higher auto correlation than observations that are close in time (but in different seasons.

For example for data that is monthly we would expect the autocorrelation function to look like this

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8

10

12

14

16

18

20

22

24

26

28

30

The AR(1) seasonal model

This model satisfies the equation:1 12t t tx x u

121or t tI B x u

The autocorrelation for this model can be shown to be:

1 if 12

0 elsewhere

k h kh

This model is also an AR(12) model with

1 = … = 11 = 0

Graph: (h)

0

0.2

0.4

0.6

0.8

1

1.2

0 2 4 6 8

10

12

14

16

18

20

22

24

26

28

30

The AR model with both seasonal and serial correlation

This model satisfies the equation:

121 2 t tI B I B x u

The autocorrelation for this model will satisfy the equations:

This model is also an AR(13) model.

12 131 2 2 1or t tI B B B x u

1 1 2 12 2 1 13h h h h

1 1 2 12 2 1 13or t t t t tx x x x u

The Yule-Walker Equations:1 1 2 11 2 1 12

2 1 1 2 10 2 1 11

3 1 2 2 9 2 1 10

4 1 3 2 8 2 1 9

5 1 4 2 7 2 1 8

6 1 5 2 6 2 1 7

7 1 6 2 5 2 1 6

8 1 7 2 4 2 1 5

9 1 8 2 3 2 1 4

10 1 9 2 2 2 1 3

11 1

10 2 1 2 1 2

12 1 11 2 2 1 1

13 1 12 2 1 2 1

or:

2 1 2

1 2 1 2

1 2 1 2

1 2 1 2

1 2 1 2

1 2 1 2

2 1 1 2

2 1 2 1

2 1 2 1

2 1 2 1

1 0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 0 0 0 0 0

0 1 0 0 0 0 0 0 0 0

0 0 1 0 0 0 0 0 0 0

0 0 0 1 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0 0 0

0 0 0 0 1 0 0 0 0 0

0 0 0 0 0 1 0 0 0 0

0 0 0 0 0 0 1 0 0

1 1

2

3

4

5

6

7

8

9

10

112 1 2 1

121 2 1 2

132 1 1 2

0

0

0

0

0

0

0

0

0 0

0 0 0 0 0 0 0 1 0 0 0

0 0 0 0 0 0 0 0 0 1 0

0 0 0 0 0 0 0 0 0 0 1

Some solutions for h

10.4,20.8

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

10.8,20.4

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

• Excel file for determining Autocorrelation function

The general ARIMA model incorporating seasonality

( ) ( )sd dk s st tI B I B B B x B B u

1p

pB I B B

qq qB I B B

( ) ( )1

s

s

s kps k spB I B B

1

s

s

s s s kqkqB I B B

where

Prediction

Three Important Forms of a

Non-Stationary Time Series

The Difference equation Form:

xt = 1xt-1 + 2xt-2 +... +p+dxt-p-d + + ut +1ut-1 + 2ut-2 +...+ qut-q

(B)dxt = +(B)ut

The Random Shock Form:

xt =(t) + ut+1ut-1 + 2ut-2 +3ut-3 +...

xt =(t) +(B)ut

The Inverted Form:

xt = 1xt-1 + 2xt-2 +3xt-3+ ... + + ut

(B)xt = + ut

Example

Consider the ARIMA(1,1,1) time series

(I – 0.8B)xt = (I + 0.6B)ut

(I – 0.8B) (I –B) xt = (I + 0.6B)ut

(I – 1.8B + 0.8B2) xt = (I + 0.6B)ut

xt = 1.8 xt - 1 - 0.8 xt - 2 + ut+ 0.6ut -1

Difference equation form

(I – 1.8B + 0.8B2) xt = (I + 0.6B)ut

The random shock form

xt = (I – 1.8B + 0.8B2)-1(I + 0.6B)ut

xt = (I + 2.4B + 3.52B2 +

4.416B3 + 5.1328B4 + … )ut

xt = ut + 2.4 ut - 1 + 3.52 ut - 2 + 4.416 ut - 3

+ 5.1328 ut - 4 + …

(I – 1.8B + 0.8B2) xt = (I + 0.6B)ut

The Inverted form

(I + 0.6B)-1(I – 1.8B + 0.8B2)xt = ut

(I - 2.4B + 2.24B2 – 1.344 B3 + 0.8064B4 +… )xt = ut

xt = 2.4 xt - 1 - 2.24 xt - 2 + 1.344 xt - 3 -

0.8064 xt - 4 + … + ut

Forecasting an ARIMA(p,d,q) Time Series

• Let PT denote {…, xT-2, xT-1, xT} = the “past” until time T.

• Then the optimal forecast of xT+l given PT is denoted by:

• This forecast minimizes the mean square error

TlTT PxElx ˆ

Three different forms of the forecast

1. Random Shock Form

2. Inverted Form

3. Difference Equation Form

Note:

0 if ˆ lxPxElx lTTlTT

0 if0

0 ifˆ

l

luPuElu lT

TlTT

Random Shock Form of the forecast

Recall

0 if0

0 ifˆ

l

luPuElu lT

TlTT

xt =(t) + ut+1ut-1 + 2ut-2 +3ut-3 +...

xT+l =(T + l) + uT+l +1uT+l-1 + 2uT+l-2 +3uT+l-3 +... or

Taking expectations of both sides and using

2ˆ1ˆˆˆ 21 lulululTlx TTTT

2211 TlTlTl uuulT

To compute this forecast we need to compute

{…, uT-2, uT-1, uT} from {…, xT-2, xT-1, xT}.

3322111 1ˆ tttt uuutx

1ˆ 1 ttt xxu

Note:

xt =(t) + ut +1ut-1 + 2ut-2 +3ut-3 +...

Thus

Which can be calculated recursively

and

The Error in the forecast:

112211 TllTlTlT uuuu lxxle TlTT ˆ

TTT PleElMSE 22

21111 TllTlT uuuE

221

211 l

21

211 lT l

The Mean Sqare Error in the Forecast

Hence

Prediction Limits for forecasts

(1 – )100% confidence limits for xT+l

lzlx TT 2/ˆ

The Inverted Form:

(B)xt = + ut or

xt = 1xt-1 + 2xt-2 +3x3+ ...

+ + ut

where

(B) = [(B)]-1(B) = [(B)]-1[(B)d]

= I - 1B - 2B2 - 3B3 - ...

The Inverted form of the forecast

xt = 1xt-1 + 2xt-2 +... + + ut

2ˆ1ˆˆ 21 lxlxlx TTT

and for t = T+l

xT+l = 1xT+l-1 + 2xT+l-2 + ... + + uT+l

Taking conditional Expectations

Note:

The Difference equation form of the forecast

xT+l = 1xT+l-1 + 2xT+l-2 + ... + p+dxT+l-p-d

+ + uT+l +1uT+l-1 + 2uT+l-2 +... + quT+l-q

dplxlxlxlx TdpTTT ˆ2ˆ1ˆˆ 21

qlululu TqTT ˆ1ˆˆ 1

Taking conditional Expectations

Example: ARIMA(1,1,2)The Model:

xt - xt-1 = 1(xt-1 - xt-2) + ut + 1ut + 2ut

or

xt = (1 + 1)xt-1 - 1 xt-2 + ut + 1ut + 2ut

or

(B)xt = (B)(I-B)xt = (B)ut

where

(x) = 1 - (1 + 1)x + 1x2 = (1 - 1x)(1-x) and

(x) = 1 + 1x + 2x2 .

The Random Shock form of the model:

xt =(B)ut

where

(B) = [(B)(I-B)]-1(B) = [(B)]-1(B)

i.e.

(B) [(B)] = (B).

Thus

(I + 1B + 2B2 + 3B3 + 4B4 + ... )(I - (1 + 1)B + 1B2)

= I + 1B + 2B2

Hence

1 = 1 - (1 + 1) or 1 = 1 + 1 + 1.

2 = 2 - 1(1 + 1) + 1 or 2 =1(1 + 1) - 1 + 2.

0 = h - h-1(1 + 1) + h-21

or h = h-1(1 + 1) - h-21 for h ≥ 3.

The Inverted form of the model:

(B) xt = ut

where

(B) = [(B)]-1(B)(I-B) = B)]-1(B)

i.e.

(B) [(B)] = (B).

Thus

(I - 1B - 2B2 - 3B3 - 4B4 - ... )(I + 1B + 2B2)

= I - (1 + 1)B + 1B2

Hence

-(1 + 1) = 1 - 1 or 1 = 1 + 1 + 1.

1 = -2 - 11 + 2 or 2 = -11 - 1 + 2.

0 = h - h-11 - h-22 or h = -(h-11 + h-22) for h ≥ 3.

Now suppose that 1 = 0.80, 1 = 0.60 and 2 = 0.40 then the Random Shock Form coefficients and the Inverted Form coefficients can easily be computed and are tabled below:

h 1 2 3 4 5 6 7 8 9 10

2.40 2.32 2.26 2.20 2.16 2.13 2.10 2.08 2.07 2.05 2.40 -1.84 0.14 0.65 -0.45 0.01 0.17 -0.11 0.00 0.05h 11 12 13 14 15 16 17 18 19 20 2.04 2.03 2.03 2.02 2.02 2.01 2.01 2.01 2.01 2.01 -0.03 0.00 0.01 -0.01 0.00 0.00 0.00 0.00 0.00 0.00h 21 22 23 24 25 26 27 28 29 30 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

h

h

h

h

h

h

The Forecast Equations

The Difference Form of the Forecast Equation

lulxlxlx tTTT ˆ2ˆ1ˆ1ˆ 11

0 if0

0 ifˆ and

l

lulu lT

T

2ˆ1ˆ 21 lulu tt

0 if ˆ where lxlx lTT

Computation of the Random Shock Series, One-step Forecasts

12111111ˆ ttttt uuxxx

1ˆ - 1 ttt xxu

One-step Forecasts

Random Shock Computations

221121111 ttttt uuxxx

Computation of the Mean Square Error of the Forecasts and Prediction Limits

2ˆˆ lxxMSElxMSEl TlTTT 2

lzlx TT2

2/ˆ

Mean Square Error of the Forecasts

Prediction Limits

21

23

22

21

2 1 l

Table: MSE of Forecasts to lead time l = 12 (2 = 2.56)

l 1 2 3 4 5 6 7 8 9 10 11 12

1.60 4.16 5.58 6.64 7.52 8.28 8.95 9.57 10.13 10.66 11.15 11.62 (l)T

t x x (1) u u t x x (1) u u 1 97.68 0.00 97.68 -1.11 41 149.57 149.98 -0.40 -0.402 102.99 234.42 -131.43 0.72 42 149.47 147.36 2.11 2.113 105.69 67.45 38.24 -1.54 43 150.10 150.49 -0.39 -0.394 105.00 78.22 26.78 -2.21 44 148.95 151.22 -2.27 -2.275 103.30 135.81 -32.51 0.80 45 145.57 146.51 -0.94 -0.946 100.64 93.15 7.49 -0.90 46 138.43 141.39 -2.96 -2.967 101.09 90.00 11.08 2.79 47 131.52 130.56 0.96 0.968 105.48 111.09 -5.61 2.72 48 124.64 125.39 -0.75 -0.759 108.78 110.07 -1.29 -2.97 49 120.15 119.06 1.09 1.0910 110.14 108.40 1.74 -0.58 50 117.34 116.92 0.42 0.4211 107.24 111.75 -4.51 -2.45 51 118.28 115.77 2.51 2.5112 103.71 102.91 0.80 0.49 52 120.31 120.71 -0.40 -0.4013 99.76 99.56 0.20 -0.44 53 122.63 122.69 -0.06 -0.0614 98.53 97.04 1.48 1.99 54 127.22 124.29 2.93 2.9315 97.75 98.51 -0.76 -0.81 55 129.60 132.62 -3.02 -3.0216 101.01 97.27 3.74 3.57 56 131.33 130.87 0.46 0.4617 104.49 105.56 -1.07 -0.95 57 130.48 131.79 -1.30 -1.3018 108.84 108.12 0.71 0.71 58 129.09 129.21 -0.12 -0.1219 114.21 112.32 1.89 1.85 59 130.11 127.37 2.74 2.7420 121.22 119.93 1.29 1.32 60 130.37 132.53 -2.15 -2.1521 127.95 128.36 -0.42 -0.42 61 130.54 130.39 0.15 0.1522 133.30 133.59 -0.28 -0.30 62 127.69 129.89 -2.20 -2.2023 137.23 137.26 -0.02 -0.01 63 124.37 124.16 0.20 0.2024 140.25 140.25 -0.01 0.00 64 122.42 120.95 1.48 1.4825 142.58 142.64 -0.07 -0.07 65 119.46 121.83 -2.38 -2.3826 145.97 144.40 1.57 1.58 66 117.00 116.25 0.74 0.7427 148.52 149.61 -1.09 -1.08 67 116.57 114.52 2.04 2.0428 149.90 150.54 -0.64 -0.64 68 117.97 117.75 0.23 0.2329 150.15 150.18 -0.02 -0.02 69 119.33 120.05 -0.72 -0.7230 149.94 150.09 -0.15 -0.15 70 121.91 120.07 1.85 1.8531 148.26 149.66 -1.40 -1.40 71 125.02 124.80 0.22 0.2232 147.43 146.02 1.40 1.40 72 129.80 128.37 1.43 1.4333 145.64 147.04 -1.40 -1.40 73 136.48 134.57 1.91 1.9134 143.15 143.94 -0.79 -0.79 74 143.56 143.53 0.03 0.0335 142.69 140.12 2.57 2.57 75 151.87 150.01 1.86 1.8636 146.74 143.55 3.19 3.19 76 159.38 159.65 -0.27 -0.2737 150.35 152.93 -2.57 -2.57 77 166.29 165.96 0.32 0.3238 153.64 152.98 0.67 0.67 78 173.58 171.90 1.68 1.6839 153.37 155.64 -2.27 -2.27 79 177.82 180.54 -2.73 -2.7340 152.01 152.06 -0.05 -0.05 80 182.60 180.24 2.36 2.36

t ttt tt tt ^^^^

t x x (1) u u t x x (1) 81 189.46 186.74 2.72 2.72 116 288.17 287.6582 196.45 197.53 -1.08 -1.08 117 280.04 280.1183 206.18 202.48 3.70 3.70 118 274.16 273.7084 216.82 215.76 1.06 1.06 119 269.46 269.7185 227.51 227.45 0.06 0.06 120 266.15 265.7386 239.30 236.52 2.78 2.78 121 266.90 263.6687 251.10 250.41 0.68 0.68 122 269.58 269.6188 263.66 262.06 1.60 1.60 123 272.31 273.0189 274.55 274.94 -0.39 -0.39 124 275.66 274.0590 281.21 283.67 -2.46 -2.46 125 279.90 279.0391 285.60 284.91 0.70 0.70 126 287.34 284.4792 288.97 288.55 0.42 0.42 127 298.01 295.3793 292.32 292.19 0.13 0.13 128 307.67 309.2894 295.22 295.25 -0.03 -0.03 129 317.59 315.4895 295.88 297.57 -1.69 -1.69 130 323.78 326.1596 298.67 295.38 3.29 3.29 131 328.90 328.1597 301.33 302.21 -0.88 -0.88 132 332.20 332.5098 306.55 304.24 2.31 2.31 133 332.42 334.9699 313.98 311.76 2.22 2.22 134 330.91 330.97

100 323.12 322.19 0.94 0.94 135 328.21 328.65101 329.06 331.89 -2.83 -2.83 136 326.38 325.76102 330.83 332.50 -1.67 -1.67 137 324.11 325.11103 330.49 330.11 0.38 0.38 138 323.97 321.95104 329.80 329.78 0.02 0.02 139 325.45 324.66105 326.90 329.42 -2.52 -2.52 140 325.70 327.91106 324.03 323.08 0.96 0.96 141 325.40 324.88107 318.47 321.30 -2.84 -2.84 142 324.41 324.59108 311.85 312.69 -0.84 -0.84 143 322.55 323.72109 308.86 304.93 3.93 3.93 144 316.41 320.28110 309.02 308.48 0.54 0.54 145 306.58 308.72111 311.54 311.05 0.49 0.49 146 297.83 295.88112 312.19 314.07 -1.88 -1.88 147 290.30 291.14113 310.47 311.78 -1.31 -1.31 148 285.24 284.55114 305.66 307.55 -1.89 -1.89 149 282.77 281.27115 297.09 300.15 -3.06 -3.06 150 284.11 281.97

u 0.52-0.070.46-0.250.423.24-0.03-0.711.610.872.882.64-1.622.11-2.370.75-0.30-2.53-0.06-0.440.62-1.002.010.79-2.210.52-0.18-1.17-3.86-2.141.95-0.840.691.502.14

u 0.52-0.070.46-0.250.423.24-0.03-0.711.610.872.882.64-1.622.11-2.370.75-0.30-2.53-0.06-0.440.62-1.002.010.79-2.210.52-0.18-1.17-3.86-2.141.95-0.840.691.502.14

t t t t t t tt^ ^^^

Raw Observations, One-step Ahead Forecasts, Estimated error , Error

Forecasts with 95% and 66.7% prediction Limits

lower lower upper uppert 95% Limit 66.7% Limit Forecast 66.7% Limit 95%Limit

151 283.93 285.47 287.07 288.67 290.21152 282.14 286.13 290.29 294.45 298.44153 281.94 287.29 292.87 298.45 303.80154 281.91 288.29 294.93 301.57 307.95155 281.84 289.06 296.58 304.10 311.32156 280.35 288.95 297.90 306.85 315.45157 280.21 289.39 298.96 308.53 317.71158 279.94 289.67 299.80 309.93 319.66159 279.59 289.82 300.48 311.14 321.37160 279.16 289.87 301.02 312.17 322.88161 278.67 289.83 301.45 313.07 324.23162 278.15 289.73 301.80 313.87 325.45

Graph: Forecasts with 95% and 66.7% Prediction Limits

150130250

300

350 95% and 66.7% Forecast Limits

Time

95% Limits

66.7% Limits

95% Limits

66.7% Limits