ARMA modelsARMA models
Gloria González-RiveraUniversity of California, RiversideandJesús Gonzalo U. Carlos III de Madrid
White NoiseWhite Noise
A sequence of uncorrelated random variables is called a white noise process.
0for 0),(
)(
)0 (normally )(:2
kaaCov
aVar
aEa
ktt
at
aatt
00
01
00
01
0 0
0
ationautocorrel and anceAutocovari2
k
k
k
k
k
k
kk
k
ak
. . . .1 2 3 4 k
k
The Wold DecompositionThe Wold Decomposition
If {Zt} is a nondeterministic stationary time series, then
tic.determinis is }tV{ 5.
and ,ts,sZ of nscombinatiolinear oflimit theis ta .4
,t and s allfor 0)tV ,sa(Cov .3
,02 with ),2,0(WN is }ta{ .2
,
0j
2j and 10 .1
where
,tVta)L(tVjta
0j
jtZ
Some Remarks on the Wold DecompositionSome Remarks on the Wold Decomposition
n as 02]jta
n
0j
jtZ[E
???
0j
by mean wedo What )(
,...]2tZ,1tZ|tZ[PtZta )(
What the Wold theorem does not sayWhat the Wold theorem does not say
• The at need not be normally distributed, and hence need not be iid
• Though P[at|Zt-j]=0, it need not be true that E[at|Zt-j]=0 (think on the possible consequences???)
• The shocks a need not be the “true” shocks to the system. When will this happen???
• The uniqueness result only states that the Wold representation is the unique linear representation where the shocks are linear forecast errors. Non-linear representations, or representations in terms of non-forecast error shocks are perfectly possible.
Birth of the ARMA modelsBirth of the ARMA models
)L(p
)L(q)L(
Under general conditions the infinite lag polynomial of the WoldDecomposition can be approximated by the ratio of two finite lag polynomials:
Therefore
qtaq...1ta1taptZp...1tZ1tZ
ta)qLq...L11(tZ)pLp...L11(
ta)L(qtZ)L(p
, ta)L(p
)L(qta)L(tZ
AR(p) MA(q)
MA(1) processesMA(1) processes
Let ta a zero-mean white noise process )2a,0(ta
)1(MA1tatatZ Expectation
Variance
Autocovariance
)()()( 1ttt aEaEZE
)1()2(
)()()(22
12
122
21
2
atttt
tttt
aaaaE
aaEZEZVar
221
22
211
2111
)(
))(())(
order 1st.
attttttt
tttttt
aaaaaaaE
aaaaEZE(Z
MA(1) processes (cont)MA(1) processes (cont)Autocovariance of higher order
Autocorrelation
1 0 ) (
) )( ( ) )( (
1 12
1 1
1 1
j a a a a a a a a E
a a a a E Z Z E
j t t j t t j t t j t t
j t j t t t j t t
10
1)1( 222
2
0
11
jj
MA(1) process is covariance-stationary because22 )1()()( tt ZVarZE
MA(1) process is ergodic because
0
222 )1(j
j
If were Gaussian, then would be ergodic for all momentsta tZ
Plot the function 21 1
1-1
0.5
-0.5
1
2for 4.0
5.0for 4.0
1for 5.0 )max(
1
1
1
22121 1)/1(1
/1,
1 substitute we
1 in If
1
1
)1
( ttt
ttt
aaZ
aaZ
Both processes share the same autocorrelation function
MA(1) is not uniquely identifiable, except for 1
InvertibilityInvertibility
Definition: A MA(q) process defined by the equation
is said to be invertible if there exists a sequence of constants
and
ta)L(qtZ
|
0jj|such that }j{ ,...1,0t ,jtZ
0j
jta
Theorem: Let {Zt} be a MA(q). Then {Zt} is invertible if and only if
The coefficients {j} are determined by
the relation
1.|x|such that C xallfor 0)x(
1.|x| ,)x(
1jx
0j
j)x(
Identification of the MA(1)Identification of the MA(1)
• If we identify the MA(1) through the autocorrelation structure, we need to decide with value of to choose, the one greater than one or othe one less than one. Requiring the condition of invertibility (think why????) we will choose the value .
• Another reason to choose the value less than one can be found by paying attention to the error variance of the two “equivalent” representations:
)ta(V)ta(V
invertible-non ,)2
11(
0)ta(V , ta)L1
11(tZ
invertible ,)2
11(
0)ta(V , ta)L11(tZ
2
1
MA(q)MA(q)
qtqtttt aaaaZ 2211
Moments
011
MA(2)Example
for 0
for )(
))((
)1()var(
)(
4322
21
222
22
1
2111
1
2
2211
0
22211
1111
2222
210
k
q
ii
jqqjjjjj
jqqjjjj
qjtqjtjtqtqttj
aqt
t
qj
qj
aaaaaaE
Z
ZE
MA(q) is covariance-stationary
and ergodic for the same reasonsas in a MA(1)
MA(infinite)MA(infinite)
0j
10jtajtZ
Is it covariance-stationary?
0i
2i
0i
jii
j
0i
jii2)jtZ)(tZ(Ej
0i
2i
2a)tZ(Var,)tZ(E
The process iscovariance-stationaryprovided that
0
2
ii
(square summable sequence)
Some interesting resultsSome interesting results
Proposition 1.
Proposition 2.
0
2
0 ii
ii
(absolutelysummable)
00 ii
ii
(squaresummable)
Ergodic for the mean
Proof 1.
0
2
0 ii
ii
Nii
N
ii
Nii
N
ii
ii
Nii
Niiii
ii
i
Ni
NiN
||
Now,
1 that such If
1
0
221
0
2
0
2
22
0
(1) (2)
(1) It is finite because N is finite(2) It is finite because is absolutely summable
0
2
iithen
Proof 2.
00 ii
ii
M
MM
jji
i ii
jjii
jji
ii
j j ijiij
ijii
ijiij
ijiij
0
22
0 0
2
0
2
0 00 0
2
0
2
0
2
0
2
0
2
assumptionby because
AR(1)AR(1)
ttt aZcZ 1Using backward substitution
22
12
122
)1( ttt
tttt
aaac
aaZccZ
geometric progression )( MA
1 if 1
1)2(
sequence bounded 1
11)1(
1 if
00
2
j
j
jj
Remember:
0jj is the condition for stationarity and ergodicity
AR(1) (cont)AR(1) (cont)
Hence, this AR(1) process has a stationary solution if 1
Alternatively, consider the solution of the characteristic equation:
11
01
xx
i.e. the roots of the characteristic equation lie outside of the unit circle
Mean of a stationary AR(1)
1)(
1 22
1
cZE
aaac
Z
t
tttt
Variance of a stationary AR(1)
2
2242
0 1
11 a
Autocovariance of a stationary AR(1)
Rewrite the process as ttt aZZ )()( 1 11
1
jjttjtt
jtttjttj
ZaZZE
ZaZEZZE
11 jjj Autocorrelation of a stationary AR(1)
jjjjj
jj
o
jj j
033
22
10
1 1
ACF
PACF: from Yule-Walker equations
20
011
1
1
1
21
22
21
212
1
1
21
1
22
1̀11
kkk
Causality and StationarityCausality and Stationarity
Definition: An AR(p) process defined by the equation
is said to be causal, or a causal function of {at}, if there exists a sequence of constants
and
Causality is equivalent to the condition
tatZ)L(p
|
0jj|such that }j{ ,...1,0t ,jta
0j
jtZ
1.|x|such that C xallfor 0)x(
Definition: A stationary solution {Zt} of the equation exists (and is also the unique stationary solution) if and only if
tatZ)L(p
1.|x|such that C xallfor 0)x(
From now on we will be dealing only with causal AR models
AR(2)AR(2)
t t t ta Z Z c Z 2 2 1 1
Stationarity Study of the roots of the characteristic equation
01 221 xx
2
22
112
2
22
111
12
2
2
4
2
4
01
x
x
xx
(a) Multiply by -1 (b) Divide by 2x
2
41
2
41
0/1)/1(
22
11
2
22
11
1
212
x
x
xx
For a stationary causal solution is required that
21111
111
1
2,111
1
211
21
2221
xxxx
xx
ix
xi
i
Necessary conditions for a stationary causal solution
22
11
1
2
Roots can be real or complex.(1) Real roots
1 (2) From
1 (1) From
11
2
41
2
41
04
12
21
)1()2(
1
22
11
2
22
11
22
1
xx
(2) Complex roots
04
042
12
22
1
1
-1
1 2-1-2
12 1 12 1
4
21
2
1
2
real
complex
Mean of AR(2)
t t t ta Z Z c Z 2 2 1 1
211
c
Variance and Autocorrelations of AR(2)
2211
2
0
20220110
222110
22112
0
1
)()())(()(
a
a
a
ttttttt aZEZZEZZEZE
1))(( 2211 jZZE jjjttj
12211 jjjj
12213
22
21
2
2
11
02112
12011
3
1
1
2
1
j
j
j
Difference equation
different shapes according to the roots, real or complex
Partial autocorrelations: from Yule-Walker equations
0;1
;1 332
1
212
222
1111
AR(p)AR(p)
tptpttt aZZZcZ .......2211
stationarity All p roots of the characteristic equation outside of the unit circle
ACF
02211
2021112
112011
2211
......
......
......
......
pppp
pp
pp
pkpkkk
System to solve for the first pautocorrelations:p unknowns and p equations
ACF decays as mixture of exponentials and/or damped sine waves, Depending on real/complex roots
PACFpkkk for 0
Relationship between AR(p) and MA(q)Relationship between AR(p) and MA(q)Stationary AR(p)
)()(
1
....)1()()()(
1
)....1()()(
221
221
LL
LLLaLaL
Z
LLLLaZL
p
ttp
t
pppttp
1)()( LLp ? from obtain How to
Example
1222
113
22
12
11
12213
2112
11
312
22
321
2111
33
221
221
221
)(0
0
0
:spolynomial both from tscoefficien equating
1.............
......
......1
1.....)1)(1()2(
LL
LLL
LLL
LLLLAR
22211 jjjj
Invertible MA(q)
)()(
1
....)1()()(
1)(
)....1()()(
221
221
LL
LLLaZL
ZL
LLLLaLZ
q
ttq
t
qqqtqt
1)()( LLq
? from obtain How to
Write an example, i.e. MA(2), and proceed as in the previous example
ARMA (p,q)ARMA (p,q)
ttp
q
ttq
pt
q
tqtp
aLaL
L
aZL
LZL
xx
xx
aLZL
)()(
)(ZtionrepresentaMA Pure
)(
)()( tion representa AR Pure
10)( of roots Causal
10)( of roots ity Invertibil
)()(
t
p
Autocorrelations of ARMA(p,q)Autocorrelations of ARMA(p,q)
ktqtqkttkttktptpkttktt
qtqttptptt
ZaZaZaZZZZZZ
aaaZZZ
..........
:zero toequal mean assume ,generality of loss ofwithout
..........
1111
1111
taking expectations:
1 and on depend will
1......
ii
2211
qk
qk
k
pkpkkk
ikaZE itkt 0)(
that Note
PACFARMAMA
ARMA(1,1)ARMA(1,1)
1)((L)formMA pure
1)((L)Zform AR pure
1ityinvertibil
1 causal
)1()1(
1
1t
jaZ
ja
aLZL
jjtt
jjt
tt
ACF of ARMA(1,1)
kttkttkttktt ZaZaZZZZ 11
taking expectations
)()( 11 kttkttkk ZaEZaE
1
201
2210
21
2
2
1
)(
)()()(0
kk
a
aa
attatt
k
k
ZaEZaEk
10 and for solve
unknowns 2 and equations 2 of system
2
121
1)(
0 1
1
2
k
k
k
k
k
PACF
decays lexponentia
)1,1()1( ARMAMA
ACF
ACF and PACF of an ARMA(1,1)ACF and PACF of an ARMA(1,1)
ACF and PACF of an MA(2)ACF and PACF of an MA(2)
ACF and PACF of an AR(2)ACF and PACF of an AR(2)
ProblemsProblems
P1: Determine which of the following ARMA processes are casual and which of them are invertible (in each case at denotes a white noise):
P2: Show that the two MA(1) processes
have the same autocovariances functions.
ta2tZ81.01tZ8.1tZ .d1ta2.1ta1tZ6.0t Z.c
2tZ7.01tZ2.0ta2tZ88.01tZ9.1tZ .bta2tZ48.01tZ2.0tZ .a
1||0 where
)22 WN(0,is }t{a 1ta1
tatZ
)2 WN(0,is }t{a 1tatatZ
Problems (cont)Problems (cont)
P.3: Let {Zt} denote the unique stationary solution of the autoregressive equations
Where . Then is given by the expression
Define the new sequence
These calculations show that {Zt} is the (unique stationary) solution of the causal
AR equations
1,.... 0, t,ta1tZtZ
1|| and )2,0(WN is }ta{ jta
1j
jtZ
. and of in terms express and ) WN(0,is }W{ that show
,1tZ1
tZtW
22
W
2
Wt
1,... 0, t,tW1tZ1
tZ
Problems (cont)Problems (cont)
P4: Let Yt be the AR(1) plus noise time series defined by Yt =Zt + Wt, where
for all s and t.
• Show that {Yt} is stationary and find its autocovariance functions.
• Show that the time series is an MA(1).
• Conclude from the previous point that {Yt} is an ARMA(1,1) and express the three
parameters of this model in terms of
0)tasE(W and )2a WN(0,a ta with ta1tZt Zand ) 2
W WN(0,is }tW{
1tYtYtU
2a ,2
W ,
Appendix: Lag Operator LAppendix: Lag Operator L
Definition 1 tt ZLZ
Properties
11
1
)(.3
)(.2
.1
tttttt
ttt
kttk
YZLYLZYZL
ZLZZL
ZZL
Examples
tttttt aZLLaZZZ )1(.1 2212211
tt
tttt
tt
aZL
aLaaZ
ZLLLZLL
)1(.4
)1(.3
)1()1)(1(.2
1
2212121
Appendix: Inverse OperatorAppendix: Inverse Operator
Definition
operator)(identity )1()1( that such
).......1(lim)1(01
33221
LLL
LLLLL jjj
Note that :
1 if this definition does not hold because the limit does not exist
Example:
......
)1()1()1(
)1()1(
22
1
11
tttt
tt
tt
aaaZ
aLZLL
aZLAR
Appendix: Inverse Operator (cont)Appendix: Inverse Operator (cont)
Suppose you have the ARMA model and want to findthe MA representation . You could try to crank out directly, but that’s not much fun. Instead you could find
and matching terms in Lj to make sure this works.
ta)L(tZ)L(
ta)L(tZ
)L()L(1
)L()L()L( hence , ta)L()L(tZ)L(ta)L(
Example: Suppose .
Multiplying both polynomials and matching powers of L,
)2L2L10()L( and )L10()L(
3.j ; j01j10
...............2
10011
000
which you can easily solver recursively for the TRY IT!!!j
Appendix: Factoring Lag PolynomialsAppendix: Factoring Lag Polynomials
Suppose we need to invert the polynomial
We can do that by factoring it:
Now we need to invert each factor and multiply:
)2L2L11()L(
121
221
with)L21)(L11()2L2L11(
jL)
0j
j
0k
kj2
k1(
...L)21(1()
0j
jL
0j
j2)(jLj
1(1)L21(1)L11(
Check the last expression!!!!
Appendix: Partial Fraction TricksAppendix: Partial Fraction Tricks
There is a prettier way to express the last inversion by using the partial fraction tricks. Find the constants a and b such that
)L21)(L11(
)L21(b)L11(a
)L21(
b
)L11(
a
)L21)(L11(
1
The numerator on the right hand side must be 1, so
)j2)21(
2
0j
j1)21(
1(
)L21(
1
)21(2
)L11(
1
)21(1
)L21)(L11(
1
so
,21
1a ,12
2b
Solving,
0b1a2
1ba
Appendix: More on InvertibilityAppendix: More on Invertibility
Consider a MA(1)
t
AR
t
ttt
t
aZLLL
aaLLZ
L
aL
)(
3322
11-
1
t
).......)(1(
)1()1()(L)(1
defined is )1(,1 If
1Z
Definition
A MA process is said to be invertible if it can be written as an AR( )
• For a MA(1) to be invertible we require ]11
01[ 1
xx
• For a MA(q) to be invertible, all roots of the characteristic equation should lie outside of the unit circle• MA processes have an invertible and a non-invertible representations• Invertible representation optimal forecast depends on past information• Non-invertible representation forecast depends on the future!!!