lecture 4: parameter estimation methods 1: prediction...
TRANSCRIPT
![Page 1: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/1.jpg)
Lecture 4: Parameter Estimation
Methods 1
Sahar Moghimi
System Identification
![Page 2: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/2.jpg)
We have a set of models (based on the parameter vector)
…and a set of measurements
2
![Page 3: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/3.jpg)
3
Prediction error minimization
State space
Probabilistic approach
![Page 4: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/4.jpg)
Prediction error identification methods
Approach: choose parameters to make the PE as small as
possible
In order to define a scalar measure:
Linear filter: if set properly
can reduce the effect of noise
Time varying norm:
Weighted norm: 4
![Page 5: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/5.jpg)
Least squares method
Special case of prediction error methods:
5
![Page 6: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/6.jpg)
On the consistency of LSE
6
![Page 7: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/7.jpg)
7
Weighted LES: to value different measurements in
a different fashion:
![Page 8: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/8.jpg)
Dealing with the issue of best regressors
8
Weighted Principle Component Regression (PCR) Consider
The number of columns (p+1) in X should be sufficiently large to cover the whole duration of the impulse response function which can approximate the system relatively accurately.
If p+1 is greater than the number of nonzero samples in the true impulse response, then the true value of the corresponding extraneous elements in A should be zero.
Assumed uncorrelated
![Page 9: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/9.jpg)
9
We consider decomposing X into its principal components and solving the
linear regression problem in the domain of PCs using Singular Value
Decomposition (SVD).
Treat as black box: code widely available
In MATLAB: [U,D,V]=svd(X,0)
T
1
00
00
00
VU
nd
d
X
Dealing with the issue of best regressors
![Page 10: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/10.jpg)
The di are called the singular values of X
If D is singular, some of the di will be 0
In general rank(X) = number of nonzero di
The column vectors of U (principal components) are the eigenvectors of
the matrix XXT.
The column vectors of V are the eigenvectors of the matrix XTX, and
D is a diagonal matrix with its diagonal elements (singular values) being the
square root of the eigenvalues of XTX (or XXT).
Dealing with the issue of best regressors
![Page 11: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/11.jpg)
Why is SVD so useful?
Application #1: inverses
A-1=(VT)-1 W-1
U-1 = V W-1 UT
Using fact that inverse = transpose
for orthogonal matrices
Since W is diagonal, W-1 also diagonal with reciprocals of
entries of W
Dealing with the issue of best regressors
![Page 12: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/12.jpg)
A-1=(VT)-1 W-1
U-1 = V W-1 UT
This fails when some wi are 0
It’s supposed to fail – singular matrix
Pseudoinverse: if wi=0, set 1/wi to 0 (!)
“Closest” matrix to inverse
Defined for all (even non-square, singular, etc.) matrices
Equal to (ATA)-1AT if ATA invertible
Dealing with the issue of best regressors
![Page 13: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/13.jpg)
Solving Ax=b by least squares
x=pseudoinverse(A) times b
Compute pseudoinverse using SVD
Lets you see if data is singular
Even if not singular, ratio of max to min singular values
(condition number) tells you how stable the solution will be
Set 1/wi to 0 if wi is small (even if not exactly 0)
Dealing with the issue of best regressors
![Page 14: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/14.jpg)
14
If some of the diagonal elements of D are small, it means that the corresponding
PCs have small variances and consequently less information.
We rank the PCs in a descending order according to their singular values in
evaluating their contributions to the output.
We form a series of matrices U each containing a subset of PCs by adding one PC
at a time as a column vector.
The matrices U differ by the number of columns involved and their associated
regression equations (Y = UB + E ) represent the candidate models from which the
"best" model should be selected.
Note that these candidate models are data-specific.
For model selection, we employ the widely used model order selection criteria.
Dealing with the issue of best regressors
![Page 15: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/15.jpg)
Dealing with the issue of best regressors
15
With q PCs and the corresponding U, V and D matrices
denoted as 𝑈 (𝑁 × 𝑞), 𝑉 (𝑝 × 𝑞) and 𝐷 (𝑞 × 𝑞)
![Page 16: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/16.jpg)
Estimating State space models: Subspace
method
16
Estimating the state space model using LSE
If we do not have an insight into the particular structure,
for different state vectors, we can have an indefinite
number of solutions
![Page 17: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/17.jpg)
17
If we knew the sequence of state vectors:
Therefore the state and output can be estimated
using LS.
Estimating State space models: Subspace
method
![Page 18: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/18.jpg)
Estimating State space models: Subspace
method
18
If we define
![Page 19: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/19.jpg)
Estimating State space models: Subspace
method
19
K step ahead prediction based on a finite number of
measurements:
Algorithm steps:
![Page 20: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/20.jpg)
20
![Page 21: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/21.jpg)
Instrumental variable method
21
![Page 22: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/22.jpg)
22
![Page 23: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/23.jpg)
23
![Page 24: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/24.jpg)
24
![Page 25: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/25.jpg)
25
![Page 26: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/26.jpg)
26
![Page 27: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/27.jpg)
27 Slides 14-21 from: http://www.it.uu.se/edu/course/homepage/systemid/vt05
![Page 28: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/28.jpg)
28
![Page 29: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/29.jpg)
Example
29
![Page 30: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/30.jpg)
Example:
30
![Page 31: Lecture 4: Parameter Estimation Methods 1: Prediction ...s.moghimi.profcms.um.ac.ir/imagesm/1021/stories/... · The d i are called the singular values of X If D is singular, some](https://reader034.vdocuments.us/reader034/viewer/2022042107/5e86752d23474e47770594a0/html5/thumbnails/31.jpg)
31
Exercise:
Implement the LSE and IV methods on your data.
Analyze the estimation error and discuss your findings