lecture 20 empirical orthogonal functions and factor analysis
Post on 21-Dec-2015
220 views
TRANSCRIPT
Lecture 20
Empirical Orthogonal Functionsand
Factor Analysis
Motivation
in Fourier Analysis the choice of sine and cosine “patterns” was prescribed
by the method.
Could we use the data itself as a source of information about the shape of the
patterns?
Example
maps of some hypothetical function,say, sea surface temperatureforming a sequence in time
the data
timetime
the data
pattern number
patt
ern
impo
rtan
ce
patt
ern
impo
rtan
ce
pattern number
3
Choose just the most important patterns
3 most important patterns
comparison
original reconstruction using only 3 patterns
Note that this process has reduced the noise(since noise has no pattern common to all the images)
amplitudes of patterns
time
timeNote: no requirement that pattern is periodic in time
amplitudes of patterns
Discussion:
mixing of end members
A B
C
Useful tool for data that has three “components”
ternary diagram
B
C
100% A
75% A
50% A
25% A
0% A
works for 3 end-members, as long as A+B+C=100%
… similarly for B and C
B
C
Suppose data fall near line on diagram
A
= data
B
C
Suppose data fall near line on diagram
A
= end-members or factors
f1
f2
B
C
Suppose data fall near line on diagram
A
= end-members or factors
f1
f2
B
C
Suppose data fall near line on diagram
A
= end-members or factors
f1
f2mixing line
50%
data idealize as being on mixing line
B
C
A
f1
f2
B
C
You could represent the data exactly with a third ‘noise’ factor
A
f1
f2 f3
doesn’t much matter where you put f3, as long as it’s not on the line
S: components (A, B, C, …) in each sample, s
(A in s1) (B in s1) (C in s1)
(A in s2) (B in s2) (C in s2)
(A in s3) (B in s3) (C in s3)
…
(A in sN) (B in sN) (C in sN)
S =
Note: a sample is along a row in SN samplesM componentsS is NM
F: components (A, B, C, …) in each factor, f
(A in f1) (B in f1) (C in f1)
(A in f2) (B in f2) (C in f2)
(A in f3) (B in f3) (C in f3)
F =
M componentsM factorsF is MM
C: coefficients of the factors
(f1 in s1) (f2 in s1) (f3 in s1)
(f1 in s2) (f2 in s2) (f3 in s2)
(f1 in s3) (f2 in s3) (f3 in s3)
…
(f1 in sN) (f2 in sN) (f3 in sN)
C =
N samplesM factorsC is NM
SamplesNM
(f1 in s1) (f2 in s1) (f3 in s1)
(f1 in s2) (f2 in s2) (f3 in s2)
(f1 in s3) (f2 in s3) (f3 in s3)
…
(f1 in sN) (f2 in sN) (f3 in sN)
(A in s1) (B in s1) (C in s1)
(A in s2) (B in s2) (C in s2)
(A in s3) (B in s3) (C in s3)
…
(A in sN) (B in sN) (C in sN)
=
(A in f1) (B in f1) (C in f1)
(A in f2) (B in f2) (C in f2)
(A in f3) (B in f3) (C in f3)
S = C F
Coefficients NM
Factors MM
SamplesNM
(f1 in s1) (f2 in s1)
(f1 in s2) (f2 in s2)
(f1 in s3) (f2 in s3)
…
(f1 in sN) (f2 in sN)
(A in s1) (B in s1) (C in s1)
(A in s2) (B in s2) (C in s2)
(A in s3) (B in s3) (C in s3)
…
(A in sN) (B in sN) (C in sN)
=
(A in f1) (B in f1) (C in f1)
(A in f2) (B in f2) (C in f2)
S C’ F’
selectedcoefficients
Np
selectedfactors pM
ignore f3
igno
re f
3
data approximated with only most important factors
p most important factors = those with the biggest coefficients
view samples as vectors in space
A
B
C
s1 s2s3
f
Let the factors be unit vectors …
… then the coefficients are the projections (dot products) of the sample
onto the factors
Suggests a method of choosing factors so that they have large coefficients:
A
B
C
s1 s2s3
f
Find the factor f that maximizes
E = i [ si f ]2
with the constraint that f f =1
Note: square the dot product since it can be
negative
Find the factor f that maximizesE = i [ si f ]2 with the constraint that L = f f – 1 = 0
E = i [ si f ]2 = i [j Sij fj] [k Sik fk] = j k [i Sij Sik] fj fk
= j k Mjk fj fk with Mjk= i Sij Sik or M=STS
L = i fi2 – 1
Use Lagrange Multipliers, minimizing =E-2L, where 2 is the Lagrange Multiplier. We solved this problem 2 lectures ago. It’s solution is the algebraic eigenvalue problemMf = 2 f. Recall that the eigenvalue is the corresponding value of E.
symmetricWrite as square for reasons that will become apparent later
So factors solve the algebraic eigenvalue problem:
[STS] f = 2 f.
[STS] is a square matrix with the same number of rows and columns as there are components. So there are as many factors as there are components. The factors must span a space of the same dimension as the components.
If you sort the eigenvectors by the size of their eigenvectors, then the ones with the largest eigenvalue have the largest components. So selecting the most important factors is easy.
An important tidbit from the theory of eigenvalues and eigenvectors that we’ll use later on …
[STS] f = 2 f.
Let2 be a diagonal matrix of eigenvalues, i
2
and letV be a matrix whose columns are the corresponding
factors, f(i)
Then
[STS] = V 2 VT
Note also that the factors are orthogonal
f(i) f(j) = 0 if ij
This is a mathematically pleasant property
But it may not always be the physically most-relevant choice
B
C
A
f1
B
C
Af2
not orthogonal orthogonal
f1
f2
contains negative A
close to mean of data
Upshot
eigenvectors of [STS] f = 2 f with the p eigenvalues
identify a p-dimensional sub-spacein which most of the data lie
you can use those eigenvectors as factors
Or
You can chose any other p factors that span that subspace In the ternary diagram example, they must lie on the line connecting the two SVD factors
Singular Value Decomposition (SVD)
Any NM matrix S and be written as the product of three matrices
S = U VT
where U is NN and satisfies UTU = UUT
V is MM and satisfies VTV = VVT
and is an NM diagonal matrix of singular values
Now note that it
S = U VT
then
STS = [U VT]T [U VT] = V UTU VT = V 2VT
Compare with the tidbit mentioned earlier STS=V2VT
The SVD V is the same V we were talking about earlierThe columns of V are the eigenvectors f, so
F = VT
So we can use the SVD to calculate the factors, F
But its even better than that! Write
S = U VT
as
S = U VT = [U ] [VT] = C F
So the coefficients are C = U
and, as shown previously, the factors are
F = VT
So we can use the SVD to calculate the coefficients, C, and the factors, F
MatLab Codefor computing C and F
[U,LAMBDA,V] = svd(S);
C = U*LAMBDA;
F = V’;
MatLab Codeapproximating SSp using only the p most
important factors
p = (whatever);Up=U(:,1:p);LAMBDAp=LAMBDA(1:p,1:p);Cp = Up*LAMBDAp;Vp = V(:,1:p);Fp = (Vp)’;Sp = Cp * Fp;
back to my example
Each pixel is a component of the imageand the patters are factors
our derivation assumed that the data (samples, s(i)) were vectors
However, in this example, the data are images (matrices)
so what I had to do was to write out the pixels of each image as a vector
Steps
1) load images2) reorganize images into S3) SVD of S to get U and V4) Examine to identify number of significant factors5) Build S’, using only significant factors6) reorganize S’ back into images
MatLab code for reorganizing a sequence of imagesD(p,q,r) (p=1 …Nx) (q=1 …Nx) (r=1 …Nt)
into the sample matrix, S(r,s) (r=1 …Nt) (q=1 …Nx2)
for r = [1:Nt] % time rfor p = [1:Nx] % row pfor q = [1:Nx] % col q s = Nx*(p-1)+q; % index s S(r,s) = D(p,q,r);endendend
MatLab code for reorganizing the sample matrixS(r,s) (r=1 …Nt) (s=1 …Nx
2) back into a sequence of images
D(p,q,r) (p=1 …Nx) (q=1 …Nx) (r=1 …Nt)
for r = [1:Nt] % time pfor s = [1:Nx*Nx] % index s p = floor( (s-1)/Nx+0.01 ) + 1; % row p q = s - Nx*(p-1); % col q D(p,q,r) = S(r,s);endend
Reality of Factorsare factors intrinsically meaningful, or just a convenient way of
representing data?
Example:Suppose the samples are rocksand the components are element concentrationsthenthinking of the factors as minerals might make intuitive sense
Minerals: fixed element composition
Rock: mixture of minerals
Many rocks – but just a few minerals
mineral (factor) 1mineral (factor) 2mineral (factor) 3
rock 1 rock 2rock 3
rock 4rock 5 rock 6 rock 7
Possibly Desirable Properties of Factors
Factors are unlike each otherdifferent minerals typically contain different elements
Factor contains either large or near-zero componentsa mineral typically contains only a few elements
Factors have only positive componentsminerals composed of positive amount of chemical elements
Coefficient of factors are positive rocks composed of positive amount of minerals
Coefficient typically either large or near-zero rocks composed of just a few major minerals
Transformations of Factors
S = C F
Suppose we mix factors together to get new factors set of factors
New FactorsMM
(f1 in f’1) (f2 in f’1) (f3 in f’1)
(f1 in f’s2) (f2 in f’2) (f3 in f’2)
(f1 in f’3) (f2 in f’3) (f3 in f’3)=
(A in f1) (B in f1) (C in f1)
(A in f2) (B in f2) (C in f2)
(A in f3) (B in f3) (C in f3)
Transformation MM
Old Factors MM
(A in f’1) (B in f’1) (C in f’1)
(A in f’2) (B in f’2) (C in f’2)
(A in f’3) (B in f’3) (C in f’3)
Fnew = T Fold
Transformations of FactorsFnew = T Fold
A requirement is that T-1 exists, else Fnew will not span the same space as Fold
S = C F = C I F = (C T-1) (T F)= Cnew Fnew
So you could try to implement the desirable factors by designing an appropriate transformation matrix, T
A somewhat restrictive choice of T is T=R, where R is a rotation matrix
(rotation matrices satisfy R-1=RT)
A method for implementing this property
Factors are unlike each otherdifferent minerals typically contain different elements
Factor contains either large or near-zero componentsa mineral typically contains only a few elements
Factors have only positive componentsminerals composed of positive amount of chemical elements
Coefficient of factors are positive rocks composed of positive amount of minerals
Coefficient typically either large or near-zero rocks composed of just a few major minerals
Factor contains either large or near-zero components
More-or-less equivalent to
Lots of variance in the amounts of components contained in the factor
Usual formula for variance for data, x
d2 = N-2 [ N ixi
2 - (i xi)2 ]
Application to factor, f
f2 = N-2 [ N ifi
4 - (i fi2)2 ]
Note that we are measuring the variance of the squares of the elements of , f. Thus a factor has large f
2 if the absolute-value of its elements has a lot of variation. The sign of the elements is irrelevant.
Varimax Factors
Procedure for maximizing the variance of the factors
while still preserving their orthogonality
Based on rotating pairs of factorsin their plane
f1oldf2
old
f1new
f2new
f1
f2
f3
f4
f1
cos()f2 + sin()f3
f4
-sin()f2 + cos()f3= R
rotating a pair of factors in their plane by an amount
1 0 0 0 0 cos() sin() 00 -sin() cos() 00 0 0 1
R = Called a Givens rotation, by the way
Varimax Procedure
for a pair of factors fs and ft
find that maximizes the sum of their variances
f’s2+f’t
2) = Nif’is4-(i f’i
s2)2+Nif’it4-(i f’i
t2)2
where fi’s = cos( fi
s + sin( fit
where fi’t = -sin( fi
s + cos( fit
Just solve dE/d = 0
After much algebra
= ¼ tan-1
2Ni uivi – iui ivi
where ui = fis2 - fi
t2 and vi = 2 fis2 fi
t2
Ni (ui2-vi
2) – (iui)2 (ivi)2
Then just apply this rotation to every pair of factors*
the result is a new set of factor that are mutually orthogonal
but that have maximal variance
hence the name Varimax
*Actually, you need to do the whole procedure multiple times to get convergence, since subsequent rotations to some extent undo the work of previous rotations
Example 1fs = [ ½, ½, ½, ½ ]T and ft = [ ½, -½, -½, -½ ]T
= 45°f’s = [ 1/2, 0, 1/2, 0 ]T and f’t = [ 0, -1/2, 0, - 1/2 ]T
rotation angle,
fs2 +
ft
2
sum
of
vari
ance
s
°
worst case: zero variance
Example 2fs = [0.63, 0.31, 0.63, 0.31]T ft = [0.31, - 0.63, 0.31, -0.63]T
= 26.56°
fs = [0.71, 0.00, 0.71, 0.00]T ft = [0.00, -0.71, 0.00, -0.71]T
rotation angle,
fs2 +
ft
2
sum
of
vari
ance
s
°