investigation in mathematics singular value decomposition value...abstract this investigation looks...

Investigation in MathematicsSingular Value Decomposition

Rebekah Wheeler10214927

28th March 2013

1

Abstract

This investigation looks into the application of Singular Value de-composition and it’s use as a tool for image compression. The equationM = UΣVT is the underlying basis for the investigation. It concludesthat SVD is a reliable way of compressing any size matrix with realentries, and so can be applied to images effectively.

2

Contents

1 Introduction 4

2 Singular Value Decomposition 52.1 What is Singular Value Decomposition? . . . . . . . . . . . . . 52.2 What are Singular Values? . . . . . . . . . . . . . . . . . . . . 52.3 How are the other matrices defined? . . . . . . . . . . . . . . . 72.4 How is SVD used in compression? . . . . . . . . . . . . . . . . 92.5 Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 92.6 Required storage space for a matrix . . . . . . . . . . . . . . . 102.7 Condition of a Matrix . . . . . . . . . . . . . . . . . . . . . . 11

3 SVD Examples 123.1 A 2× 2 Matrix SVD Example . . . . . . . . . . . . . . . . . . 123.2 A 4× 3 Matrix SVD Example using MATLAB . . . . . . . . . 16

4 Image Compression 184.1 An Introduction to using SVD within Image Compression . . . 184.2 A Basic SVD Image Compression Example . . . . . . . . . . . 184.3 Image Compression using SVD . . . . . . . . . . . . . . . . . 234.4 Applying SVD to colour images . . . . . . . . . . . . . . . . . 30

5 Other Applications of SVD 39

6 Conclusion 40

7 Appendix 417.1 MATLAB commands for section (3.2) . . . . . . . . . . . . . . 417.2 MATLAB commands for section (4.2) . . . . . . . . . . . . . . 427.3 MATLAB commands for section (4.3) . . . . . . . . . . . . . . 447.4 MATLAB commands for section (4.4) . . . . . . . . . . . . . . 46

8 Glossary 49

3

1 Introduction

The method of Singular Value Decomposition (SVD) has many different usesand applications which can be very valuable. SVD involves the decomposi-tion of a matrix into a set of three multiplicative matrices which containa smaller amount of information, but can still produce the original matrixwhen multiplied together. This application is very useful as it can be appliedto any size matrix. Where larger or more complicated matrices are involvedit will be beneficial to include the use of MATLAB.

In a sense, SVD can be used to abstract the most relevant and impor-tant data, removing any slight variations which won’t matter to the overallpicture, but will reduce the amount of data it uses significantly. It takesthe closest approximation to the required degree of accuracy. One analogyof this is looking at a regression line (or ‘line of best fit’). It takes the bestapproximation of all the points on the graph and fits the most appropriateline. The equation of this line is a lot simpler than storing the equationsto all the smaller lines if you were to connect the dots, whilst the points ofinterest have been preserved [1].

A very interesting and useful application of SVD is image compression.Images are often very detailed and require a large amount of ‘space to bestored on a computer. To reduce the amount of storage space the imagetakes up, SVD can be used. It compresses the image to the required degree ofaccuracy, taking out detail which is often unnoticeable. This can be beneficialfor computational efficiency, especially when the images are needed to betransferred for example by email.

4

2 Singular Value Decomposition

2.1 What is Singular Value Decomposition?

Singular Value Decomposition is an application of linear algebra in whichmatrices are factorised. The entries of the matrices can be of real or complexform and the matrix can be of any size, which implies that it’s uses are vast.

This investigation will explore the application of SVD on matrices withreal entries.

SVD starts by decomposing or factorising a two dimensional matrix ofsize m× n (where m and n are real numbers) into the following form:

M = UΣVT (1)

Where:M is a m× n matrixU is a m×m orthogonal matrixΣ is a m× n diagonal like matrix, with possible non-zero terms along

the diagonalVT is the transpose of an n× n orthogonal matrix

Hence the matrix M is broken into three matrices. The application of thiswill come later.

2.2 What are Singular Values?

From it’s name, ‘Singular Value Decomposition’ relies upon singular values.To understand SVD we must first understand the idea of singular values.

Singular values of any m× n matrix M are defined as the square root ofthe eigenvalues of the matrix A where:

A = MTM (2)

5

Matrix A has some special properties. It is a square, symmetric matrix.

Since matrix M has not been confined to a square matrix, and eigenvaluescan only be found of square matrices, we must look at the matrix multipli-cation MTM, where MT is the transpose of the matrix M.

If matrix M is a m × n sized matrix then MT is of size n × m. Whenmultiplied, A = MTM is a m×m sized matrix, hence A is a square matrix.

Now, any matrix L is defined to be symmetric if LT = L [16]. We cantherefore prove that the matrix A from equation (2) is symmetric [17]:

AT = (MTM)T

= (MT )(MT )T

= MTM= A

The benefits of A being an n × n square, symmetric matrix is that weknow it has n real eigenvalues and therefore all of it’s singular values can befound [18].

From the decomposition equation (1), matrix M is a m × n matrix. Tofind it’s singular values we must first find the eigenvalues of the square sym-metric matrix A.

An eigenvalue is a scalar, and each eigenvalue has a corresponding eigen-vector. A = MTM is the square matrix, I is the identity matrix, λ is theeigenvalue and v is the eigenvector, the following equation holds[2]:

(A− λI)v = 0 (3)

And for this to have a non trivial solution (i.e. v 6= 0), the determinantis equated to zero:

det(A− λI) = 0

Hence, the eigenvalues can be found, and thus the singular values (σn) ofthe matrix can also be found. The singular values are the square roots of theeigenvalues of the MTM matrix [3]:

6

σn =√λn (4)

These singular values need to be ordered. Matrix M of size m× n has rnumber of singular values (r ≤ n where r is the number of non zero singularvalues or rank of matrix, see section (2.5)). These are ordered as follows [4]:

σ1 ≥ σ2 ≥ ... ≥ σr > 0 (5)

These ordered singular values are then used to make up the ‘Σ’ matrixfrom equation (1):

Σ =

σ1 0 ... 0

... 0

0 σ2 0...

... 0... 0

. . . 0... 0

0 ... 0 σr... 0

(6)

It is important that this matrix is of the correct size (m × n), to enablethe matrices to be multiplied, hence zero rows may be added where necessary.

2.3 How are the other matrices defined?

So far, from the SVD equation (1) (M = UΣVT ), we can see that ‘Σ’ ismade up of the singular vales of matrix ‘M’. The other two matrices, ‘U’and ‘VT ’ are orthogonal matrices of size m×m and n× n respectively.

[Orthogonal matrices are useful as their inverse is equal to their trans-pose. This becomes helpful later.]

The corresponding eigenvectors to the eigenvalues which have been pre-viously found make up the matrix ‘V’ and the transpose of this is taken tofind ‘VT ’, which is needed in equation (1).

The eigenvectors corresponding to the eigenvalues which have alreadybeen found are calculated by substituting the eigenvalues back into the matrixfound in equation (3) [2]:

(A− λI)v = 0

7

a11 ... an1...

. . ....

a1n ... ann

− λ

1 0 0

0. . . 0

0 0 1

v1...vn

=

0...0

These eigenvectors must be normalised, to give a magnitude of 1, so thatwhen V is found, it is an orthogonal matrix. To normalise a vector, all ofthe entries are divided by the magnitude of the vector.

Once the normalised eigenvectors for all λn have been found, matrix Vcan be set up. The normalised eigenvectors must be introduced to the matrixin the same order that the eigenvalues are ordered (see equation (5)):

V =

v11 ... vn1...

. . ....

v1n ... vnn

(7)

And the transpose can be found easily:

VT =

v11 ... v1n...

. . ....

vn1 ... vnn

(8)

Now that VT and Σ have been found, the remaining matrix U must befound.

From the definition given in section 2.2 we know that both U and VT

are orthogonal. By definition, the transpose of an orthogonal matrix is thesame as its inverse [5]. This allows some rearranging of the original equation(1) to gain U as follows:

M = UΣVT

⇔M = UΣV−1

⇔MV = UΣ

⇔ U = MVΣ−1 (9)

8

From this we know that to find U we need to know what matrices V, Mand also the matrix Σ−1 are. Σ isn’t defined as an orthogonal matrix so wecan’t just take the transpose, it is however a diagonal matrix which makesthe inverse easy to find [6]:

Σ =

s1 0 0

0. . . 0

0 0 sn

(10)

Σ−1 =

1s1

0 0

0. . . 0

0 0 1sn

(11)

So we can use this to find U and thus matrix M has been decomposedinto 3 matrices.

2.4 How is SVD used in compression?

The main use of SVD is enabling the easy compression of matrices so thatthe space they require to be stored is reduced. SVD decomposes matrix Minto three matrices, U, Σ and VT . Matrix Σ contains the singular values ofmatrix M along it’s diagonal. To reduce the storage requirements the num-ber of singular values, or the rank (see section (2.5)), needs to be reducedto the required degree of accuracy. Limitations on this follow in section (2.6).

The smallest singular values, to the required compression rate, are re-placed with zeros. This reduces the storage requirements whilst preserv-ing the size of the matrix, enabling it still to be used in equation (1):M = UΣVT .

2.5 Rank of a Matrix

The rank of the matrix is very important in SVD as it gives us a guide to mea-suring the storage capacity required for the matrix, which in turn suggeststhe accuracy of the matrix once it has been decomposed. It is a measure to

9

the number of terms which the matrix is made up of, which allows matricesto be compared against each other.

The rank is a measurement of the number of linearly independent rowswithin the matrix or the number or non zero singular values. If the matrixhas any zero singular values, these do not show in the decomposition of ma-trix M and therefore they do not count towards the rank [7].

We can therefore state that, where M is an m× n matrix [19]:

rank(M) ≤ min(m,n)

2.6 Required storage space for a matrix

Storage space required by a matrix is closely linked to it’s rank. For anuncompressed matrix, M of size m × n where rank(M) = min(m,n) it’srequired storage space, z is [11]:

zM = mn (12)

For a matrix, M whose SVD has been approximated to k singular valuesi.e. rank(M) = k it’s required storage space, z is:

zM = k(m+ n+ 1) (13)

There must be some limits put into place on the rank, k which M can bedecomposed to as the approximated matrix can’t take up more storage spacethan the original matrix.

Taking this into account we can equate the above equations (12, 13) tofind out the maximum rank, k which M can be decomposed to [20]

k(m+ n+ 1) < mn

k < mnm+n+1

10

2.7 Condition of a Matrix

The condition number of a matrix is a very useful way of measuring theworst/ biggest change in the rank of a matrix, or the amount of error fromone matrix to another. It is the measure of the ratio between the smallestand largest singular values of a matrix. The larger the condition number thegreater the error, or the more ‘ill-conditioned’ the matrix is [8]. It can beworked out as follows [9]:

Let the maximum and minimum singular values of a matrix M be σ1(M)and σr(M) respectively, and let the condition number be denoted as ‘C’.

C(M) =σ1(M)

σr(M)(14)

Singular matrices, or matrices which are ‘nearly’ singular have infinitecondition numbers as their lowest singular values are or are close to zero.

If a matrix is non-singular then the condition number measures how closeto being singular it is [21]. The closer a matrix is to being singular, thecloser it’s condition number is to infinity and the closer a matrix is to beingnon-singular the closer it’s condition number is to 1.

11

3 SVD Examples

3.1 A 2× 2 Matrix SVD Example

By hand calculation, apply SVD to the following 2× 2 matrix:

M =

(1 23 4

)

Begin by finding MT and A = MTM:

MT =

(1 32 4

)

A = MTM =

(1 23 4

)(1 32 4

)=

(5 1111 25

)

Now find the eigenvalues (λn) of A:

det

(5− λ 11

11 25− λ

)= 0

⇔ (5− λ)(25− λ)− 121 = 0⇔ λ2 − 30λ+ 4 = 0

⇔ λ1 = 29.87, λ2 = 0.134

And so the singular values (σr) are:

σ1 =√λ1 =

√29.87 = 5.46

σ2 =√λ2 =

√0.134 = 0.366

Which allows us to set up the matrix Σ:

Σ =

(5.46 0

0 0.366

)

Now to find the corresponding eigenvectors (vn) to the eigenvalues (λn):

(λ1 = 29.87) (5− 29.87 11

11 25− 29.87

)(v1v2

)=

(00

)

⇔(−24.87 11

11 −4.87

)(v1v2

)=

(00

)

⇔(−24.87v1 + 11v2

11v1 − 4.87v2

)=

(00

)

12

This can be written:

v1 =

{−24.87v1 + 11v2 = 0,

11v1 − 4.87v2 = 0

v1 =

{24.87v1 = 11v2,

11v1 = 4.87v2

These are equivalent equations, so vector v1 is given by:

v1 =

(1

4.8711

)=

(1

0.443

)

Normalising:

| v1 |=√

12 + 0.4432 = 1.094

v1 =

(0.4050.915

)

Now finding the second eigenvector, v2:

(λ2 = 0.134) (5− 0.134 11

11 25− 0.134

)(v1v2

)=

(00

)

⇔(

4.866 1111 24.866

)(v1v2

)=

(00

)

⇔(

4.866v1 + 11v211v1 + 24.866v2

)=

(00

)

This can be written:

v2 =

{4.866v1 + 11v2 = 0,

11v1 + 24.866v2 = 0

v2 =

{4.866v1 = −11v2,

11v1 = −24.866v2

13

These are equivalent equations, so vector v2 is given by:

v2 =

(1

−4.86611

)=

(1

−0.442

)

Normalising:

| v2 |=√

12 + 0.4422 = 1.093

v2 =

(−0.9150.405

)

We are now able to put together the matrix V:

V =

(0.405 −0.9150.915 0.405

)

And the transpose of V is:

VT =

(0.405 0.915−0.915 0.405

)

To gain U we must first find the matrix Σ−1 by finding the reciprocals of thediagonal entries:

Σ =

(5.46 0

0 0.366

)

Σ−1 =

(0.183 0

0 2.732

)

Now we can find U:

U = MVΣ−1

U =

(1 32 4

)(0.405 −0.9150.915 0.405

)(0.183 0

0 2.732

)

=

(3.15 0.34.47 −0.21

)(0.183 0

0 2.732

)

=

(0.57 0.810.81 −0.57

)

So we have now used SVD to decompose matrix M:

14

M = UΣVT

=

(0.57 0.810.81 −0.57

)(5.46 0

0 0.366

)(0.405 0.915−0.915 0.405

)

And checking this:

=

(3.1 0.34.4 −0.2

)(0.405 0.915−0.915 0.405

)

=

(1 32 4

)

Here it can be seen that SVD has been applied successfully to the originalmatrix.

It may also be necessary to know the rank and condition number of thematrix. We already know that the matrix M has 2 singular values (5.46,0.366), and hence the rank of the matrix is 2. The condition number can befound by dividing these:

C(M) = 5.460.366

= 14.92

The condition number shows that the matrix is close to being non-singular.

Also, the storage space required to save this matrix to a computer is foundby:

zM = 2× 2 = 4 storage spaces

This is a very small amount, and so compression of this matrix wouldn’t benecessary. It will however become necessary when we start introducing largermatrices and images.

This is quite a lengthy process, even for a 2×2 matrix, so from this pointthe use of MATLAB is essential to carry out matrix multiplications.

15

3.2 A 4× 3 Matrix SVD Example using MATLAB

(All commands for this section can be found in the appendix, section 7.1)

We have seen in the previous example that SVD can be applied to a squarematrix to reach 3 multiplicative matrices, which when multiplied togetherwill return the original matrix. SVD can also be applied to larger, non-square matrices, an example of which follows.

Applying SVD to the following 4× 3 matrix:

M =

1 5 92 6 103 7 114 8 12

Using the MATLAB SVD command, the following matrix decomposition

can easily be found ([10]):

M = UΣVT

U =

−0.4036 −0.7329 −0.2819 −0.4696−0.4647 −0.2898 0.7259 0.4160−0.5259 0.1532 −0.6060 0.5768−0.5870 0.5962 0.1620 −0.5232

Σ =

25.4368 0 0

0 1.7226 00 0 00 0 0

V =

−0.2067 0.8892 −0.4082−0.5183 0.2544 0.8165−0.8298 −0.3804 −0.4082

And using MATLAB to find the transpose of V:

VT =

−0.2067 −0.5183 −0.82980.8892 0.2544 −0.3804−0.4082 0.8165 −0.4082

And again, when checking this by multiplying U, Σ and VT together, the

original matrix M is found.

16

MATLAB can also be used to find the rank of the matrix:

rank(M) = 2

i.e. matrix M has 2 singular values (or linearly independent rows).

The condition number (C) of the matrix M can be found by dividing thelargest singular value(25.4368) by the smallest singular value (1.7226):

C(M) = 14.7665

We know from the definition of a condition number that the closer it is to1 the less there is variation within the matrix. In this case, the conditionnumber isn’t too large hence the matrix isn’t far from being non-singular.

The storage space required to store matrix M on a computer is given by:

zM = 4× 3 = 12 storage spaces

Again, this is a very small amount, so compression wouldn’t be necessaryhere.

17

4 Image Compression

4.1 An Introduction to using SVD within Image Com-pression

There is a great need for compression of information to enable the reductionof storage requirements. Singular value decomposition is an effective wayof minimising the amount of storage space required by images. It removesunnecessary detail but still preserves the important information.

Computers store some types of images as matrices. Within grayscale im-ages each shade of grey, ranging from black to white is assigned a number.When analysed, these numbers of the matrix will form an image. Essentially,SVD can be used to minimise the amount of different numbers (or shades)used to make up an image by minimising the number of singular values whichthe corresponding matrix has [11].

An image, M which is uncompressed and of size m × n pixels requireszM = mn amount of space. Once SVD has been applied to the image it takesup zM = k(m + n + 1) (recall: k is the rank of matrix M). This shows thatas SVD is used to reduce the rank of the matrix, the storage requirementsreduce also (see section (2.6)).

4.2 A Basic SVD Image Compression Example

(All Matlab commands for this section can be found in section (7.2))

Take the following image:

18

Figure 1: A black and white rectangle

If we were to define the white pixels with the number 0 and the blackpixels with the number 1, we can represent figure 1 in matrix form as follows:

N =

0 0 00 1 00 1 00 1 00 0 0

We can see that this matrix has only 1 linearly independent rows, and so

it has a rank of 1. We cannot have a rank of less than this without having azero matrix, hence SVD wouldn’t be beneficial here.

Instead, lets look at the following image:

Figure 2: A black, grey and white rectangle

19

We will define the black and white pixels as before, and we will define thegrey pixels with the number 0.5. The matrix representation of figure 4 is asfollows:

M =

0.5 0 0.50 1 00 1 0.5

0.5 1 00 0.5 0

This matrix has a rank of 3, as it has 3 linearly independent rows.

We can determine the maximum rank which will be beneficial in thecompression of this image by equation (2.6):

k < mnm+n+1

< 5×33+5+1

< 159

< 1.6

This tells us that to compress the original image, the rank of the matrixmust be below 1.6. We will show, however what happens to matrix M if wetry to compress it to a rank of 2, and compare this against compression witha rank of 1.

Using the MATLAB SVD command [14], matrix M can be decomposedinto the following matrices:

U =

−0.9071 0.9106 0 −0.3948 0.0724−0.5217 −0.3390 0 −0.7135 −0.322−0.5702 0.1163 0.7071 0.3948 −0.0734−0.5702 0.1163 −0.7071 0.3948 −0.0734−0.2608 −0.1695 0 −0.1523 0.9381

Σ =

1.8537 0 0

0 0.7509 00 0 0.50 0 00 0 0

20

VT =

−0.1800 −0.9671 −0.18000.6838 −0.2546 0.6838−0.7071 0 0.7071

We know that matrix M has a rank of 3 (i.e. it has 3 singular values).

To begin with we will change this to a rank of 2 (i.e. 2 singular values). Todo this we will choose to keep the 2 largest singular values (i.e. The 3 singu-lar vales are 1.8537, 0.7509 and 0.5. We will discard 0.5 and replace it with 0.)

Our new matrix decomposition becomes:

U =

−0.9071 0.9106 0 −0.3948 0.0724−0.5217 −0.3390 0 −0.7135 −0.322−0.5702 0.1163 0.7071 0.3948 −0.0734−0.5702 0.1163 −0.7071 0.3948 −0.0734−0.2608 −0.1695 0 −0.1523 0.9381

Σ =

1.8537 0 0

0 0.7509 00 0 00 0 00 0 0

VT =

−0.1800 −0.9671 −0.18000.6838 −0.2546 0.6838−0.7071 0 0.7071

And when we multiply these together, we get the following matrix:

M2 =

0.5 0 0.50 1 0

0.25 1 0.250.25 1 0.25

0 0.5 0

Which, when put into image form is the following:

Predictably, this matrix looks more complex than the original, and hasmore pixels to store, even though the number of singular values has beenreduced. We will look at the storage requirements later in the section.

21

Figure 3: Figure 4 with a rank of 2

Now if we were to take out the 2 smallest singular values, we would havea rank of 1 (i.e. The only singular value is 1.8537).

The matrix decomposition is as follows:

U =

−0.9071 0.9106 0 −0.3948 0.0724−0.5217 −0.3390 0 −0.7135 −0.322−0.5702 0.1163 0.7071 0.3948 −0.0734−0.5702 0.1163 −0.7071 0.3948 −0.0734−0.2608 −0.1695 0 −0.1523 0.9381

Σ =

1.8537 0 0

0 0 00 0 00 0 00 0 0

VT =

−0.1800 −0.9671 −0.18000.6838 −0.2546 0.6838−0.7071 0 0.7071

And when these are multiplied together we have the following matrix:

M1 =

0.0324 0.1741 0.03240.1741 0.9352 0.17410.1903 1.0222 0.19030.1903 1.0222 0.19030.0870 0.4676 0.0870

22

After rounding the terms to one decimal place, the new image might looksomething like the following:


We can also work out the difference in the storage requirements from theoriginal to the current image:

Rank 3 2 1

Storage 15 18 9

We can see that the storage requirements have reduced between the orig-inal image and the image with a rank of 1, even though it were low to beginwith. Also, as predicted, when trying to compress the image to a rank of2 (i.e. larger than the maximum beneficial compression rate), the storagerequirements increased.

To gain a better perspective of how images can be compressed by manip-ulating their corresponding matrices, a larger image is needed.

4.3 Image Compression using SVD

(All MATLAB commands for this section can be found in section (7.3))

To introduce SVD with large pictures, MATLAB is an essential tool. Ma-trix manipulations become incredibly large and almost impossible by hand

23

and the required functions are carried out easily in MATLAB [12].

To show the computationally efficiency of SVD, lets look at the followingimage:

Figure 5: A grayscale image with a rank of 664

This image has dimensions 664× 1000, and so the number of storage lo-cations required to save this image is:

664× 1000 = 664000 storage locations

And has a condition number of [13]:

C(M) = 393.29440.0130

= 30253.415

This is fairly large, as the smallest singular value is close to zero. It sug-gests that the matrix isn’t close to being non-singular.

The maximum beneficial rank which should be used to compress the im-age is given by:

24

k < mnm+n+1

< 664×1000664+1000+1

< 6640001665

< 398.799

Hence we won’t be looking at a rank, k, greater than 398

The image has bee imported into MATLAB using the ‘M=imread(‘...’);’function and converted to grayscale using the ‘M=rgb2gray(M);’ command[23]. ‘imagesc(M); colormap(gray)’ [15] can then be used to view the image ingrayscale with axes. We now need to apply SVD to the images’ correspondingmatrix (each colour is assigned a number, as in the previous example) usingthe command ‘[U S V] = svd (M);’. This decomposes matrix M into threematrices of size:

U = 664× 664, Σ = 664× 1000, V = 1000× 1000

We can then carry out the MATLAB command ‘M=U(:,1:50)*S(1:50,1:50)*transpose(V(:,1:50))’ [22] to decompose the original matrix M into three ma-trices of size:

U = 664× 50, Σ = 50× 50, VT = 50× 1000

When multiplied back together to gain a new image M2 the image willtake up:

50(664 + 1000 + 1) = 83250 storage locations

And the corresponding image to matrix M2 is:

We can see that the amount of storage locations which is needed by thisimage compared with the original is reduced by nearly 90%. We can also see,however, that the image quality has been greatly reduced, and a lot of thedetail has been removed.

Lets look at the condition number:

C(M2) = 393.29443.7511

= 104.84775

25


This is greatly lower than the original, which suggests that there is much lessvariation within the matrix, i.e. there are less singular values, and thereforeless variation between the rows of the matrix.

Some more examples of SVD applied to the image are as follows:

26



27



28



29

The number of storage locations needed for each image are as follows:

Figure Rank Storage Locations Percentage of Original27 664 664000 100%12 350 582750 88%11 300 499500 75%10 150 249750 38%9 50 83250 13%8 20 33300 5%7 10 16650 3%

We can easily see that the fewer singular values there are, the less detailedthe picture is. However, with the aim of reducing storage locations, figure 10with a rank of 150 shows very little variation from the original image, andhas reduced the required storage by over 60%.

It can therefore be seen that SVD can successfully be used to reducethe storage capacity required by images, and is beneficial to computationalefficiency.

4.4 Applying SVD to colour images

(All MATLAB commands for this section can be found in section(7.4))Now that we have applied SVD to grayscale images, it is of interest to

see how it can be applied to colour images. When a colour image is storedon a computer, it is stored as three matrices correlating to the brightness ofthe ‘red’, ‘green’ and ‘blue’ within the pixels [11]. When applying SVD toa colour image, it must be applied to each red, green and blue matrix sep-arately, these can be treated as grayscale images, and SVD can be appliedas in the previous example. The matrices can then be compressed and thenrecompiled to show the compressed image.

As before, MATLAB can be used to apply the SVD to the matrices, andcan also be used to separate the three layers of the image.

Take the following image, a colour version of the image we have previ-ously used:

30

Figure 13: Original Colour Image, ‘Nature.jpg’

Now, using MATLAB to split this image into three separate images cor-responding to ‘red’, ‘green’ and ‘blue’ gives us three images [28]

Figure 14: The grayscale image showing the brightness of the ‘Red’ pixels

31

Figure 15: The grayscale image showing the brightness of the ‘Green’ pixels

Figure 16: The grayscale image showing the brightness of the ‘Blue’ pixels

32

Now applying SVD to these images individually, and then compressingthem each to a rank of 50 gives:

Figure 17: The grayscale image showing the brightness of the ‘Red’ pixels ofimage 13 with a rank of 50

Figure 18: The grayscale image showing the brightness of the ‘Green’ pixelsof image 13 with a rank of 50

33

Figure 19: The grayscale image showing the brightness of the ‘Blue’ pixelsof image 13 with a rank of 50

We can then compile these to gain our colour image which has been de-composed with a rank of 50:

Figure 20: Image 13 compressed using SVD with a rank of 50

This image, with a rank of 50 would take up three times the amount ofstorage space as it’s corresponding image in grayscale, as it is storing three

34

elements to every one pixel to gain the ‘red’, ‘green’ and ‘blue’, within theimage [29].

Some more examples of compression with different ranks:



35



36



37

These Images show that SVD can be used to successfully decompose acolour image. Reducing the rank to around 150 saves a lot of storage space,whilst keeping the detail within the image, and the vibrancy of the colours.

The storage requirements for each of the colour images are:

Figure Rank Storage Locations Percentage of Original27 664 1992000 100%12 350 1748250 88%11 300 1498500 75%10 150 749250 38%9 50 249750 13%8 20 99900 5%7 10 49950 3%

The percentage of change in space from the original to the compressed isthe same as for the grayscale. The colour images take up more storage spaceon a computer, which is as one would predict, as it needs more informationabout the colours, rather than just brightness.

This way of decomposing colour images might not be the most effective,as it is a lengthy process, but it shows how SVD can be used to decomposeany matrix or set of matrices.

38

5 Other Applications of SVD

SVD is a very useful tool and has many other applications. The equationM = UΣVT is the basis for solving many linear algebra problems, for exam-ple:

• Linear least squares problems

SVD is useful in solving matrices which are rank deficient, where there areinfinite solutions to the least squares problems [30].

• Empirical Orthogonal Functions Analysis

In this application, matrix U is the same size as matrix M and Σ and V areboth square matrices. The rows of VT then form an orthonormal basis ofM, and hence are known as Emperical orthogonal functions [31].

• Pseudo Inverse

The SVD can be used to find the pseudo inverse of a matrix, which is atool for solving a linear system Ax = b and gives a geometric interpretation[32].

39

6 Conclusion

SVD uses the equation M = UΣVT to break down matrix M into threemultiplicative matrices. This decomposition has many applications, of whichimage compression has been analysed in this report.

SVD gives u a tool for compressing images. This has been seen to bene-ficial when reducing the amount of data that an image requires to be storedon a computer and hence the computational efficiency of storing an image orsending it via email.

Although this method for image compression may not be the most effi-cient for colour images, it can be seen from this report that it is possible.

Further investigation is required to analyse the rate to which the imagesshould be compressed, perhaps by the analysis of an original image againstit’s compressed image. This would give a visual interpretation of how muchdetail has been removed. These can be compared for each rank to gain anoptimal compression rate.

40

7 Appendix

7.1 MATLAB commands for section (3.2)

>> M=[1 5 9; 2 6 10; 3 7 11; 4 8 12];

>> [U S T]=svd(M)

U =-0.4036 -0.7329 -0.2819 -0.4696-0.4647 -0.2898 0.7259 0.4160-0.5259 0.1532 -0.6060 0.5768-0.5870 0.5962 0.1620 -0.5232

S =25.4368 0 0

0 1.7226 00 0 0.00000 0 0

T =-0.2067 0.8892 -0.4082-0.5183 0.2544 0.8165-0.8298 -0.3804 -0.4082

>> V=transpose(T)

V =-0.2067 -0.5183 -0.82980.8892 0.2544 -0.3804

-0.4082 0.8165 -0.4082

>> U*S*V

ans =1.0000 5.0000 9.00002.0000 6.0000 10.00003.0000 7.0000 11.00004.0000 8.0000 12.0000

>> rank(M)

41

ans =2

>> svd(M)

ans =25.43681.72260.0000


>> N=[0 0 0; 0 1 0; 0 1 0; 0 1 0; 0 0 0]

N =0 0 00 1 00 1 00 1 00 0 0

>> rank(A)

ans =1

>> M=[0.5 0 0.5; 0 1 0; 0 1 0.5; 0.5 1 0; 0 0.5 0]

M =0.5000 0 0.5000

0 1.0000 00 1.0000 0.5000

0.5000 1.0000 00 0.5000 0

>> rank(M)

42

ans =3

>> [U,S,T]=svd(M)

U =-0.0971 0.9106 0.0000 -0.3948 0.0734-0.5217 -0.3390 -0.0000 -0.7135 -0.3222-0.5702 0.1163 0.7071 0.3948 -0.0734-0.5702 0.1163 -0.7071 0.3948 -0.0734-0.2608 -0.1695 -0.0000 -0.1523 0.9381

S =1.8537 0 0

0 0.7509 00 0 0.50000 0 00 0 0

T =-0.1800 0.6838 -0.7071-0.9671 -0.2546 -0.0000-0.1800 0.6838 0.7071

>> V=transpose(T)

V =-0.1800 -0.9671 -0.18000.6838 -0.2546 0.6838

-0.7071 -0.0000 0.7071

>> S=[1.8537 0 0; 0 0.7509 0; 0 0 0; 0 0 0; 0 0 0]

S =1.8537 0 0

0 0.7509 00 0 00 0 00 0 0

>> M2=U*S*V

43

M2 =0.5000 0.0000 0.50000.0000 1.0000 0.00000.2500 1.0000 0.25000.2500 1.0000 0.25000.0000 0.5000 0.0000

>> rank(M2)

ans =2

>> S=[1.8537 0 0; 0 0 0; 0 0 0; 0 0 0; 0 0 0]

S =1.8537 0 0

0 0 00 0 00 0 00 0 0

>> M1=U*S*V

M1 =0.0324 0.1741 0.03240.1741 0.9352 0.17410.1903 1.0222 0.19030.1903 1.0222 0.19030.0870 0.4676 0.0870

>> rank(M1)

ans =1


44

Figure 27: Original Colour Image, ‘Nature.jpg’

>> M=imread(’Nature.jpg’);>> M=rgb2gray(M);>> M=im2double(M);>> imagesc(M); colormap(gray)

>> L=svd(M);

>> min(L)

ans =0.0130

>> max(L)

ans =393.2944

>> rank(M)

ans =664

45

>> [U S V]=svd(A);

>> A=U(:,1:350)*S(1:350,1:350)*(transpose(V(:,1:350)));>> imagesc(A); colormap(gray)







>> M=imread(‘Nature.jpg’);

>> R=M(:,:,1);>> G=M(:,:,2);>> B=M(:,:,3);

>> R=im2double(R);>> G=im2double(G);>> B=im2double(B);

>> [UR SR VR]=svd(R);>> R50=UR(:,1:50)*SR(1:50,1:50)*transpose(VR(:,1:50));

46

>> imshow(R50)

>> [UG SG VG]=svd(G);>> G50=UG(:,1:50)*SG(1:50,1:50)*transpose(VG(:,1:50));>> imshow(G50)

>> [UB SB VB]=svd(B);>> B50=UB(:,1:50)*SB(1:50,1:50)*transpose(VB(:,1:50));>> imshow(B50)

>> Q50(:,:,1)=R50;>> Q50(:,:,2)=G50;>> Q50(:,:,3)=B50;>> imshow(Q50)

>> R350=UR(:,1:350)*SR(1:350,1:350)*transpose(VR(:,1:350));>>G350=UG(:,1:350)*SG(1:350,1:350)*transpose(VG(:,1:350));>> B350=UB(:,1:350)*SB(1:350,1:350)*transpose(VB(:,1:350));>> Q350(:,:,1)=R350;>> Q350(:,:,2)=G350;>> Q350(:,:,3)=B350;>> imshow(Q350)



47

>> R20=UR(:,1:20)*SR(1:20,1:20)*transpose(VR(:,1:20));>> G20=UG(:,1:20)*SG(1:20,1:20)*transpose(VG(:,1:20));>> B20=UB(:,1:20)*SB(1:20,1:20)*transpose(VB(:,1:20));>> Q20(:,:,1)=R20;>> Q20(:,:,2)=G20;>> Q20(:,:,3)=B20;>> imshow(Q20)

>> R10=UR(:,1:10)*SR(1:10,1:10)*transpose(VR(:,1:10));>> G10=UG(:,1:10)*SG(1:10,1:10)*transpose(VG(:,1:10));>> B10=UB(:,1:10)*SB(1:10,1:10)*transpose(VB(:,1:10));>> Q10(:,:,1)=R10;>> Q10(:,:,2)=G10;>> Q10(:,:,3)=B10;>> imshow(Q10)

48

8 Glossary

Determinant The determinant of a matrix, usually denoteddet(A) is a real number associated with squarematrices. For a 2× 2 matrix, it is found by [24]

det

(a bc d

)= ad− bc

Determinants of matrices larger than a 2 × 2matrix are harder to find, and so MATLAB isused.

Diagonal Matrix A Diagonal matrix has entries along it’s top leftto bottom right diagonal, and zeros elsewhere.It is usually a square matrix, and can containzero entries along the diagonal also.

Identity Matrix Usually denoted with the letter ‘I’, the Iden-tity matrix is a diagonal matrix of any size, butwhose diagonal entries are all ones i.e.

I =

1 0 0

0. . . 0

0 0 1

Inverse (of a matrix) The inverse of a matrix, A is such that AA−1 =I [25].

Normalised (vector) Also known as a unit vector. It has a lengthof 1 and is found by dividing the entries of thevecotr by it’s magnitude:

Orthogonal An orthogonal matrix is one whose column en-tries are made from orthogonal unit vectors [26].A set of orthogonal unit vectors are ones whoselength are 1 and dot product are 0.

49

Scalar A quantity which is a magnitude and has nodirection [27].

Singular matrix A matrix which is singular has a determinantof zero and doesn’t have an inverse. A non-singular matrix is one which does have an in-verse and whose determinant in non-zero [33].

Transpose To transpose a matrix is to reflect it along it’sdiagonal so that the rows of the matrix becomeit’s columns.

Zero Matrix A Zero matrix is a matrix of any size, but whoseentries are all zeros.

50

References

[1] Kirk Baker, Singular Value Decomposition Tutorial, March 2005,http://www.ling.ohio-state.edu/∼kbaker/pubs/Singular Value DecompositionTutorial.pdf

[2] Kuldeep Singh, Eigenvalues and Eigenvectors, 2003Engineering Mathematics through Applications, Pages 549–550

[3] Susan Blackford, Singular Values and Singular Vectors, 2000http://web.eecs.utk.edu/ dongarra/etemplates/node40.html

[4] Kuldeep Singh, Singular Value Decompositionwww.mathsforall.co.uk/linear algebra.php

[5] Todd Rowland, Orthogonal Matrix, 1999-2013http://mathworld.wolfram.com/OrthogonalMatrix.html

[6] Gilbert Strang, Introduction to Linear Algebra, 2009http://math.mit.edu/linearalgebra/ila0205.pdf

[7] David Austin, American Mathematical Societyhttp://www.ams.org/samplings/feature-column/fcarc-svd

[8] Elliott Ward Cheney, David Kincaid, Condion Number and Ill-Conditioning, 2008Numerical Mathematics and Computing, Page 321

[9] Dan Spielman, Linear Algebra Review, December 2002http://www.cs.yale.edu/homes/spielman/BAP/lect2.pdf

[10] R2013a Documentation, MathWorks, 2013http://www.mathworks.co.uk/help/matlab/ref/svd.html?searchHighlight=SVD

[11] Ian Cooper, Craig Lorenc, Image compression using SVD, Decem-ber 2006http://online.redwoods.edu/instruct/darnold/laproj/fall2006/iancraig/SVD Slideshow.pdf

[12] Kuldeep Singh, Reduced SVD and Image Compression UsingMATLAB

51

[13] Jamshed Iqbal, John D’Errico, ’Min. and max. singular values ofa non-square matrix’, July 2009http://www.mathworks.co.uk/matlabcentral/newsreader/viewthread/257475

[14] Mathworks, R2013a Documentation,http://www.mathworks.co.uk/help/matlab/ref/svd.html?searchHighlight=svd

[15] Mathworks, R2013a Documentationhttp://www.mathworks.co.uk/help/matlab/ref/imagesc.html

[16] Kuldeep Singh, Diagonalisation of Symmetric Matriceshttp://mathsforall.co.uk/userfiles/SECTION%207D%20Diagonalisation%20of%20Symmetric%20Matrice.pdf

[17] James S. Cook, Symmetric Matrix Proof, September 2012http://math.stackexchange.com/questions/200030/symmetric-matrix-proof

[18] Inder K. Rana, Existence of Eigenvalues for Symmetric Matrices,2003-2006http://www.math4all.in/public html/linear%20algebra/chapter10.2.html

[19] The Rank of a matrixhttp://www.cliffsnotes.com/study guide/The-Rank-of-a-Matrix.topicArticleId-20807,articleId-20790.html

[20] Lijie Cao, Singular Value Decomposition Applied To Digital Im-age Processinghttp://www.lokminglui.com/CaoSVDintro.pdf

[21] Clemson University, Supplementary material: Singular Valueshttp://www.clemson.edu/ces/crb/ece801/notes/chap04t svds.pdf

[22] Joshua Peterson, Image Compression in Matlab using SVD, March2011http://users.humboldt.edu/jmpeterson/jmp svd imgs matlab.pdf

[23] UC Berkeley, Applications of SVD: Image Compression, Septem-ber 2011https://inst.eecs.berkeley.edu/ee127a/book/login/l svd apps image.html

[24] James Jones, The determinant of a square matrix, 2013http://people.richland.edu/james/lecture/m116/matrices/determinant.html

52

[25] Eric Weisstein, Matrix Inversehttp://mathworld.wolfram.com/MatrixInverse.html

[26] Orthogonal Matrixhttp://www.princeton.edu/achaney/tmve/wiki100k/docs/Orthogonal matrix.html

[27] The free dictionary, Scalarhttp://www.thefreedictionary.com/scalar

[28] Jordan Rosenthal, Red, Green and Blue Component, May 2001http://www.mathworks.co.uk/matlabcentral/newsreader/view thread/24291

[29] Ben Arnold, An Investigation into using Singular Value Decom-position as a method of Image Compression, September 2000http://www.cs.bgu.ac.il/na022/Arnold.pdf

[30] Linear Least Squares Problem, 2004http://classes.soe.ucsc.edu/cmps290c/Spring04/paps/lls.pdf

[31] Marc Speigelman, Spplications of the SVDhttp://www.columbia.edu/itc/applied/e3101/SVD applications.pdf

[32] David Clark, A Note on the Pseudoinversehttp://faculty.rmc.edu/davidclark/math/papers/pseudoinverse.pdf

[33] Eric Weisstein, Singular Matrix/ Nonsingular Matrixhttp://mathworld.wolfram.com/NonsingularMatrix.html

53

investigation in mathematics singular value decomposition value...abstract this investigation looks...

Documents