qr factorization chapter4

8/3/2019 QR Factorization Chapter4

1/12

Matrix Algebra 1 QR-decomposition 86

4. QR-DECOMPOSITION

4.1 Introduction

In this chapter we show that given any high matrix (m n )A, that has rank n,it can be written as a product of an mxm unitary matrices Q and an mxn uppertriangular matrixR:

A = QR.

QR decomposition has various applications. It is used, for example, to find theleast squares solution for a system of linear equations Ax=b (multiplying by aunitary matrix Q does not change the norm) and surprisingly the best knownmethod for finding the eigenvalues of a general matrix is based on QRdecomposition. QR decomposition is also used to construct an orthonormal basis

for a subspace S of Fm. We take the matrix A consisting of the original basis

vectors of the subspace, the first n column vectors of Q now form the basis

required and the rest of the columns form an orthonormal basis for S.

4.2 Householder matrix

In chapter 2 we noticed that left multiplication of a matrix A by suitableGaussian matrices transformsA to an upper triangular matrix. The same kind oftransformation can be done by using unitary Householder matrices.Multiplication by a unitary matrix does not change the length of a vector, only

the direction, so they are safe to use in numerical computation.

Let v0 be a vector in Fm. A Householder matrix is a Hermitian, unitary,mxm matrix, of the form

P =I 2vv

v

2

*

.

Exercise 4.2.1 Prove, thatP is Hermitian and unitary.


2/12


v

v 2 v

v2

xv

v2

{ }v

Px x

Picture 4.2.1 The transformation carried out by a Householder matrix.The vectorx is transformed into a mirror image of itself with respect to the

orthogonal complement {v} of the vector v.

Letx0 be a vector in Fm. We try to choose a vector v such that its associatedHouseholder matrix gives us,

Px=e1.

We have shown in exercise 4.2.1 thatP is unitary, so ||Px||=||x||, which means

that || =||x ||. We write the solution in the following form. The motivation forthis will become evident later.

Px= ei||x||e1.When we writeP in terms of its components, we get the equation

Px =x 2vv*x

v2=x 2

< v,x >

v2

v = -ei x e

1

and further

2< v,x >

v2

v = x +eix e

1.

The left and right sides of the equation must be parallel for a solution to exist.This means that

v = x +ei x e1( )


3/12


where is an unknown scalar for the time being. Substituting v into the previousequation gives

1 = 2

< x +ei x e1

( ),x >

x +ei x e1( )

2

= 2< x +ei x e

1( ),x >< x +ei x e

1( ), x +ei x e1( ) >

= 2< x +e

ix e

1( ),x >

< x +ei x e1( )

, x +ei x e1( ) >

=

2 < x +eix e

1( ),x >< x +e

ix e

1( ), x +ei x e1( ) >

because does not occur in the equation anymore we must find a coefficient sothat the equation holds. We compute all the inner products and get the equation

1 =

2 x2+e

ix x

1

x2+ x

2+e

ix x

1+e

ix x

1

where x1 is the first element of the vector x and = x1. Dividing the

equation by ||x|| gives

1 =

2 x +ei

x1( )

2 x +ei

x1+e

ix1

2 x + ei

x1+ e

ix1= 2 x +e

ix1( )

ei

x1+ e

ix1= 2e

ix1

eix1= e

ix1

In general the elementx1 is a complex number and it can be written in the form

x1= ei|x1|. (x1 0) Substituting this into the equation above reveals that thesolutions are of the form:


4/12


=+ k, k= ..-n,..-1,0,1,...,n,....

Only the solutions=and =+ give different values for ei . Lets compare thesolutions. The formula for the Householder matrix is easy to compute

numerically if the norm of the vector v does not get too small. This is not a bigproblem because

v

2=

22 x

2+e

ix x

1+e

ix x

1

If=, then

v

2=

22 x

2+ 2 x x

1

If=+ , then

v

2=

22 x

2 2 x x

1

.

The first solution of these two is numerically well behaved. If we choose the lattersolution, the vector v could be zero or the vector could become very small. So we

choose = and the vector required is

v = x +ei x e1

,x1 =e

i x1

.

We have also shown that the selection ofv does not depend on the scalar , so we

can choose it to be one. If |x1|=0, we choose =0. NB. If the vectorx is real and x

1is its first element, then

a) ifx1 0 , then x

1=e

i0x1

, so = 0 .

b) ifx1< 0 , then x

1=e

ix1

, so = .

Example 4.2.1 Letx =[3,1,5,1]T. We form the Householder matrix Psuch thatPx span e

1{ }.

because 3>0, we choose = 0 and

v =x + x e1=

3

1

5

1

+

6

0

0

0

=

9

1

5

1

.


5/12


Substitution into the formula gives us

P =I 2vv

T

v

2=

1

54

27 9 45 9

9 53 5 1

45 5 29 5

9 1 5 53

.

and a little computation shows us that

Px =1

54

27 9 45 9

9 53 5 1

45 5 29 5

9 1 5 53

3

1

5

1

=

-6

0

0

0

= -6e1.

4.3 Constructing the QR-decomposition by using Householdermatrices

4.3.1 Introduction

In this section we transform a matrix A to upper triangular form as we did whenconstructing the LU-decomposition. The main difference being that we use

Householder matrices instead of Gaussian matrices. Householder matrices areunitary, so they have excellent numerical properties.

4.3.2 QR decomposition

Let

A = a1,a2,....,an

be an mxn matrix, that has rank rank(A) = n and mn .1st step.rank(A) = n, a10, and so we can choose an mxm Householder matrix

P1 to satisfy the equation

1a1= e

ia1

e1= r

11e

1.

So


6/12


P1A = r11e1,P1a2,....,P1an =

r11

0

||

||

|

|

r12

A2( )

r1n

.

because a10, r11 0 .

2nd step. We write the (m-1)x(n-1) matrixA(2) in terms of its columns,

A2( )= a2

2( ),a3

2( ),....,a

n

2( )

.

Our previous assumption about the rank says that a2(2)0. (because ifa2

(2)=0,

then rank(A) = rank(P1A) < n , a contradiction.) We can thus form the (m-1)x(m-1)

Householder matrixB2 such that

B2a2

2( )= ei a

2

2( ) 1,0,...,0

T= r

22e

1.

because a20, r22 0 .

Let

P2=

1 0T

0 B2

.

and left multiplyP1A with it. Now we have

P2P1A =

r11

r12

r1n

0 r22

r2n

0 0 A3( )

.

It is easy to see thatP2 is unitary.

3rd step. Againwe can form the Householder matrixB3 such that

B3a3(3) = r33e1,

since a3(3)0 where a3

(3) is the first column of the matrix A(3), so we have

a3(3)0 and r33 0.We next construct


7/12


P3=

I2

0

0 B3

.

Now the first three columns ofP3P2P1A are in upper triangular form.

nth step. Thematrix A(n) is a m-n+1-vector. According to the rank assumption

A(n)0. If m = n, we can stop. Ifm > n, we can find a Householder matrix Bn

such that

BnA

n( )= r

nne1

where rnn0. We let

Pn=

In1 0

0 Bn

.

We denote Q* =Pn.....P2P1 which gives us the solution we wanted

Q*A =R =Rn0

,

where R is mxn upper triangular matrix ( Rn is nxn upper triangular matrix) ,

and Q is unitary as it is a product of unitary mxm matrices.

Example 4.3.1 The QR decomposition of the matrix

A =

1 1 1

1 2 1

1 2 1

is

1

3

+2

3

0

1

3

1

6

1

2

1

3

1

6

+1

2

3 35

3+

3

3

0 2

3

2

3

0 0 2

.

Exercise 4.3.2. Assume thatA is a square matrix and rank(A) = n.Show that Qcan be chosen so that the diagonal elements ofR are positive.


8/12


(Hint: If the vectors in a orthogonal set {qi} are multiplied by complex numbers

i, that have absolute value |i|=1, then the new set of vectors {iqi} is

orthogonal as well.)

Use this result to transform the diagonal elements of the matrix R to positive

numbers in the QR decomposition of the example 4.2.1.

Answer is

QR =

1

3

2

30

1

3

1

6

1

2

1

3

1

6

1

2

3 35

3

3

3

02

3

2

3

0 0 2

.

Example 4.3.2 Computing the QR decomposition with a pen and paper is alaborious task. Programming it is easy. Let

A =

0

1

0

0

3

2

.

Lets construct its QR decomposition.

1st step

v =

1

1

0

,P1 =

0 1 0

1 0 0

0 0 1

P1A =

1

0

0

3

0

2

.

2nd step

A2( )=

0

2

, v =

2

2

,B2 =

0 11 0

,

P2=

1 0 0

0 0 1

0 1 0

. P2P1A =

1

0

0

3

2

0

=

R2

0

Q = P2P

1

( )

1=P

1

*P2

*=

0 0 1

1 0 0

0 1 0

.


9/12


Check, that

A = QR.

If the diagonal elements ofR are chosen to be positive then

R =

1

0

0

3

2

0

,Q =

0 0 1

1 0 0

0 1 0

.

Computation times. Constructing the QR decomposition of an mxn matrix

takes n2(m-n/3) flops.

Exercise 4.3.3Let

A =

0 0

1 2

0 1

.

Construct the QR decomposition by using Householder matrices and give thesolution where the diagonal elements ofR are positive.

Exercise 4.3.4 Let A be an nxn matrix and rank(A)=n. Show that the QR

decomposition is unique if the diagonal elements ofR are chosen to be positive.

Exercise 4.3.5a) Use QR decomposition to solve the least square problem. Letmn and the columns ofA be linearly independent. Find the vector x such thatthe error

||Ax-b||

is as small as possible. How big is the error? Hint: Multiplication by a unitarymatrix does not change the norm of a vector.

b) Find the least squares solution, when b=[1, 1, 1]Tand

A =

0 0

1 3

0 2

Hint: Example 4.3.2.

4.4 Orthonormal basis and decomposition of space


10/12


QR decomposition can be used to transform a given linearly independent set ofvectors to an orthonormal set. The orthonormal set spans the same subspace asthe original vector set. An orthonormal basis for the orthogonal complement isconstructed at the same time.

Let {x1,x2,...,xn} be a linearly independent set of vectors in Fm

(mn) and

R=span x1,x2,...,xn{ } .

Form the mxn matrix X= [x1,x2,...,xn]. It has rank rank(X) = n (why). Assume

thatXhas the following QR decomposition

n

X= x1,...,xn =QRn0

= q1,q2,...,qm

r11

r12

r1n

0 r22

r23

.. r2n

.. .. ... ...

0 .. 0 rnn

0 0 0

.. .. .. ..

0 0... 0 0

m

Direct computation shows, that

X=x1,...,xn

=

q1,q2,...,qn

R

n,

whereRn is a square upper triangular matrix. It has nonzero diagonal elements,

so by theorem 3.4.1

span x1,...,x

n{ } =span q1,q2,...,qn{ }.

The set {q1,q2,...,qn} is orthonormal because it consists of the column vectors of a

unitary matrix (exercise 1.20.3). We note also that

x1 = r11q1,

so x1 and q1 are parallel.

Further we notice that the column vectors ofQ

q1,q

2,...,q

n, q

n+1, ...,q

m{ } ,

form an orthonormal basis for Fm (as any m linearly independent vectors form abasis). In addition


11/12


R=span q1,q

2,...,q

n{ }.

A method to find coordinates in an orthonormal basis can be derived from thegeneral formula for coordinates, which was presented in chapter 3. Because this

result is frequently used we give it here.

Theorem 4.4.1. Let {q1,q2,...,qm} be an orthonormal basis ofFm. Every vector in

the space Fm has the following representation as a linear combination of basisvectors

x = < qi

i=1

m

,x > qi.

Also

x2= < q

i,x >

2

i=1

m

.

Proof. Every vector is a linear combination of basis vectors

x = ci

i=1

m

qi = q1,q2,...,qm [c1,....,cm]T= Qc .

We can solve the coefficient vector c.

c =

c1

cm

= Q *x=

< q1,x >

< qm,x >

.

Furthermore, because Q is unitary,

x2= Qc

2= c

2= < q

i,x >

2

i=1

m

Theorem 4.4.2. Let RFm be a subspace. Now

Fm= RR.

Proof. According to theorem 4.5.1 a basis {x1,...,xn} ofRcan be extended to an

orthonormal basis {q1,q2,...,qn,qn+1,...,qm} ofFm so that R=span {q1,q2,...,qn}.

We show that

R =span qn+1

, qn+2

,..., qm{ }.


12/12


We can easily see that span{qn+1,qn+2,...,qm} R. Let x be a vector in R,

because {q1,q2,...,qn,qn+1,...,qm} is a basis,x has the following representation:

x = < qi

i=1

m

,x > qi = < qii=1

n

,x > qi + < qi,x > qii=n+1

m

.

and sincexR the inner products = 0 , when i=1,...,n. Sox = < qi,x > qi

i=n+1

m

span qn+1, qn+2 ,...,qm{ }.

{q1,q2,...,,qm} is orthonormal basis ofFm and so we know that Fm = RR.

Exercise 4.4.1 Let Sbe a subspace. Show, that (S

)

= S.Hint: It is easy to prove, thatS (S

) . The other direction could be shown by

using the partition Fm = SS.

Exercise 4.4.2 LetA be an mxn matrix. Show, that

Fn= R(A*)N A( ) , Fm = R(A)N A *( ).

This result verifies the separation of the domain and codomain spaces intoorthogonal sums of subspaces as represented in picture 3.6.1.

Exercise 4.4.3 A well known method for transforming a given set of vectors toan orthonormal set is the Gram-Schmidt algorithm. Deduce it from the QRdecomposition where the diagonal elements ofR are positive by researching theinteraction between matrices X and Q. In numerical computation the Gram-Schmidt algorithm is not as good as QR decomposition. The Gram-Schmidtalgorithm might not work properly in a situation where the basis vectors arealmost linearly dependent.

qr factorization chapter4

Documents