steepest decent and cg
Post on 09-Apr-2018
228 Views
Preview:
TRANSCRIPT
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 1/68
Steepest Decent and Conjugate Gradients (CG)
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 2/68
Steepest Decent and Conjugate Gradients (CG)
Solving of the linear equation system b Ax !
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 3/68
Steepest Decent and Conjugate Gradients (CG)
Solving of the linear equation system
Problem: dimension n too big, or not enough time for
gauss eliminationIterative methods are used to get an
approximate solution.
b Ax !
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 4/68
Steepest Decent and Conjugate Gradients (CG)
Solving of the linear equation system
Problem: dimension n too big, or not enough time for
gauss eliminationIterative methods are used to get an
approximate solution.
Definition Iterative method: given starting point , do
stepshopefully converge to the right solution
b Ax !
0 x
-,,21x xp
x
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 5/68
starting issues
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 6/68
starting issues
Solving is equivalent to minimizingb Ax !
c xb Ax x x f T T !
2
1:)(
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 7/68
starting issues
Solving is equivalent to minimizing
A has to be symmetric positive definite:
b Ax !
c xb Ax x x f T T !
2
1:)(
00 {"! x Ax x A A T T
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 8/68
starting issues
02
1
2
1)(
!
!!! b Axb Ax x A x f symmetric A
T
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 9/68
starting issues
If A is also positive definite the solution of is theminimum
02
1
2
1)(
!
!!! b Axb Ax x A x f symmetric A
T
b Ax !
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 10/68
starting issues
If A is also positive definite the solution of is theminimum
02
1
2
1)(
!
!!! b Axb Ax x A x f symmetric A
T
b Ax !
-00
11
2
1
2
1)(
{"
!!d
T T Ad d cb Abd b A f
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 11/68
starting issues
error:
The norm of the error shows how far we are away from the
exact solution, but can¶t be computed without knowing of the exact solution .
x xe ii !:
x
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 12/68
starting issues
error:
The norm of the error shows how far we are away from the
exact solution, but can¶t be computed without knowing of the exact solution .
residual:
can be calculated
x xe ii !:
x
)(: x f Ae Axbr iii !!!
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 13/68
Steepest Decent
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 14/68
Steepest Decent
We are at the point . How do we reach ?i x 1i x
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 15/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( )
i x 1i x
)( x f
ii r x f !! )(
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 16/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
i x 1i x
)( x f
ii r x f !! )(
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 17/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 18/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
0)( ! ii r x f
d
d E
E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 19/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
0)( ! ii r x f
d
d E
E0)( ! iii r r x f E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 20/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
0)( ! ii r x f
d
d E
E0)( ! iii r r x f E
0))(( ! iii r br x E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 21/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
0)( ! ii r x f
d
d E
E0)( ! iii r r x f E
0))(( ! iii r br x E
i
r
iii r xbr r
i
)()(
!E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 22/68
Steepest Decent
We are at the point . How do we reach ?
Idea: go into the direction in which decreases most
quickly ( ) how far should we go?
Choose so that is minimized:
i x 1i x
)( x f
ii r x f !! )(
E )( ii r x f E
0)( ! ii r x f d
d E
E0)( ! iii r r x f E
0))(( ! iii r br x E
i
r
iii r xbr r
i
)()(
!Ei
T
i
i
T
i
Ar r
r r !E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 23/68
Steepest Decent
one step of steepest decent can be calculated as follows:
iiii
i
iii
ii
r x x
r r
r r
xbr
E
E
!
!
!
1
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 24/68
Steepest Decent
one step of steepest decent can be calculated as follows:
stopping criterion: or with an givensmall
It would be better to use the error instead of the residual,
but you can¶t calculate the error.
iiii
i
iii
ii
r x x
r r
r r
xbr
E
E
!
!
!
1
maxii " 0r r i I
I
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 25/68
Steepest Decent
Method of steepest decent:
1
)(
0
00max
0
0
n
n
n
n
"
nn
n
ii
Axbr
r x x
Ar r
r r
r r r r and iiwhile
r r Axbr
i
T
T
T T
E
E
I
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 26/68
Steepest Decent
As you can see the starting point is important!
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 27/68
Steepest Decent
As you can see the starting point is important!
When you know anything about the solution use it to guessa good starting point. Otherwise you can choose a starting
point you want e.g. .00! x
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 28/68
Steepest Decent - Convergence
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 29/68
Steepest Decent - Convergence
Definition energy norm: A x x x A!:
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 30/68
Steepest Decent - Convergence
Definition energy norm:
Definition condition:
( is the largest and the smallest eigenvalue of A)
A x x x A!:
min
max
: P
P
O !
maxP minP
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 31/68
Steepest Decent - Convergence
Definition energy norm:
Definition condition:
( is the largest and the smallest eigenvalue of A)
convergence gets worse when the condition gets larger
A x x x A!:
min
max
: P
P
O !
maxP minP
A
i
Ai ee0
1
1¹ º
¸©ª
¨
e
O
O
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 32/68
Conjugate Gradients
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 34/68
Conjugate Gradients
is there a better direction?
Idea: orthogonal search directions110
,,, nd d d -
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 35/68
Conjugate Gradients
is there a better direction?
Idea: orthogonal search directions110
,,, nd d d -
§
!
!1
0
n
i
iid x P
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 37/68
Conjugate Gradients
is there a better direction?
Idea: orthogonal search directions
only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
110,,, nd d d -
§
!
!1
0
n
i
iid x P
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 38/68
Conjugate Gradients
is there a better direction?
Idea: orthogonal search directions
only walk once in each direction and minimize
maximal n steps are needed to reach the exact solution
has to be orthogonal to
110,,, nd d d -
§
!
!1
0
n
i
iid x P
1 ie id
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 39/68
Conjugate Gradients
example with the coordinate axes as orthogonal search
directions:
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 40/68
Conjugate Gradients
example with the coordinate axes as orthogonal search
directions:
Problem: can¶t becomputed because
(you don¶t know !)i
T
i
i
T
ii
d d
ed !E
ie
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 41/68
Conjugate Gradients
new idea: A-orthogonal110,,, nd d d -
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 42/68
Conjugate Gradients
new idea: A-orthogonal
Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
110,,, nd d d -
ji d d , 0! j
T
i Ad d
ji d d , 0! j
T
i d d
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 43/68
Conjugate Gradients
new idea: A-orthogonal
Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
now has to be A-orthogonal to
110,,, nd d d -
ji d d , 0! j
T
i Ad d
ji d d , 0! j
T
i d d
1ie id
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 44/68
Conjugate Gradients
new idea: A-orthogonal
Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
now has to be A-orthogonal to
110,,, nd d d -
ji d d , 0! j
T
i Ad d
ji d d , 0! j
T
i d d
1ie id
iT i
i
T
i
iT i
i
T
i
i Ad d
r d
Ad d
Aed !!E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 45/68
Conjugate Gradients
new idea: A-orthogonal
Definition A-orthogonal: A-orthogonal
(reminder: orthogonal: )
now has to be A-orthogonal to
can be computed!
110,,, nd d d -
ji d d , 0! j
T
i Ad d
ji d d , 0! j
T
i d d
1ie id
iT i
i
T
i
iT i
i
T
i
i Ad d
r d
Ad d
Aed !!E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 46/68
Conjugate Gradients
A set of A-orthogonal directions can be found with n
linearly independent vectors and conjugate Gram-
Schmidt (same idea as Gram-Schmidt).iu
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 47/68
Conjugate Gradients
Gram-Schmidt:
linearly independent vectors10,, nuu -
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 48/68
Conjugate Gradients
Gram-Schmidt:
linearly independent vectors10,, nuu -
j
T
j
j
T
i
i j
i
j
ji jii
d d
d u
d ud i
ud
!
!"
!§
!
F
F1
0
00
:0
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 49/68
Conjugate Gradients
Gram-Schmidt:
linearly independent vectors
conjugate Gram-Schmidt:
10,, nuu -
j
T
j
j
T
i
i j Ad d
Ad u! F
j
T
j
j
T
i
i j
i
j
ji jii
d d
d u
d ud i
ud
!
!"
!
§
!
F
F1
0
00
:0
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 50/68
Conjugate Gradients
A set of A-orthogonal directions can be found with n
linearly independent vectors and conjugate Gram-
Schmidt (same idea as Gram-Schmidt).
CG works by setting (makes conjugate Gram-
Schmidt easy)
iu
ii r u !
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 51/68
Conjugate Gradients
A set of A-orthogonal directions can be found with n
linearly independent vectors and conjugate Gram-
Schmidt (same idea as Gram-Schmidt).
CG works by setting (makes conjugate Gram-
Schmidt easy)
with1! iiii d r d F11
!i
T
i
i
T
ii
r r
r r F
ii r u !
iu
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 53/68
Conjugate Gradients
0:1
0
1
!!!! §§
!!
!
n
jk
k
T
ik
n
jk
k k
T
ii
T
i j
T
i Ad d d Ad Aed r d ji
PP
§
!!
1
0
i
k k ik ii d ud F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 54/68
Conjugate Gradients
0:1
0
1
!!!! §§
!!
!
n
jk
k
T
ik
n
jk
k k
T
ii
T
i j
T
i Ad d d Ad Aed r d ji
PP
§
!!
1
0
i
k k ik ii d ud F
]
§
!!
!!
1
00
0:i
k jk
j
T
k ik j
T
i j
T
i r d r ur d ji F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 55/68
Conjugate Gradients
0:1
0
1
!!!! §§
!!
!
n
jk
k
T
ik
n
jk
k k
T
ii
T
i j
T
i Ad d d Ad Aed r d ji
PP
§
!!
1
0
i
k k ik ii d ud F
]
§
!!
!!
1
00
0:i
k jk
j
T
k ik j
T
i j
T
i r d r ur d ji F
jir u j
T
i ! 0
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 56/68
Conjugate Gradients
0:1
0
1
!!!! §§
!!
!
n
jk
k
T
ik
n
jk
k k
T
ii
T
i j
T
i Ad d d Ad Aed r d ji
PP
§
!!
1
0
i
k k ik ii d ud F
]
§
!!
!!
1
00
0:i
k jk
j
T
k ik j
T
i j
T
i r d r ur d ji F
jir u j
T
i ! 0
i j j
T
i
j
T
iii
r r
jir r r u
H !
!! 0:
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 57/68
Conjugate Gradients
0:1
0
1
!!!! §§
!!
!
n
jk
k
T
ik
n
jk
k k
T
ii
T
i j
T
i Ad d d Ad Aed r d ji
PP
i j j
T
i
j
T
iii
r r jir r r u
H !
!!0
:
]
i
T
i
i
k
jk
j
T
k ik i
T
ii
T
i r ur d r ur d !! §
!
!
1
00
F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 58/68
Conjugate Gradients
ji Ad d
Ad r
j
T
j
j
T
i
i j "! F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 59/68
Conjugate Gradients
ji Ad d
Ad r
j
T
j
j
T
i
i j "! F
j j j j j j j jAd r d e A Aer EE !!!
)(
11
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 60/68
Conjugate Gradients
ji Ad d
Ad r
j
T
j
j
T
i
i j "! F
j j j j j j j jAd r d e A Aer EE !!!
)(
11
j
T
i j j
T
i j
T
i Ad r r r r r E!1
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 61/68
Conjugate Gradients
ji Ad d
Ad r
j
T
j
j
T
i
i j "! F
j j j j j j j jAd r d e A Aer EE !!!
)(
11
j
T
i j j
T
i j
T
i Ad r r r r r E!1
1! j
T
i j
T
i j
T
i j r r r r Ad r E
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 62/68
Conjugate Gradients
1! j
T
i j
T
i j
T
i j r r r r Ad r E
±±±
°
±±±
¯
®
{{
!
!
!
!
10
11
ji ji
jir r
jir r
Ad r
r r
i
i
T
i
i
i
T
i
j
T
i
i j j
T
i
E
E
H
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 63/68
Conjugate Gradients
°¯
"!
10 jii j F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 64/68
Conjugate Gradients
±°
±¯
®
!!
"
!
! 1
10
1111111
jir r
r r
r d
r r
Ad d
r r
ji
i
T
i
i
T
i
i
T
i
i
T
id ef
i
T
ii
i
T
ii j
E
E
F
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 65/68
Method of Conjugate Gradients:
0
0
0
n
nn
n
i
r r r d
Axbr
1
)(00max
n
n
n
n
n
n
n
"
ii
d r d
r r
r r
A xbr
r r
d x x
Ad d
r r
r r r r and iihile
old
T
old
T
old
T
T
T T
F
F
E
E
I
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 66/68
Conjugate Gradients - Convergence
8/8/2019 Steepest Decent and CG
http://slidepdf.com/reader/full/steepest-decent-and-cg 67/68
Conjugate Gradients - Convergence
A
i
Ai ee0
1
12 ¹
¹ º
¸©©ª
¨
e
O
O
top related