linear nonlinear programming -- basic properties of solutions and algorithms

Linear & Nonlinear Programming --Linear & Nonlinear Programming --Basic Properties of Solutions and AlgorithmsBasic Properties of Solutions and Algorithms

22

OutlineOutline

First-order Necessary ConditionFirst-order Necessary Condition Examples of Unconstrained ProblemsExamples of Unconstrained Problems Second-order ConditionsSecond-order Conditions Convex and Concave FunctionsConvex and Concave Functions Minimization and Maximization of Convex Minimization and Maximization of Convex

FunctionsFunctions Global Convergence of Descent AlgorithmsGlobal Convergence of Descent Algorithms Speed of ConvergenceSpeed of Convergence

33

IntroductionIntroduction Considering optimization problem of the formConsidering optimization problem of the form

where where f f is a real-valued function and is a real-valued function and Ω, the Ω, the feasible set, is a subset of feasible set, is a subset of EEnn..

minimun ( )f xsubject to x

44

Weierstras TheoremWeierstras Theorem

If If ff is continuous and is continuous and ΩΩ is compact, a is compact, a solution point of minimization problem solution point of minimization problem exists. exists.

55

Two Kinds of Solution Points Two Kinds of Solution Points

Definition of a relative minimum pointDefinition of a relative minimum point– A pointA point is said to be a relative minimum is said to be a relative minimum

point of point of ff over over Ω if there is an such that Ω if there is an such that ff((xx) ≥ ) ≥ ff((xx**)) for all for all within a distance of within a distance of xx**. . If If ff((xx) ) >> ff((xx**)) for all for all , , xx≠≠xx**, within a distance of , within a distance of xx**,, then then xx** is said to be a is said to be a strictstrict relative minimum point of relative minimum point of ff over . over .

Definition of a global minimum pointDefinition of a global minimum point– A pointA point is said to be a global minimum point is said to be a global minimum point

of of ff over over Ω if Ω if ff((xx) ≥ ) ≥ ff((xx**)) for all . If for all . If ff((xx) ) >> ff((xx**)) for all for all , , xx≠≠xx**, then , then xx** is said to be a is said to be a strictstrict global minimum global minimum point of point of ff over . over .

x*0

x

*x

x

x

x

66

Two Kinds of Solution Points (cont’d)Two Kinds of Solution Points (cont’d)

We can achieve relative minimum by using We can achieve relative minimum by using differential calculus or a convergent differential calculus or a convergent stepwise procedure. stepwise procedure.

Global conditions and global solutions can, Global conditions and global solutions can, as a rule, only be found if the problem as a rule, only be found if the problem possesses certain convexity properties that possesses certain convexity properties that essentially guarantee that any relative essentially guarantee that any relative minimum is a global minimum. minimum is a global minimum.

77

Feasible DirectionsFeasible Directions

To derive necessary conditions satisfied by a relTo derive necessary conditions satisfied by a relative minimum point ative minimum point xx**, the basic idea is to consi, the basic idea is to consider movement away from the point in some giveder movement away from the point in some given direction. n direction.

A vector A vector dd is a feasible direction at is a feasible direction at xx if there is a if there is an n such thatsuch that for all for all αα, . , . x d 0 0

88

Feasible Directions (cont’d)Feasible Directions (cont’d)

Proposition 1 ( first-order necessary conditions)Proposition 1 ( first-order necessary conditions)– Let Let Ω be a subset of Ω be a subset of EEn n and let be a function on Ω. and let be a function on Ω.

If If x*x* is a relative minimum point of is a relative minimum point of ff over Ω, then for any over Ω, then for any that is a feasible direction at that is a feasible direction at x*x*, we have , we have

Proof :Proof :

1f C

nEd ( ) 0f x* d

( )Define ( ) ( ( )) By the ordinary calculus we have

( ) - (0) (0) ( )If (0) 0 then for any sufficintly small 0 ,we will get ( ) - (0) 0, Thus (0) ( ) 0

contradiction !.

g fg g g o

gg g

g f

x x* dx

x* d

99

Feasible Directions (cont’d)Feasible Directions (cont’d)

Corollary ( unconstrained case )Corollary ( unconstrained case )– Let Let Ω be a subset of Ω be a subset of EEn n and let be a function on Ω. and let be a function on Ω.

If If x*x* is a relative minimum point of is a relative minimum point of ff over Ω and over Ω and x*x* is an is an interior point of Ω, then interior point of Ω, then . .

Since in this case Since in this case dd can be any direction from can be any direction from x*x*, and hence , and hence for all . This impliesfor all . This implies . .

1f C

( ) 0f x*

nEd( ) 0f x* d ( ) 0f x*

1010

Example 1 about Feasible DirectionsExample 1 about Feasible Directions

Example 1 ( unconstrained )Example 1 ( unconstrained ) minimize minimize

There are no constrains, so There are no constrains, so Ω =Ω = E Enn

These have the unique solution These have the unique solution xx11=1,=1, x x22=2, so it is a =2, so it is a global minimum of global minimum of ff . .

2 21 2 1 1 2 2 2( , ) 3f x x x x x x x

1 21

1 22

2 0

2 3 0

f x xxf x xx

1111

Example 2 about Feasible DirectionsExample 2 about Feasible Directions

Example 2 (a constrained case)Example 2 (a constrained case)minimize minimize subject to subject to xx11 ≥ 0, ≥ 0, xx22 ≥ 0. ≥ 0.

since we know that there is a global minimum at since we know that there is a global minimum at xx11=1/2=1/2, , xx22=0=0, then , then

21 2 1 1 2 1 2( , )f x x x x x x x

1 21

12

2 1 0

312

f x xxf xx

1 1 322

2 21 2

( ) 0 3/2d df ff dd dx x

x* d

1212

Example 2 (cont’d)Example 2 (cont’d)

322( ) 0f d x* d

x1

x2

0 (1/2, 0) 1 3/2 2

1/2

1

3/2

d = (d1 , d2)

Feasible region

1313

Example 3 of Unconstrained Example 3 of Unconstrained ProblemsProblems The problem faced by an electric utility when selecting its pThe problem faced by an electric utility when selecting its p

ower-generating facilities.ower-generating facilities.Its power-generating requirements are summarized by a cuIts power-generating requirements are summarized by a curve, rve, h(x)h(x), as shown in Fig.6.2(a), which shows the total hour, as shown in Fig.6.2(a), which shows the total hours in a year that a power level of at least s in a year that a power level of at least xx is required for ea is required for each ch xx..For convenience the curve is normalized so that the upper lFor convenience the curve is normalized so that the upper limit is unity. imit is unity. The power company may meet these requirements by ins-tThe power company may meet these requirements by ins-talling generating equipment, such as (1) nuclear or (2) coaalling generating equipment, such as (1) nuclear or (2) coal-fired, or by purchasing power from a central energy. l-fired, or by purchasing power from a central energy.

1414


Associated with type i ( i = 1,2 ) of generating equipment is Associated with type i ( i = 1,2 ) of generating equipment is a yearly unit capital cost a yearly unit capital cost bbii and a unit operating cost and a unit operating cost ccii. .

The unit price of power purchased from the grid is The unit price of power purchased from the grid is cc33..

The requirements are satisfied as shown in Fig.6.2(b), wheThe requirements are satisfied as shown in Fig.6.2(b), where re xx11 and and xx22 denote the capacities of the nuclear and coal-fir denote the capacities of the nuclear and coal-fired plants, respectively.ed plants, respectively.

1515


Fig.6.2 Power requirement curve

hours required h(x)

power (megawatts)1

(a)

hours required h(x)

power (megawatts)1

(b)

x1 x2

coalnuclea

rpurchas

e

1616


The total cost isThe total cost is

And the company wishes to minimize the set defined And the company wishes to minimize the set defined byby

xx1 1 ≥ 0 , ≥ 0 , xx2 2 ≥ 0 , ≥ 0 , xx11++xx22 ≤ 1 ≤ 1

1

1 2

1 1 2

1 2 1 1 2 2 1 0

1

2 3

( , ) ( )

( ) ( )

x

x x

x x x

f x x b x b x c h x dx

c h x dx c h x dx

1717


Assume that the solution is Assume that the solution is interiorinterior to the to the constraints, by setting the partial derivatives equal constraints, by setting the partial derivatives equal to zero, we obtain the two equations to zero, we obtain the two equations

which represent the necessary conditionswhich represent the necessary conditions In addition,In addition,

If If xx11=0, then equality (1) relax to =0, then equality (1) relax to ≥ 0≥ 0

If If xx22=0, then equality (2) relax to =0, then equality (2) relax to ≥ 0≥ 0

1 1 2 1 2 3 1 2

2 2 3 1 2

( ) ( ) ( ) ( ) 0 (1) ( ) ( ) 0 (2)

b c c h x c c h x xb c c h x x

1818

Second-order ConditionsSecond-order Conditions

Proposition 1 ( second-order necessary conditions )Proposition 1 ( second-order necessary conditions )– Let Let Ω be a subset of Ω be a subset of EEnn and let be a function on Ω. and let be a function on Ω.

if if x*x* is a relative minimum point of is a relative minimum point of ff over Ω, then for any over Ω, then for any which is a feasible direction at which is a feasible direction at x*x* we have we havei)i)ii) if ii) if , then , then

Proof :Proof :The first condition is just propotion1, and the second The first condition is just propotion1, and the second condition applies only if .condition applies only if .

2f C

nEd( ) 0f x* d

( ) 0f x* d 2 ( ) 0f Td x* d

( ) 0f x* d

1919

Proposition 1 (cont’d)Proposition 1 (cont’d)

Proof (cont’d) :Proof (cont’d) :

2 212

In the second conditon, introducting ( ) and ( ) ( ( )) as before,we have, in view of (0) 0,

( ) (0) (0) ( )If (0) 0 then for any sufficintly small 0 ,we w

g f g

g g g og

x x* dx

2

contradictill get ( ) - (0) 0,

Thus (0)

ion

( ) 0.

!g g

g f

Td x* d

2020

Example 1 about Proposition 1 Example 1 about Proposition 1 For the same problem as For the same problem as Example 2 of Section 6.1Example 2 of Section 6.1, we have , we have d d = (= (dd11, ,

dd22))

Thus condition (ii) of Proposition 1 applies only if Thus condition (ii) of Proposition 1 applies only if dd22 = 0. In that case = 0. In that case we have we have

so condition (ii) is satisfied.so condition (ii) is satisfied.

1 322

2

( ) 0 3/2d

f dd

x* d

2 2

1 1 1 2 11 2 2 2

2

2 1 2 2

2

11 21 1 1

1

22 1 0 0 2

1 0

( )

0 0

f fx x x x d

d ddf f

x x x x

ddd d d

f

d

Td x* d

2121

Proposition 2Proposition 2

Proposition 2 (unconstrained case)Proposition 2 (unconstrained case)– Let Let x*x* be an be an interiorinterior point of the set point of the set Ω, and suppose Ω, and suppose x*x*

is a relative minimum point over Ω of the function . is a relative minimum point over Ω of the function . Then Then i)i)ii) for all ii) for all dd, , . .

– It means that It means that FF((x*x*)), simplified notation of , simplified notation of , is , is positive semi-definite.positive semi-definite.

2f C

( ) 0f x*2 ( ) 0f td x* d

2 ( )f x*

2222

Example 2 about Proposition 2Example 2 about Proposition 2

Consider the problem Consider the problem

– If we assume the solution is in the If we assume the solution is in the interiorinterior of the of the feasible set, that is, if feasible set, that is, if xx11 > 0, > 0, xx22 > 0 > 0, then the first-order , then the first-order necessary conditions arenecessary conditions are

3 2 21 1 2 2

1 2

minimize ( 1, 2) 2subject to 0, 0

f x x x x x xx x

2 21 1 2 1 23 2 0, -x 4 0x x x x

2323

Example 2 (cont’d)Example 2 (cont’d) Boundary solution is Boundary solution is xx1 1 = = xx2 2 = 0 = 0 Another solution atAnother solution at xx1 1 = 6, = 6, xx2 2 = 9= 9

– If we fixed If we fixed xx11 at at xx11 = 6 = 6, then the relative minimum with , then the relative minimum with respect to respect to xx22 at at xx22 = 9 = 9..

– Conversely, with Conversely, with xx22 fixed at fixed at xx22 = 9 = 9, the objective attains a , the objective attains a relative minimum w.r.t.relative minimum w.r.t. xx11 at at xx11 = 6 = 6..

– Despite this fact, the point Despite this fact, the point xx1 1 = 6, = 6, xx2 2 = 9= 9 is not a relative is not a relative minimum point, because the Hessian matrix minimum point, because the Hessian matrix FF(x*)(x*)xx1 = 6, 1 = 6, xx2 = 92 = 9 is is not a positive semi-definite since its determinant is negative. not a positive semi-definite since its determinant is negative.

1 26, 9

18 -12( ) 0

-12 4x xDet F(x*)

2424

Sufficient Conditions for a Relative Sufficient Conditions for a Relative MinimumMinimum We give here the conditions that apply only to We give here the conditions that apply only to unconstrainedunconstrained

problems, or to problems where the minimum point is problems, or to problems where the minimum point is interiorinterior to the feasible solution.to the feasible solution.

Since the corresponding conditions for problems where the Since the corresponding conditions for problems where the minimum is achieved on a boundary point of the feasible set minimum is achieved on a boundary point of the feasible set are a good deal more difficult and of marginal practical or are a good deal more difficult and of marginal practical or theoretical value.theoretical value.

A more general result, applicable to problems with functional A more general result, applicable to problems with functional constrains, is given in Chapter10. constrains, is given in Chapter10.

2525

Proposition 3Proposition 3

Proposition 3 Proposition 3 (2-order sufficient conditions-unconstrained case)(2-order sufficient conditions-unconstrained case)

– Let be a function defined on a region in which the Let be a function defined on a region in which the point point x*x* is an interior point. Suppose in addition that is an interior point. Suppose in addition that

i)i) ii)ii) is positive definiteis positive definiteThen Then x*x* is a is a strictstrict relative minimum point of relative minimum point of ff..Proof :Proof : since is positive definite , there is an since is positive definite , there is an a a > 0 such that> 0 such that for all for all dd, . Thus by Taylor’s Theorem , . Thus by Taylor’s Theorem

2f C

( ) 0f x*F(x*)

F(x*)a 2Td F(x*)d d

12( ) ( ) ( ) ( 2) ( )f f o a o 2 2 2Tx* +d x* d F(x*)d d d d

2626

Convex FunctionsConvex Functions

DefinitionDefinition– A function A function ff defined on a convex set defined on a convex set Ω is said to be Ω is said to be convexconvex..

If, for every If, for every and every and every αα, 0 ≤ , 0 ≤ αα ≤ 1, there holds ≤ 1, there holds

If, for every If, for every αα, 0 <, 0 <αα < 1, and < 1, and xx11≠≠xx2, 2, there holdsthere holds

then then ff is said to be is said to be strictly convex.strictly convex.

1 2x ,x

( (1 ) ) ( ) (1 ) ( )f f f 1 2 1 2x x x x

( (1 ) ) ( ) (1 ) ( )f f f 1 2 1 2x x x x

2727

Concave Functions Concave Functions

DefinitionDefinition– A function A function gg defined on a convex set defined on a convex set Ω is said to Ω is said to

be be concaveconcave if the function if the function ff = = -g-g is convex. The is convex. The function function gg is is strictly concavestrictly concave if if -g-g is strict is strict convex.convex.

2828

Graphs of Strict Convex FunctionGraphs of Strict Convex Function

f

x1 x2αx1+(1-α)x2

2929

Graphs of Convex FunctionGraphs of Convex Function

f

x1 x2

αx1+(1-α)x2

3030

Graphs of Concave FunctionGraphs of Concave Function

f

x1 x2αx1+(1-α)x2

3131

GraphGraph of Neither Convex or Concave of Neither Convex or Concave

f

x1 x2

3232

Combinations of Convex FunctionsCombinations of Convex Functions

Proposition 1Proposition 1– Let Let ff11 and and ff22 be convex function on the convex be convex function on the convex

set set Ω. Then the Ω. Then the ff11 ++ ff22 is convex on Ω is convex on Ω..

Proposition 2Proposition 2– Let Let ff be a convex function over the convex set be a convex function over the convex set

Ω. Then Ω. Then aa ff is convex for any is convex for any aa ≥ 0. ≥ 0.

3333

Combinations (cont’d)Combinations (cont’d)

Through the above two propositions it follows Through the above two propositions it follows that a that a positivepositive combination combination aa1 1 ff11++aa1 1 ff22++……aam m ffmm of is again convex.of is again convex.

3434

Convex Inequality ConstrainsConvex Inequality Constrains

Proposition 3Proposition 3– Let Let ff be a convex function on a convex set be a convex function on a convex set Ω. Ω.

The set The set is a convex for is a convex for every real number every real number cc..

: , ( ) c f c x x x

3535

ProofProof

– Let Let , then , then and for 0 <and for 0 <αα< 1 ,< 1 ,

Thus Thus

, c1 2x x ( ) , ( )f c f c 1 2x x

2 1 2( ) (1 ) (( (1 ) ) )f ff c 1x xx x

1 2(1 ) c x x

3636

Properties of Differentiable Convex Properties of Differentiable Convex FunctionsFunctions Proposition 4 Proposition 4

– LetLet , then , then ff is convex over a convex set is convex over a convex set Ω if Ω if and only if and only if

for all for all

1f C

( ) ( ) ( )( - )f f f y x x y x, x y

3737

RecallRecall

The original definition essentially states that linear The original definition essentially states that linear interpolation between two points overestimates the interpolation between two points overestimates the function,function,

while here stating that linear approximation based while here stating that linear approximation based on the local derivative underestimates the function. on the local derivative underestimates the function.

3838

Recall (cont’d)Recall (cont’d)

ff is a convex function between two points is a convex function between two points

f

x1 x2αx1+(1-α)x2

3939

Recall (cont’d)Recall (cont’d)

f(y)

xy

( ) ( )( )f f x x y x

4040

Two Continuously Differentiable Two Continuously Differentiable FunctionsFunctions Proposition 5Proposition 5

– Let Let , then , then ff is convex over a convex set is convex over a convex set Ω Ω containing an interior point if only if the Hessian containing an interior point if only if the Hessian matrix F of matrix F of ff is positive semi-definite through Ω. is positive semi-definite through Ω.

2f C

4141

Proof Proof

By Taylor’s theorem we haveBy Taylor’s theorem we have

for some for some αα, 0 , 0 ≤≤ αα ≤≤ 1. 1.if the Hessian is everywhere positive semi-definite, if the Hessian is everywhere positive semi-definite, we have we have

12( ) ( ) ( )( - ) ( ) ( ( ))( )f f f Ty x x y x y x F x y x y x

( ) ( ) ( )( - )f f f y x x y x

4242

Minimization and Maximization of Minimization and Maximization of Convex FunctionsConvex Functions Theorem 1Theorem 1

– Let Let ff be a convex function defined on the convex set be a convex function defined on the convex set Ω, Ω, then the setthen the set where where ff achieves its minimum is convex, achieves its minimum is convex, and any relative minimum of and any relative minimum of ff is a global minimum. is a global minimum.

Proof (contradiction)Proof (contradiction)

we assume that is the relative minumumanother point with ( ) ( )on the line (1- ) ,0 1 we have

( (1 ) ) ( ) (1 ) ( ) ( )contradicting the fact that is a relative minim

y f f

f f f f

x*y x*

y x*y x* y x* x*

x * un point

4343

Minimization and Maximization of Minimization and Maximization of Convex Functions (cont’d)Convex Functions (cont’d) Theorem 2Theorem 2

– Let Let be a convex on the convex set be a convex on the convex set Ω. If Ω. If there is a pointthere is a point such that, for all such that, for all

then then x*x* is a global minimum point of is a global minimum point of ff over Ω. over Ω. ProofProof

1f Cx*

, ( )( ) 0f y x* y x*

( ) ( )( )since ( )( ) 0so, is global minimum poi

( )

t

( )

n

f ff ff

x* x* y x*x* y x*

x *

y x*

4444

Minimization and Maximization of Minimization and Maximization of Convex Functions (cont’d)Convex Functions (cont’d)

Theorem 3Theorem 3– Let Let ff be a convex function defined on the bounded, be a convex function defined on the bounded,

closed convex set closed convex set Ω. If Ω. If ff has a has a maximummaximum over Ω,over Ω, then it is achieved at an then it is achieved at an extreme pointextreme point of of Ω.Ω.

4545

Global Convergence of Descent Global Convergence of Descent AlgorithmsAlgorithms A good portion of the remainder of this book is A good portion of the remainder of this book is

devoted to presentation and analysis of various devoted to presentation and analysis of various algorithms designed to solve nonlinear programming algorithms designed to solve nonlinear programming problems. However, they have the common heritage problems. However, they have the common heritage of all being of all being iterative descentiterative descent algorithms. algorithms.

IterativeIterative– The algorithm generated a series of points, each point The algorithm generated a series of points, each point

being calculated on the basis of the points preceding it.being calculated on the basis of the points preceding it. DescentDescent

– As each new point is generated by the algorithm the As each new point is generated by the algorithm the corresponding value of some function (evaluated at the corresponding value of some function (evaluated at the most recent point) decreases in values. most recent point) decreases in values.

4646

Global Convergence of Descent Global Convergence of Descent Algorithms (cont’d)Algorithms (cont’d) Globally convergentGlobally convergent

– If for arbitrary starting points the algorithm is If for arbitrary starting points the algorithm is guaranteed to generate a sequence of points guaranteed to generate a sequence of points converging to a solution, then the algorithm is converging to a solution, then the algorithm is said to be globally convergent.said to be globally convergent.

4747

Algorithm and Algorithmic MapAlgorithm and Algorithmic Map

We formally define an algorithm We formally define an algorithm AA as a mapping ta as a mapping taking king pointspoints in a space in a space XX intointo other other pointspoints in in XX, then , then the generated sequence the generated sequence xxkk defined by defined by

– With this intuitive idea of an algorithm in mind, we now gWith this intuitive idea of an algorithm in mind, we now generalize the concept somewhat so as to provide greateeneralize the concept somewhat so as to provide greater flexibility in our analysis.r flexibility in our analysis.

DefinitionDefinition– An algorithm An algorithm AA is a mapping defined on a space is a mapping defined on a space XX that that

assigns to every pointassigns to every point a subset of a subset of XX..

1 ( )k k x A x

Xx

4848

MappingsMappings

GivenGiven the algorithm yields the algorithm yields AA((xxk k )) which is a which is a subset of subset of XX. From this subset . From this subset an arbitrary elementan arbitrary element xxk+1k+1 is selected. In this way, given an initial point is selected. In this way, given an initial point xx00, , the algorithm generates sequences through the itethe algorithm generates sequences through the iterationration

The most important aspect of the definition is that tThe most important aspect of the definition is that the mapping he mapping AA is a is a point-to-setpoint-to-set mapping of mapping of XX. .

k Xx

1 ( )k k x A x

4949

Example 1Example 1

Suppose for Suppose for xx on the real line we define on the real line we define

so that so that AA((xx) is an interval of the real line. Starting at ) is an interval of the real line. Starting at xx00 = 100, each of the sequences below might be = 100, each of the sequences below might be generated from iterative application of this generated from iterative application of this algorithm.algorithm.100, 50, 25, 12, -6, -2, 1, 1/2,…100, 50, 25, 12, -6, -2, 1, 1/2,…100, -40, 20, -5, -2, 1, 1/4, 1/8,…100, -40, 20, -5, -2, 1, 1/4, 1/8,…100, 10, 1/16, 1/100, -1/1000,…100, 10, 1/16, 1/100, -1/1000,…

( ) [ | | / 2,| | / 2]A x x x

5050

Descent Descent

DefinitionDefinition– LetLet be a given solution set and let A be an be a given solution set and let A be an

algorithm on X. A continuous real-valued algorithm on X. A continuous real-valued functions Z on X is said to be a descent functions Z on X is said to be a descent function for and A if it satisfiesfunction for and A if it satisfies

X

i) and ( ), then ( ) ( )ii) and ( ), then ( ) ( )

Z ZZ Z

x y A x y xx y A x y x

5151

Closed MappingClosed Mapping Definition Definition

– A point-to-set mapping A from X to Y is said to be closed A point-to-set mapping A from X to Y is said to be closed atat if the assumptionsif the assumptions

The point-to-set map A is said to be closed on X if it is The point-to-set map A is said to be closed on X if it is closed at each point of X.closed at each point of X.

Xx

i) , ,ii) , ( )implyiii) ( )

k k

k k k

X

x x xy y y A x

y A x

5252

Closed Mapping in Different SpaceClosed Mapping in Different Space

Many complex algorithms are regards as the Many complex algorithms are regards as the composition of two or more simple point-to-point composition of two or more simple point-to-point mappings.mappings.

DefinitionDefinition– LetLet be point-to-set mappings. The be point-to-set mappings. The

composite mapping composite mapping C = BAC = BA is defined as the point-to-set is defined as the point-to-set mapping mapping

The definition is illustrated in Fig. 6.6The definition is illustrated in Fig. 6.6

: and :X Y Y Z A B

: withX ZC

( )( ) ( ) y A xC x B y

5353

Fig. 6.6Fig. 6.6

A

BC

XY

Z

A(x)

B(x)C(x)

x y

5454

Corollaries of Closed MappingCorollaries of Closed Mapping

Corollary 1Corollary 1

Corollary 2Corollary 2

Let : and : be point-to-set mappings.If is closed at , is closed on ( ) and Y is compact,then the composite map is closed at

X Y Y Z

A BA x B A x

C BA x

point-to-pointcon

Let : be a mapping and :be a point-to-set mapping. If is at and isclosed on ( ), then the composite mapping

tin is

closed

s

at

uou

X Y Y Z

A BA x B

A x C BAx

5555

Global Convergence Theorem Global Convergence Theorem The Global Convergence Theorem is used to establish The Global Convergence Theorem is used to establish

convergence for the following situation.convergence for the following situation.

There is a There is a solution setsolution set . Points are generated according . Points are generated according to the algorithm to the algorithm , and each new point always , and each new point always strictly decreases a strictly decreases a descent functiondescent function ZZ unless the solution unless the solution set is reached.set is reached.

For example, in nonlinear programming, the solution set For example, in nonlinear programming, the solution set may be the set of minimum points (perhaps only one may be the set of minimum points (perhaps only one point), and the descent function may be the objective point), and the descent function may be the objective function itself. function itself.

1 ( )k k x A x

5656

Global Convergence Theorem Global Convergence Theorem (cont’d)(cont’d) Let A be an algorithm on X, and suppose that, given Let A be an algorithm on X, and suppose that, given xx00 the s the s

equence equence is generated satisfying is generated satisfying

let a solution setlet a solution set be given, and supposebe given, and suppose

Then the limit of any convergence subsequence of Then the limit of any convergence subsequence of xxkk is a is a solutionsolution

0 k kx

1 ( )k k x A xX

i) all points are contained in a set ii) there is a continuous function Z on such that (a) if , then ( ) ( ) for all ( ) (b) if , then ( ) ( ) for a

com

ll (

pactk S XX

Z ZZ Z

x

x y x y A xx y x y A x

clo)

ii sedi) the mapping is at points outside A

5757

Global Convergence Theorem Global Convergence Theorem (cont’d)(cont’d) Corollary Corollary

– If under the conditions of the Global Convergence TheorIf under the conditions of the Global Convergence Theorem consists of a single point , then the sequence em consists of a single point , then the sequence xxkk converges to . converges to .

xx

5858

Examples to Illustrate ConditionsExamples to Illustrate Conditions

Examples 4Examples 4– On the real line consider the point-to-point algorithmOn the real line consider the point-to-point algorithm

and solution set ,descent function and solution set ,descent function

The condition (iii) does not holds since The condition (iii) does not holds since AA is is not closednot closed at at xx = = 11, so, the limit of any convergent subsequence of , so, the limit of any convergent subsequence of xxkk is not is not a solution .a solution .

0

12

12

( 1) 1 1( )

1x x

A xx x

( )Z x x

5959

Examples (cont’d)Examples (cont’d)

Example 5Example 5– On the real line X consider the solution On the real line X consider the solution

set , the descent function set , the descent function ZZ((xx) = ) = ee-x-x, and the , and the algorithm algorithm AA((xx) = ) = xx + 1. + 1.

All the conditions of the convergence theorem All the conditions of the convergence theorem holds except (i) since the sequence generated holds except (i) since the sequence generated from any starting point from any starting point xx00 diverges to infinity. diverges to infinity.

It means It means SS is is notnot a a compactcompact set. set.

6060

6.7 Order of convergence6.7 Order of convergence

DefinitionDefinition– Let the sequenceLet the sequence converge to the limit converge to the limit r*r* . .

The order of convergence of The order of convergence of rrkk is defined as t is defined as the supremum of the nonnegative numbers he supremum of the nonnegative numbers pp sa satisfyingtisfying

– Larger values of the order Larger values of the order pp imply faster conver imply faster convergence.gence.

0 k kr

*1

*

| |0 lim| |

kpk

k

r rr r

6161

Order of convergence (cont’d)Order of convergence (cont’d)

– If the sequence has orderIf the sequence has order p p and the limit and the limit

exists, then asymptotically we haveexists, then asymptotically we have

*1

*

| |lim| |

kpk

k

r rr r

* *1| | | |p

k kr r r r

6262

Examples Examples

Example 1Example 1– The sequence with The sequence with rrkk = = aakk where where 0 < 0 < aa < 1 < 1

SolutionSolution– It converges to zero with order unity, since It converges to zero with order unity, since

1 /k kr r a

6363


Example 2Example 2– The sequence withThe sequence with for for 0 < 0 < aa < 1 < 1

SolutionSolution– It converges to zero with order two, sinceIt converges to zero with order two, since

(2 )k

kr a

21 / 1k kr r

6464

Linear convergence Linear convergence

Most algorithm discussed in this book have an ordMost algorithm discussed in this book have an order of convergence equal to unity.er of convergence equal to unity.

Definition Definition – If the sequence If the sequence rrkk converges to converges to rr** in such way that in such way that

the sequence is said to converge linearly to the sequence is said to converge linearly to rr** with con with convergence ratio vergence ratio

*1

*

| |lim 1| |

k

kk

r rr r

6565

Linear convergence (cont’d)Linear convergence (cont’d)

The linearly convergence is sometimes referred to The linearly convergence is sometimes referred to as as geometric convergence.geometric convergence.

Since one with convergence ratio can be said tSince one with convergence ratio can be said to have a tail that converges at least as fast as the o have a tail that converges at least as fast as the geometric sequencegeometric sequence for some constant for some constant cc..

The smaller the ratio the faster the rate.The smaller the ratio the faster the rate. The ultimate case where is referred to as The ultimate case where is referred to as ss

uperlinear convergenceuperlinear convergence..

c

0

6666

Examples Examples

Example 3Example 3– The sequence with The sequence with rrkk = = 1/k1/k

Solution Solution – It converges to zero. The convergence is of ordIt converges to zero. The convergence is of ord

er one but it is not linear, sinceer one but it is not linear, sinceis not less than one. is not less than one.

1lim | / | 1k kkr r

6767

Examples (cont’d)Examples (cont’d) Example 4Example 4

– The sequence with The sequence with rrkk = (= (1/k)1/k)kk

Solution Solution – The sequence is of order unity, sinceThe sequence is of order unity, since

for for pp > 1. > 1.However, However,

and hence this is superlinear convergence.and hence this is superlinear convergence.

1lim | / | 0 1k kkr r

1lim( / )pk kk

r r

6868

Average ratesAverage rates

All the definitions given above can be referred to All the definitions given above can be referred to as as step-wisestep-wise concepts of convergence, since they concepts of convergence, since they define bounds on the progress made by going a define bounds on the progress made by going a single step : from single step : from kk to to kk+1.+1.

Another approach is to define concepts related to Another approach is to define concepts related to the the averageaverage progress per step over a large progress per step over a large number of steps. number of steps.

6969

Average rates (cont’d)Average rates (cont’d)

DefinitionDefinition– Let the sequence Let the sequence rrkk converge to converge to rr** . The aver . The aver

age order of convergence is the infimum of the age order of convergence is the infimum of the numbers numbers p p > 1 > 1 such that such that

The order is infinity if the equality holds for no The order is infinity if the equality holds for no p p > 1> 1

* 1/lim | | 1kp

kkr r

7070

Examples Examples

Example 5Example 5– For the sequenceFor the sequence , , 0 < 0 < aa < 1 < 1

SolutionSolution– for for pp = 2 = 2, we have ,, we have ,

for for pp > 2 > 2, we have, we haveThus the average order is two Thus the average order is two

(2 )k

kr a

1/ 2| |k

kr a1/ 2 (1/ ) (2 / )| | ( ) 1

k k k kp p pkr a a

7171


Example 6Example 6– For the sequence For the sequence rrkk = = aakk, with , with 0 < 0 < aa < 1 < 1

SolutionSolution– for for p p > 1> 1 ,we have ,we have

Thus the average order is unity. Thus the average order is unity.

1/ (1/ )( ) 1k kp k p

kr a

7272

Convergence ratio by average Convergence ratio by average methodmethod The most important case is that of unity order.The most important case is that of unity order. We define the average convergence ratio as We define the average convergence ratio as

Thus for the geometric sequence Thus for the geometric sequence rrkk = = cacakk, , for for 00 << a a << 11 the average the average convergenceconvergence ratio is ratio is a a ..

* 1/lim | | kkk

r r

linear nonlinear programming -- basic properties of solutions and algorithms

Documents