linear and nonlinear optimization - mnrlablinear and nonlinear optimization author islam s. m....

Linear and Nonlinear Optimization

Islam S. M. Khalil

German University in Cairo

Islam S. M. Khalil Hill Climbing using Newton-Raphson Method

Outline

Hill climbing

Newton-Raphson method

Newton-Raphson in optimization


Hill Climbing

In hill climbing, we start with an initial solution. We then generateneighboring solutions until there are no better solutions.

Figure: Function f (x1, x2) with a global maximum.


Hill Climbing

The initial solution affects the optimization problem.

Figure: Function f (x1, x2) with two local maxima.


Hill Climbing

The initial solution affects the optimization problem.

Figure: Function f (x1, x2) with two local minima and three local maxima.


Newton-Raphson Method

Newton-Raphson method is used in finding the roots of nonlinearfunctions

Figure: Function f (x) has two roots, shown by the red arrows.



Lets start with the following initial guess for the solution: x0 = 0.8.




Linearize the nonlinear function around the initial guess (x0 = 0.8).




Now linearize the nonlinear function around the solution (x1 = 0.1).




Now linearize the nonlinear function around the solution(x2 = −0.36).




After three iterations the solution the best solution is(x3 = −0.49).



Newton-Raphson in Optimization

But we are not after the roots of the function f (x), we are ratherinterested in its extrema.

Figure: Function f (x) and its first derivative f′(x).



So lets find the root of f′(x) instead of the roots of f (x).

Figure: Function f′(x) has one root.



Given a vector function f(x), where x = x1, . . . , xn. We expand thefunction using Taylor series around x0 as follows:

f(x) = f(x0) +∂f

∂x|x=x0 (x− x0) . (1)

Therefore, we obtain

Newton-Raphson

x = x0 −∂f

∂x

−1|x=x0f(x0). (2)

where

J(x) =∂f

∂x. (3)

is the Jacobian matrix and x0 is an initial guess to start theiterative solution of Newton-Raphson.



Newton-Raphson algorithm for finding the exterma of a functionf (x):

Define a convergence criterion (error tolerance).

Select a starting point xn.

Calculate xn+1 = xn − [∂f (xn)∂x ]−1f (xn).

Repeat the previous step till the convergence criterion issatisfied.



Example: Find the extrema of the function f (x) given by

f (x) = 3x3 − 10x2 − 56x + 5. (4)

Figure: Function f (x) has two extrema.



Figure: Function f (x) has two extrema.



Figure: Calculate the derivative of the function.



Figure: Select an initial guess xn.



Figure: Solve for xn+1.



Figure: Use the second derivative to check the nature of the extrema.



Example. Use Newton’s method to solve the nonlinear system:

f1 = 1− 4x + 2x2 − 2y3 (5)f2 = −4 + x4 + 4y + 4y4 (6)

Figure: Points of intersections are the solution.


Second Order Sufficient Conditions

First, we calculate the Jacobianmatrix J(x , y) as follows:

J(x , y) =

[∂f1∂x

∂f1∂y

∂f2∂x

∂f2∂y

](7)

J(x , y) =

[−4 + 4x −6y2

4x3 4 + 16y3

]Use the Newton-Raphson method tofind a numerical approximation tothe solution near x0 = (0.1, 0.7).

x = x0 −∂f

∂x

−1|x=x0f(x0). (8)

Figure: Points of intersectionsare the solution.



Example. Use Newton’s method to solve the nonlinear system:

f1 = 9x2 + 36y2 + 4z2 − 36 (9)

f2 = x2 − 2y2 − 20z (10)

f3 = x2 − y2 + z2 (11)

Figure: Superimposed functions.



f1 = 9x2 + 36y2 + 4z2 − 36 (12)

f2 = x2 − 2y2 − 20z (13)

f3 = x2 − y2 + z2 (14)

Figure: f1(x , y , z).



f1 = 9x2 + 36y2 + 4z2 − 36 (15)

f2 = x2 − 2y2 − 20z (16)

f3 = x2 − y2 + z2 (17)




f1 = 9x2 + 36y2 + 4z2 − 36 (18)

f2 = x2 − 2y2 − 20z (19)

f3 = x2 − y2 + z2 (20)



Thanks

Questions please


linear and nonlinear optimization - mnrlablinear and nonlinear optimization author islam s. m....

Documents