introduction to optimization -...

Institute of Transportation Engineers, Purdue Student Chapter engineering.purdue.edu/ITE

Introduction to Optimization of 18

Introduction to Optimization Updated September 19, 2010 • Compiled by Amit Kumar and Sushant Sharma

1. INTRODUCTION TO OPTIMIZATION AND POSSIBLE APPLICATIONS...................................... 2

PROBLEM FORMULATION ................................................................................................................... 2

2. QUICK REVIEW OF CALCULUS AND LINEAR ALGEBRA .......................................................... 3

FIRST DERIVATIVE ............................................................................................................................. 4

SECOND DERIVATIVE ......................................................................................................................... 4

MAXIMUM AND MINIMUM OF A FUNCTION ........................................................................................... 5

CONVEX AND CONCAVE FUNCTION ...................................................................................................... 5

CONVEXITY OF A REGION .................................................................................................................... 6

3. LINEAR OPTIMIZATION ..................................................................................................... 7

GRAPHICAL SOLUTION ....................................................................................................................... 7

SIMPLEX METHOD ............................................................................................................................ 8

4. NON-LINEAR OPTIMIZATION ............................................................................................ 10

INTRODUCTION TO NON-LINEAR OPTIMIZATION .................................................................................... 10

UNCONSTRAINED ONE DIMENSIONAL OPTIMIZATION .............................................................................. 12

Interval reduction methods ................................................................................................... 12

Curve-fitting methods ............................................................................................................ 15

5. MULTI-OBJECTIVE OPTIMIZATION .................................................................................... 16

PARETO OPTIMAL SOLUTION ............................................................................................................ 17

CLASSICAL METHOD TO SOLVE MULTI-OBJECTIVE PROBLEMS: METHOD OF WEIGHING OBJECTIVES ............... 17



1. Introduction to optimization and possible applications

Optimization: Optimization can be defined as the finding the best solution to a mathematical

problem having an objective function to be maximized or minimized within the set of

constraints if any.

So an optimization problem will have three components-

Objective function and Goal (min or max)

Decision variables

Set of constraints

Example 1:

Apple manufactures multiple products including iphone and ipad. It has three manufacturing

bases. The products are assembled at assembling unit by the component parts arriving from

three manufacturing bases.

Following table describes the percentage of capacity available at the manufacturing bases due

to the other products being developed at same location. The table also shows percentage of

capacity required per million units production of iphone and ipad, in addition to the profit.

Table 1: Capacity of manufacturing base for iphone and ipad

Capacity Used (per million unit)

Manufacturing base iphone ipad Capacity Available (per million unit)

California 1 0 4

Illinois 0 2 12

Atlanta 3 2 18

Profit (in hundred million USD per million unit)

3 5

Problem Formulation

Objective function:

x1 = number of iphones (in million) and x2 = number of ipads (in million)

Goal : To maximize profit (Z)

Or,



Decision variable:

Constraints: See Table 1 to understand the constraints

Areas in which the concept of optimization are being used-

Inventory control (Supply chain)

Project planning

Production planning and scheduling

Accounting

Transportation and logistics

Financing and Marketing

Forecasting and market planning

Quality control

Types of optimization problem-

Constrained and un-constrained

Linear and non-linear

Single objective and multi-objective

2. Quick review of calculus and linear algebra

Derivative of a function: In calculus the derivative is a measure of how a function changes as its input changes. A derivative can be thought of as how much one quantity is changing in response to changes in some other quantity. Mathematically, derivative (m) of function y=f(x) is represented by:



First Derivative First derivative is equal to the slope of the tangent at the selected point for a given function. Mathematically,

Figure 1: Slope of a function

Example: For a function ƒ(x) = x²

Then we calculate the limit by letting h go to zero:

Hence the slope of the graph of the squaring function at the point (3, 9) is 6, and so its

derivative at x = 3 is ƒ '(3) = 6. It can be also calculated analytically by knowing that f’(x)=2x.

Second Derivative



Maximum and minimum of a function

Figure 2: Global maximum and minimum and Local maximum and minimum of a function.

Convex and Concave Function A function is midpoint convex on an interval C if

A function is called strictly convex if



Figure 3: Convex function on an interval

Convexity of a region

In Euclidean space, a region is convex if for every pair of points within the region, every point on the straight line segment that joins them is also within the region. A function is convex if and only if the region (fig4c) above its graph is a convex set.

Fig-4a: Convex region Fig-4b: Non-convex region Fig-4c: Convex function

Figure 4: Examples of Convex Region and Convex Function (Source :http://en.wikipedia.org/wiki/Convex_function)



3. Linear optimization Those optimization problems fall under this category which has both the objective function and constraints as the linear functions of the design variables. The general forms of such problems are as below:

maximize c1x1+ + cnxn

subject to ai1x1+ + ainxn bi (inequality constraints)

aj1x1+ + ajnxn = bj (equality constraints) The problem formulated in the introduction section involving manufacturing decision is an example of linear optimization problem. We will now solve the problem formulated in the introduction section as example 1.

Graphical Solution

Step1: Initially note the non negative restrictions for the problem x1 x2 Then

observe the constraint x1 that means x1 cannot lie right of line x1=4.

Step 2: Similarly, says that feasible region is below line x2=6.

Step 3: The constraint means the plotting of the permissible region as line

. This can be done by either assuming x1=0 which implies

as one extreme point. Similarly other extreme can be located by assuming x2=0 which

implies x1=6 as extreme.

Step 4: We pick out the point in the region that maximizes the value of .

This can be done by trial and error. Let’s assume Z=15 and draw a line .

Assume x1=0 which means x2=3 and similarly assume x2=0, which means x1=5. Draw a line

passing through these points.

Step 5: Finally we draw lines parallel to this region in increasing direction of Z such that it is at

greatest distance from origin and at least one point within the feasible region passes through it.

By doing so the line passes through (2, 6) which represents the maximum value of Z is 3

(2)



Figure 5: Solving the linear problem by graphical method

Simplex Method

The simplex method is an algorithm which leads to final solution of linear programming

problem.

The structure of algorithm is

Initialization

Iteration

Stopping rule Final Solution

Working of this algorithm can be shown graphically as in figure 6.



Figure 6: Graphical demonstration of working of Simplex Method

The point of intersection of the constraint lines represents corner point solutions of the

problem. The feasible regions – (0,0), (4,0) , (2,6), (4,3) and (0,6) – are called corner point

feasible solutions.

In graphical terms the working of simplex algorithm can be shown as

1. Initialization Step: Start at a corner point feasible solution.

2. Iterative Step: Jump to a better adjacent corner-point feasible solution which gives

maximum value of objective function. Repeat it.

3. Optimality test: The current corner point feasible solution is optimal when none of its

adjacent corner point feasible solutions are better.

Working of algorithm for example 1

Step 1: Initialization Step: Start at (0, 0)

Step 2: Iteration 1: Jump to a better adjacent corner-point feasible solution (0, 0) to

(0,6)

Iteration 2: Move from (0, 6) to (2, 6)

Step 3: Optimality test: No solution adjacent to (2, 6) gives the best solution.

For more details on the simplex method please refer [Ref 1]



4. Non-linear optimization

Introduction to non-linear optimization

Nonlinear programming (NLP) is the process of solving a system in which an objective function is to be maximized or minimized with equalities and inequalities constraints provided some of the constraints or the objective function are nonlinear.

subject to, for i= 1,2,…….m

There is no single algorithm which will solve all sets of problems in this format [Ref 1].

Figure 7a shows the change in the problem solved before when second and third functional constraints are replaced by single non linear constraint

Now comparing this to the example presented in linear case, it is clear that although the optimal solution is still the same (2, 6) but, it is not the corner point, and it lies on the boundary of feasible region.

Figure 7a: Apple production example with non linear constraint



In the same problem as in example 1 if we consider the same linear constraint but change objective function to a non linear form. Let’s assume

Then the graphical representation of the problem can be shown as in figure 7b.

Figure 7b: Apple production example with non linear objective function

The optimal solutions is (8/3, 5), which again lies on the boundary of feasible region. It is important to notice that locus of all feasible points with Z= 857 intersects the feasible point at only this one point and locus of any larger value Z should not intersect the feasible region at all.

More importantly, in nonlinear programming a local maximum need not to be global maximum and similarly local minimum need not to be global minimum as shown in figure 8. The non linear programming algorithms generally are not able to distinguish between a local maximum and global maximum. So the conditions under which any local maximum is guaranteed to be a global maximum over the feasible conditions become crucial and should be known.



Figure 8: A Typical example of local and global maxima/minima in non linear objective function

Here we are going to deal only with unconstrained optimization. In the non-linear optimization case we generally present the problem as a minimization problem. Hence, if the goal is the maximization then we take the negative of the objective function and make it a (and change the goal to) minimization problem.

Unconstrained one dimensional optimization

Under the category of unconstrained non-linear optimization problem the simplest type is the problems that involves just one design variable and termed as one dimensional optimization. Here we include two basic approaches for solving such problems. The first is known as interval reduction methods and second is termed as curve fitting methods.

Interval reduction methods

Interval reduction methods involve an iterative approach in which each iteration aims at reducing the interval in which lies the optimum value of design variable. The iterations are continued until the minimizer (optimum design variable) is boxed in with sufficient confidence interval. Here two basic assumptions are required. First, the initial range or interval for the design variable is known and second, in the given interval, objective function is unimodal (has only one local minimum). Some popular methods under this category are:

Golden section method

Fibonacci method

Bisection method



Golden Section Method

In golden section method the interval of design variable is reduced by a constant ratio while requiring a single function evaluation in each iteration (except the first iteration) as shown in the figure 9. Function is evaluated at two intermediate points xL and xR in the first iteration. The distance between a and xL as well as between xR and b is same and is equal to ρ times the distance between a and b on the number line. Now out of two intermediate points which ever gives higher value of function becomes the end point of new interval and other intermediate point remains as the next intermediate point. Hence only one new intermediate point is found next iteration onwards. This method reduces the interval by a factor of (1- ρ) in each iteration. The complete sequence of steps of this method has been presented by the flow chart in figure 10.

Figure 9: Scheme of golden section method

In order to use the previous intermediate point in the next iteration to reduce the number of function evaluations as well as maintaining the constant ratio following condition should be satisfied

ρ (1- ρ)=1-2ρ Solving this with the condition that ρ<0.5 we get the value of ρ=(3-√5)/2≈0.382.



Figure 10: Flow chart of golden section method

Fibonacci method

This method is similar to golden section method but only difference that ρ is not constant but varies with the number of iteration. Reader is requested to refer text of optimization for further reading about this method.

Bisection method

Bisection method is useful when the derivative of the function to be minimized can be evaluated easily. The method of bisection is based on the fact that a ditonic function is monotonic on each side of the minimum given that it is unimodal in the given interval. Under such condition, the derivative of the objective function (to be minimized), dz(x)/dx, is negative for x < x* and positive for x > x*. In each iteration the first derivative (slope) of the objective function is calculated at the midpoint of the interval. If the slope at this point is positive that means x*>midpoint, hence, this midpoint should be new right limit of the new interval. On the other hand if the slope of function at the midpoint is negative it means that x*>midpoint, and hence the midpoint becomes the new left limit of the new interval. The scheme of this method is presented in figure 11.



Figure 11: Scheme of bisection method

Curve-fitting methods

Curve fitting methods approximates the non-linear objective function by lower degrees curve at a known point and then solves the approximated objective function to find the next optimal point. These methods also work iteratively till the stopping criterion is met. The stopping criteria can be relative or absolute change in design variable or objective function. When this change is less than a threshold value the iterations are stopped. Most famous method under this category is the Newton’s method and is explained here.

Newton’s Method

This method is useful when the first and second derivative of the objective function can be easily calculated in addition to the function evaluation. In this method given function is approximated by a quadratic function at the current point. Then instead of minimizing is original objective function the approximation is minimized. Let, the given objective function along with its first and second derivates are represented as Z(x), Z’(x) and Z’’(x) respectively. Then the it’s quadratic approximation at current point xk is represented as

Now, instead of minimizing the Z(x) we minimize its approximation q(x). We know that at the minimum point the slope of the objective function should be zero. Hence at the minimum of the function q(x), its slope (first derivative) will be zero. Hence,

Solving this equation we get new point which is better approximation to minimum point than the current point. Representing the solution of this equation by xk+1 we get



Above expression used iteratively to find the better approximation of minimizer point from the current point till the stopping criterion is met. Limitations of the Newton’s method- In order to start the first iteration of Newton’s method we need a good starting point. This starting point is very important for getting faster solution. In addition to this method may fail if the second derivative of the objective function is negative for some x.

5. Multi-objective optimization Many real world problems involve simultaneous optimization of multiple objectives. Multi- objective optimization problems can be found in various fields: product and process design, finance, aircraft design, the oil and gas industry, automobile design, or wherever optimal decisions need to be taken in the presence of trade-offs between two or more conflicting objectives. Some examples can be like: Maximizing profit and minimizing the cost of a product; maximizing performance and minimizing fuel consumption of a vehicle; and minimizing weight while maximizing the strength of a particular component, all these represent multi-objective optimization problems. The difference between single objective optimization and multi-objective optimization is that while in the former case the optimal solution is the one that has global minimum or maximum of a particular objective function; whereas in later case there may not exist a single solution that may be best (global minimum or maximum) with respect to all objectives. In multi-objective optimization there exists a set of solutions which are superior to rest of the solutions in search space when all objectives are considered. However, these solutions may be inferior to other solution if only one or more objectives are considered [Ref 2]. These solutions are known as Pareto-optimal solutions. A general multi-objective problem is consists of various objectives and equality and inequality constraints.

/ ( ) 1,2,3, ,

( ) 0 1,2,3, ,

( ) 0 1,2,3, ,

i

k

j

Maximize Minimize f x i N

Subject to

g x k K

h x j J



Pareto Optimal Solution

The figure below illustrates the concept of Pareto optimality. From the figure the Pareto optimal solutions are all values of x between 3 and 6. The solution x= 3 is optimal with respect to f1 (x) but not so good with respect to f2 (x). Similarly the solution x= 6 is optimal for f2 (x) but not for f1 (x). Any point in between these points is a tradeoff between two objectives and is a Pareto optimum point. However the solution beyond each of these points are not Pareto optimum points as these points are not better than points within the region x=3 to 6 with respect to either objectives

Figure 12: Representation of Pareto optimal solution for two objective functions

Classical Method to Solve Multi-Objective Problems: Method of Weighing

Objectives

In this method multiple objectives are combined into one by adding them:

1

( )N

i i

i

Z w f x

Where weight iw lies between 0 and 1 and sum of weights across all objectives is typically

1

N

i

i

w

=1. The solution can be obtained by putting up equal weights with each objective or as in



case of real world problems the weight can be used to represent the priority given to each objective. However this method suffers from few drawbacks.

1.) Fixing the values of the weights for accounting all the objectives is in itself a difficult task and may require considerable experience.

2.) There is always a need to normalize both objectives as it is quite possible that the magnitude of one of the objective compared to other might be quite high.

3.) For getting a solution for each set of weights different runs of the model are required which may be time consuming.

4.) The method of solving multi-objective optimization by converting to single objective with weights attached, suffers from a major drawback that it cannot find certain Pareto optimal solutions in the case of non convex objective space [Ref 3].

Figure 13: Pareto optimal solutions marked with continuous curves for combinations of two types of objectives. [Ref 3]

References [1]Hillier F. S. and Lieberman G. J. Introduction to Operations Research, McGraw-Hill,2002. [2]Srinivas, N and Deb, K, (1994) “Multi-objective optimization using Non Dominated Sorting Genetic Algorithm”, Journal of Evolutionary Computation, Vol. 2, No. 3, Pages 221-248.

[3]Deb, K. Multi-Objective Optimization using Evolutionary Algorithms, John Wiley and Sons, Chichester, UK, 2001.

introduction to optimization -...

Documents