This content has been downloaded from IOPscience. Please scroll down to see the full text.

Download details:

IP Address: 54.39.106.173

This content was downloaded on 24/05/2020 at 22:13

Please note that terms and conditions apply.

You may also be interested in:

Inverse Imaging with Poisson Data: Optimization methods

M Bertero, P Boccacci and V Ruggiero

A parallel optimization method for product configuration and supplier selection based on interval

Jian Zheng, Meng Zhang and Guoxi Li

Improving of the working process of axial compressors of gas turbine engines by using an

optimization method

E Marchukov, I Egorov, G Popov et al.

Optimization method of power communication network based on improved particle swarm optimization

He Wang, Lin Peng, Zehao Zhang et al.

An optimization method for speech enhancement based on deep neural network

Haixia Sun and Sikun Li

A Fuzzy Satisfactory Optimization Method Based on Stress Analysis for a Hybrid Composite Flywheel

Liping Yang, Curran Crawford and Zhengyi Ren

Optimization Method of a Low Cost, High Performance Ceramic Proppant by Orthogonal Experimental

Design

Y Zhou, Y M Tian, K Y Wang et al.

Optimal Acoustic Attenuation of Weakly Compressible MediaPermeated with Air Bubbles

Liang Bin and Cheng Jian-Chun

An Optimization Method for Ultrasonic Beamforming with a Synthetic Aperture

Ayumu Matani and Kunihiro Chihara

IOP Publishing

Modern Optimization Methods for Science, Engineering and

Technology

G R Sinha

Chapter 1

Introduction and background tooptimization theory

G R Sinha

This chapter presents an overview and brief background of optimization methodswhich are very popular in almost all applications of science, engineering, technologyand mathematics. The historical background of optimization is studied, making adistinction between optimization and robustness. A few important definitions suchas design variables, constraints and optimization objectives are highlighted followedby optimization problems and methods. Structural, control, signal processing andother important optimization methods are introduced, and finally the importance ofthe mathematical background which is necessary for implementing optimizationmethods for various tasks is briefly presented.

1.1 Historical developmentOptimization is a much talked about topic that has long been challenging. Muchliterature suggests that the concept of optimization was used in 100 BC forcalculating the appropriate distance between two points. There are several suchliterature sources, articles and stories about optimization theory and practice. In thedevelopments and advancements in any area of science, engineering, technology,mathematics, philosophy and many others, an attempt is always made to achievebetter results, products or outcomes with better performance. The improvement ofperformance over existing methods, solutions, products, devices or any advances ofscience, technology, engineering and mathematics (STEM), is always studied bynumerous scholars and research groups. Research continues in all areas of STEMfor further improvement of the performance of existing work or discovering somebetter method. In fact, robustness and optimization are two important terms whichcan be considered as analogous because both are attempts to continuously improve.

doi:10.1088/978-0-7503-2404-5ch1 1-1 ª IOP Publishing Ltd 2020

Wikipedia suggests that the optimization problem was studied in the seventeenthand eighteenth centuries, including with regard to Kepler’s law, Bernoulli’s theorem,Leibniz’s calculus of variations, etc. In calculus, probability theory, mechanics,chemical equilibrium and many other fields, optimization theorems and principlesevolved in the nineteenth and twentieth centuries. In the twentieth century, thetraveling salesman problem, which is one of the most popular among optimizationproblems, was studied and worked upon. Now, optimization theory is not onlylimited to STEM and allied branches but the concept is widely used in all fields, suchas financial management, economics, life sciences, genetics, population studies andseveral others. We can use the evolution of human beings as an example, withhuman ancestors being more ape-like. Evolution then continued and human beingscan be seen in the present context as the best possible form of the species of its kindthat natural selection could produce. The human species is now educated and istrying become even faster through computing speed, although the thinking capacityof the brain is unlimited. Thought processes and their outcomes are being improvedcontinuously. This is a simple case of optimization. A person can become a betterintellectual or individual using some methods, thought processes or techniques—employing some type of optimization concept to perform better.

Let us understand optimization problems with the help of an example. A system isdefined by the following equations:

= +M ax bx (1.1)1 2

= +P Mx c, (1.2)1

where x1 and x2 are two input variables, a and b are constants, M is an intermediateoutput, c is one more constant and P is the final output or response of the system.

Now, with a given set of conditions we are interested in obtaining the optimumvalue of P, which can be achieved in two different ways:

• Optimizing system performance by optimizing the internal parameters andconstants which are characteristics of the system.

• Optimizing by adjusting or optimizing all individual variables or selectedvariables present in the system.

Optimization is expected to produce the best possible final value otherwise somemore optimization methods need to be applied to further improve the results—therearises the need for a robust approach that can suit all requirements and conditions.Another situation for using optimization in the above example may be due to someerror introduced in the final value. If the error in the final value is more than thetolerance limit (the allowed limit) then the error can be minimized by using asuitable optimization method.

1.1.1 Robustness and optimization

Optimization and robustness are strongly related in terms of a common objective,which is to achieve some improvement. Optimization always aims to improve

Modern Optimization Methods for Science, Engineering and Technology

1-2

something, some parameter, so that a method, system or device performs better. Theterm robustness relates to attempting a process more and more times since the solutioncan never be optimum, rather it can be optimal. So, researchers will be alwaysengaging in experiments, simulations, modeling and study so that a better result orperformance is obtained. Robustness has long been a challenging research problem inall fields of science and technology irrespective of application. This could be in the fieldof automobiles, manufacturing, computing, or information and communicationtechnology (ICT) enabled services, all of which need continuous improvement. If weare to discuss any particular example of robustness then let us discuss image processingand computer vision based applications [1, 2]. This is an important application becausecomputer vision is used in almost all modern advancements. The prominent authors ofimage processing admit that there is no general theory in image processing, whichmeans that whatever is done in a novel manner becomes new in the area of imageprocessing. Obtaining a robust approach for any components of image processing suchas image de-noising, segmentation or pattern matching is challenging, and it is verydifficult to claim that the obtained solution is robust one. The robustness depends onoptimizing one or a few or all the parameters involved in the research. The extensiveresearch on image processing suggests that robustness is a permanent researchproblem, a few research contributions can be seen in [1–6] where optimization ofperformance was attempted but not as a robust approach.

Combining the two important terms, robust optimization is an emerging area ofstudy and research where robustness is investigated among a number of optimizationproblems and solutions in real-time industries and practice. The optimization processemploys a number of methods for improving the performance of a particular taskand different optimization methods produce different performance metrics andtherefore robustness needs to be investigated among the optimization methods.The best possible optimization method that can result in optimum performance of asystem will be referred to as a robust optimization method. If the method is robustthen it should also be used for a variety of tasks irrespective of different performanceevaluation parameters.

1.2 Definition and elements of optimizationOxford Dictionary defines optimization as a process or a method that can makesomething perfect and effective1. Something here may include a design, system ordecision which becomes better and better using the methodology called optimiza-tion. Cambridge Dictionary defines optimization as process that makes something asgood as possible, or as effective as possible2. We can refer to many other standarddefinitions and find one thing common, which is the improvement in some process,method, design or decision to make something more effective.

Although optimization is defined as a process or methodology, it is actually madeup of several components, such as decision variables, constraints and objectives.

1 The definition is taken from: https://www.lexico.com/en/definition/optimization and slightly modified.2 The definition is taken from https://dictionary.cambridge.org/dictionary/english/optimization and slightlymodified.

Modern Optimization Methods for Science, Engineering and Technology

1-3

Variables are the most important and guiding factors deciding the result beingproduced by the optimization and constraints are certain conditions under whichoptimization achieves its desired goal (its objective).

1.2.1 Design variables and parameters

Variables are design parameters used in any system which are actually responsiblefor obtaining a desired output or a product. Generally, the design variables are set ofone-dimensional or multi-dimensional vectors, such as

= …X x x x x( , , , , ), (1.3)n1 2 3

where x1, x2, x3,…, are n number of variables and constitute together a single vectorX. Here, the vector X is one-dimensional but it can be two-dimensional, three-dimensional or more.

These variables are controlled, manipulated and varied in the process ofoptimization. In some cases, only one variable is varied or controlled and perform-ance improvement results and optimization works, but in other cases a greaternumber of variables may also change in order to achieve the desired objective in thesystem design. In the applications where the theorem of equilibrium or inequalitiesis used to control and obtain equilibrium, many variables are used. If optimization isdesired, then there is a wide range of possibilities to optimize the behavior of amechanical system [7]. We sometimes use the terms parameters and variablesinterchangeably, but there is a slight difference between the two. Variables arevectors or values which actually vary in the system, whereas parameters may besome standard operating parameters, constants or the attributes that interrelate thedesign variables. So, both of them are attributes, and their small difference lies in therole of parameters in obtaining some relationship between the variables.

1.2.2 Objectives

An objective means a goal to be achieved. Every STEM performance improvementwill have a certain goal to achieve, maybe an error to minimize at some specific level.In all such cases, optimization is used with the important purpose to achieve the goalthat is set by the system response or method. Thus, the optimization method isassigned a minimum goal that is required to be achieved. As an example, a fewimages are presented in figure 1.1 which are the results of breast cancer segmentationand detection. The results were obtained using an optimized method that combinedgray level clustering enhancement and adaptive threshold segmentation (GLCE-ATS). If we look at four different segmentation results which all highlight breastcancer masses, they appear the same and it is very difficult to determine the cases inwhich more cancerous elements (malignant elements) are present. The optimizationmethod has the objective to achieve the maximum number of malignant massespresent in the breast mammograms [8].

The objective of optimization is also to minimize the error or value of asuitable function e(x) or f(x), y = minimize {f(x)} or minimize {e(x)}.

Modern Optimization Methods for Science, Engineering and Technology

1-4

1.2.3 Constraints and bounds

Constraints are some limitations and conditions under which the optimizationtechnique or method works. The method successfully improves the system perform-ance provided a certain set of constraints are satisfied. The limitations include somebounds as well, which may be upper bounds, lower bounds or intermediate.

The optimization operates on a set of data or samples subjected to somelimitations that may depend on certain characteristics or features of the systembeing optimized, or some external factors affecting the system design or perform-ance. The bounds are generally based on design variables but may also depend onparameters. For example,

= < >Y e x x x x xminimize { ( )} under and . (1.4)p2 0 1

(a) (b)

(c) (d)

Figure 1.1. Results of breast cancer detection, with different results for different optimization parameters usedin the GLCE-ATS method.

Modern Optimization Methods for Science, Engineering and Technology

1-5

Here the goal is to minimize the value of the error signal. The minimum oroptimum value has to be obtained as Y and the optimization has to be achievedunder the conditions given in equation (1.4). These conditions are bounds which areapparently lower and upper bounds in the case of x2 and x1, respectively. The errorsignal or value needs to be minimized between x1 and x2. In different types ofoptimization problems and applications, various types of bounds and constraints areused. The optimization methods can be generalized but the constraints are specific tothe applications or problems being addressed. Therefore, the constraints aresubjective, whereas the optimization methods are objective for the applications. In[7], the solution to achieving equilibrium requires a number of constraints in theimplementation of optimization in the work.

1.3 Optimization problems and methodsIn applications related to STEM or allied fields, there are a variety of problems thatneed to be addressed in terms of improvement of system design and performance, orminimization of error, reduction of labor and cost, etc. These problems are solvedusing suitable optimization methods [9]. In the determination of optimizationproblems, several standard aspects are used, such as:

1. Optimal input number of arguments.2. Maximum and minimum values of functions.3. Characteristics of the system.4. Constraints and bounds.

The number of arguments or attributes plays an important role in choosing theoptimization problem. In a few cases, the optimization performs in such a mannerthat the computational complexity decreases with the increase in the number ofinput arguments, whereas in some other cases the complexity is also increased. Infact, the goal should also be to reduce the computational complexity in addition toimproved performance or system design. The number of arguments is to be selectedwisely so that optimization produces the best possible result while expending theleast possible time and fewest operations. The bounds are limits to certainconditions, which are also important points to be kept in mind while designing anoptimization method and implementing it. So, the problems are governed by theupper and lower limits of constraints and bounds, and also by the minimum andmaximum possible values of the functions being optimized [7, 9]. The targets foroptimization are also decided on the basis of specific characteristics of systemresponse, design or behavior.

The number of methods for optimization is large and these can be employed forsolving a variety of problems.

1.3.1 Workflow of optimization methods

Any optimization problem and its possible solution has some typical workflowdiagram, one such workflow for the execution of any general-purpose optimizationmethod is shown in figure 1.2. This figure highlights the major steps that are

Modern Optimization Methods for Science, Engineering and Technology

1-6

executed in the implementation of any general-purpose optimization method. Theprocess begins with the need for optimization, which is the very first step in anyapplication. The need for optimization helps in the identification of possibleoptimization problems. There may be multiple problems in an application thatcould be optimized using a single method or multi-method based optimization. Oncethe selection of problems is finally made, the design variables are selected whichfurther determine the main objective of employing a suitable optimization method.The optimization has to be carried out under certain constraints and boundaryvalues, and therefore before we actually choose an appropriate method, we setvarious design constraints and limitations.

Then, the optimization method is decided based on the design variables, require-ments, optimization problem, constraints and bounds. The results of thesuitable optimization method are assessed as to whether they are the best possibleresults which were expected or not. If not, then the workflow of the implementationmay have several reverse operations (the reverse direction arrows in the diagram arenot shown as they depend on what is actually needed). If the design variables needsome modification or manipulation, or similarly the constraints require somechanges along with set-in limit values, these aspects are corrected so as to achievethe best possible response or outcome produced by the optimization method. Thus,it can be seen in the flow diagram that the optimization problem is chosen in the

Need for Optimization

Selection of Optimization of Problem

Selection of Design Variables

Set Optimization Objective

Constraints and Bounds

Selection of suitable Optimization Method/algorithm

Obtain Best Possible Solution

Figure 1.2. Major steps of the workflow of a typical optimization method.

Modern Optimization Methods for Science, Engineering and Technology

1-7

beginning as per the need and the optimization method is selected after certain steps,such as defining design variables, constraints and bounds.

1.3.2 Classification of optimization methods

As we have discussed, the optimization method is decided based on several factors,and constraints may be linear as well as non-linear. Accordingly, the methods arebroadly categories based on:

• Linear constraint terms and bounds.• Non-linear constraint terms and bounds.

The methods of optimization will fall under different categories, but the basis ofchoosing a suitable method will be the above two factors, that is, the constraints andbounds being either linear or non-linear. The optimization methods are furtherclassified as:

• Single variable or multi-variable methods.• Constrained or non-constrained methods.• Linear or non-linear optimization methods.• Single objective or multi-objective optimization methods.• Specialized or general-purpose methods.• Traditional or non-traditional methods.

The classification of optimization methods is also based on the characteristics orproperties of the objective function of optimization [9]. For example, if controlvariables are continuous and real numbers, then the optimization method is calledcontinuous optimization. Similarly, for integers it is known as discrete optimizationand for a combination of continuous real numbers and integers the optimization isknown as mixed integer. Based on different types of design variables, quadratic,linear and non-linear, the optimization method is referred to accordingly asquadratic, linear or non-linear. If the optimization problem is subject to somelimitations or constraints then the optimization method is called a constrainedoptimization method and if there are no such constraints, the method is called anunconstrained optimization method.

Conventional or traditional methods of optimization include mainly linearprogramming, non-linear programming, separable programming and integerprogramming optimization methods. The methods which are based on evolu-tionary concepts are referred to as evolutionary methods of optimization, whichinclude a few main methods such as the heuristic search, genetic algorithm,evolutionary programming, particle swarm optimization (PSO), simulated anneal-ing, bacterial foraging and differential equation based methods. Examples ofunconstrained optimization methods are the gradient method, Newton’s method,quasi-Newton method, conjugate directive method, etc. Another emerging area ofoptimization is global optimization methods, that include some probabilistic anddeterministic methods. These methods are based on probability theory and similarmathematical fundamentals. Global optimization methods are widely used in

Modern Optimization Methods for Science, Engineering and Technology

1-8

numerical analysis and feature based analysis in various signal processing andcontrol applications.

In the fast changing dynamic environments of various emerging applications ofsignal processing, control, transport, etc, optimization is employed where themethods, models and problems are dynamic in nature. Telecommunications,artificial intelligence based advancements, financial analysis and several others fieldsare rapidly changing with a wide range of dynamics wherein the problems andmodels of optimization used also change rapidly. In such applications, the methodswhich are traditionally used also need modification and dynamic updates [10]. Thereare a number of optimization methods which are recommended for such applica-tions, a few of which are: ant colony optimization, evolutionary optimization,genetic algorithms, neural networks and swarm intelligence based methods, etc. Inthe context of dynamic optimization, the quantitative measures of performance toassess how the optimization method is working become essential. Such measures arerequired in all optimization methods, but in a dynamically changing environmentthe assessment or evaluation is particularly important [1–3, 9, 10]. Some of theimportant measures which are widely used, irrespective of type of optimizationproblem and application, are:

• Cumulative fitness and mean fitness.• Off-line error.• Robustness.• Diversity.• Standard deviation.• Time.• Computational complexity.• Average error.• Average best function value.• Current best evolution.

1.4 Design and structural optimization methodsThis is an era of technological and industrial revolutions that keep on changingdynamically. In order to meet the ever-changing needs of technology enabledindustries, optimization is required to achieve better design and structural performance.For example, numerous studies suggest that airfoil design plays an important role inthe aviation sector and a number of significant research contributions can be foundin the literature in which airfoils, blades and others components of flight are designed insuch an optimal manner that the performance of airplanes improves considerably. Thestructure of any system mainly depends on the shape, size and other similar attributes,and optimization works on optimizing one or more of these attributes.

1.4.1 Structural optimization

Structural optimization deals directly with the variables related to the structure ofthe system or a product, such as size, shape, area, perimeter, volume, etc. If the

Modern Optimization Methods for Science, Engineering and Technology

1-9

comfort level has to be increased in the field of ergonomics, then the optimizationutilizes the structural properties of the system or device whose ergonomics requireimprovement. Figure 1.3 shows a simple and typical structural diagram of the outersurface of an airplane where the airfoil or blade design and their optimization areimportant factors in the performance of the flight. This is just an symbolic diagram,as there are thousands of small, medium and high level structures present in anairplane.

Structural optimization is divided into various types based on the structuralattributes. A few major structural optimization methods or techniques are asfollows:

1. Shape optimization: This is based on the different shapes of systems orproducts.

2. Area and volume optimization: Area and volume decide the performance of ahuge number of products in the transport and control based applications,and therefore the performance could be improved by optimizing either thearea or volume or both or some other similar variables.

3. Size optimization: This may include length, width and other similar dimen-sional features in the optimization of the system’s structural performance.

4. Topological optimization: Since the topology represents the overall intercon-nection of various components in a system and how those components arestructured together determines the system’s performance, in particular in theautomobile and similar sectors, and the role of optimization is to improve thetopology of the system architecture [11].

The topological optimization of structures such as seismic loads, struc-tural optimization for steel plants and reliability optimization are examplesof structural optimization methods. Topological optimization is imple-mented following a certain workflow of operations; one such flow diagramcan be seen in figure 1.4. The need for optimization arises and structuralproperties are studied. The optimization problem is identified and asuitable method is implemented that utilizes the structural attributes in theprocess of optimization. The optimization results in some post-processingwith the application of an optimization method which is sensitive to thestructural features of the system.

Figure 1.3. A typical structural design of the outer surface of an airplane.

Modern Optimization Methods for Science, Engineering and Technology

1-10

1.4.2 Design optimization

Design optimization is not entirely different from structural optimization becausethe design of a system and its structural properties are interrelated. The improve-ment in system performance, reduction in cost and other benefits of optimization canbe achieved using structural optimization methods. The design optimizationtechniques is mainly associated with computer-aided design (CAD) for variousapplications. Software tools are embedded within the CAD systems and any designmodification or manipulation can easily be done so as to obtain the desired systemcharacteristics.

1.5 Optimization for signal processing and control applicationsSignal processing has become an essential component of almost all modernapplications and implementations of STEM and allied fields, and in all suchapplications signal processing and certain signal operations are decisive factors forthe performance of application based systems. We can mention communicationsystems, robotics, self-driving cars, remote sensing, navigational aids, globalpositioning systems, aerodynamics, mechanics, geographic information systems,weather forecasting, financial analysis, mobile communications, satellite communi-cations, etc, in which signal processing and some suitable mathematical operationsare very important factors in the assessment of the performance of the systems. If theperformance improvement of the system is intended for such applications, thenapplying optimization methods in some or all signal processing variables can help inimproving performance.

For example, in the area of biometrics and CAD systems, there are a significantnumber of research publications, but still no robust approach is available andtherefore optimization can help in optimizing some parameters so that the systembehaves in a robust way.

Need for Optimization based on structure ofthe system

Optimization of Problem

Optimization Method

Application ofOptimization

StructuralAnalysis

Figure 1.4. Topological optimization design flow.

Modern Optimization Methods for Science, Engineering and Technology

1-11

1.5.1 Signal processing optimization

Signal processing involves digital signal processing (DSP) or DSP processors wherenumerous signal processing operations are performed. We need to understand thescope of optimization in signal processing applications by analyzing DSP, since DSPis the core part of signal processing and therefore let us examine the maincomponents of DSP. DSP includes some major steps:

1. Data acquisition and sampling.2. Quantization.3. Encoding.4. Signal processing operations.

We have to understand where the optimization could be applied so that signalprocessing becomes better and contributes to the overall improvement of systemperformance in which signal processing is present. If there is scope to apply asuitable optimization method in sample processing, sampling rate, or a dataacquisition step then this has to be identified. Quantization plays an importantrole in DSP and therefore optimization can also be implemented with a quantizationprocess. Encoding is the step that converts the data into a digital format which isused in all DSP, and thus optimization can be applied in the process of encoding aswell. In all such optimization processes, the following types of optimization could bebroadly required:

• Window size optimization: In applications where DSP is required, severalfiltering operations will also be required, which further employ certain typesof windowing operations. In DSP applications, several types of windowfunctions are used, but even after choosing a suitable window function, itsoperation could be improved by using some optimization methods. Windowsize along with execution time, memory requirements and threshold value canalso be optimized so that the windowing effect ultimately contributes to thesystem in optimal ways.

• System throughput optimization: Signal processing is implemented usingcomputers and a number of operations are associated with signal processing.DSP implementation affects memory requirements, central processing unit(CPU) time, memory utilization, parallelism, CPU scheduling, etc. The goalof optimization at the throughput level is to obtain more and better outputfrom computing systems.

Best candidate selection is a factor that plays an important role in operationresearch, where the core of the performance lies in signal processing. How to choosean appropriate sorting method for best candidate selection in operation researchtasks matters a lot in achieving reduced complexity and shorter computation times.Sometimes it is very difficult to achieve both reduced complexity and shortercomputation times, and one needs to sacrifice something to achieve better valueselsewhere. This balance, which is essential, can be obtained using an optimizationmethod [12].

Modern Optimization Methods for Science, Engineering and Technology

1-12

1.5.2 Communication and control optimization

We have seen that signal processing flow is communicated to various stages of thesystem design and that the control of signal flow also affects the system’s perform-ance. Therefore, we need to also understand the optimization concept for commu-nication and control related operations. There are many optimization methods inthe literature that are suitable for the applications, but these methods are all subjectto one concept in optimization theory which is a common factor in all communi-cation related applications, namely the convex theory of optimization. One exampleof such convex optimization for communication and signal processing can be seen in[13], which highlights the concepts of the theory and some examples of optimizationmethods utilizing convex theory as central theme of optimization.

The main aim of using convex optimization in signal processing and communi-cation is to deal with convex constraints that are obtained from convex mathemat-ical functions. Conic optimization and interior point method optimization areexamples of convex theory based optimization. Convex itself means to achieve asuitable objective function that can help in obtaining a local optimal function in aproblem. This optimization concept is used in obtaining both global and localoptima. Several engineering applications employ convex optimization methods.Linear sub-space, second order cone and Affine sub-space are a few major examplesof the convex functions used for optimization tasks [13]. In fact, unconstrainedmethods can also be converted into some finite constrained methods by using theconvex concept. Analog band pulse amplitude modulation and filter design are thecases where convex optimization is useful in communication related applications.

1.6 Design vectors, matrices, vector spaces, geometry and transformsIn applications related to search engine optimization [14], or any STEM application,the implementation of optimization requires a strong mathematical backgroundbecause optimization uses variables, operations, constraints, minima and maximafunctions, etc. Therefore, system design optimization in any applications ofoptimization needs sufficient consideration of the vectors, matrices and vectorspaces that support the mathematical implementation of the optimization method.This section provides a brief overview of these concepts, because all the methodsdiscussed and implemented in the other chapters use realistic and actual mathemat-ical functions and variables.

The advances in technology, evolution of the internet, increased use of computingdevices and large amounts of data are some of the factors that require optimizationof various types of system designs and operations. In the flow diagram ofimplementation of optimization, we have discussed that the process depends ondesign variables, constraints and some final conditions to check if the optimization isworking or not. All of these variables and their operations make use of linearalgebra, matrices and vector spaces [15] for different computations involved in thesystem, which are presented in this section.

Modern Optimization Methods for Science, Engineering and Technology

1-13

1.6.1 Linear algebra, matrices and design vectors

Linear algebra is the basis for linear optimization methods and the algebra utilizeslinear equations. A typical example of set of linear equations is

+ + =X Y Z P2 1.5 (1.5)1 1 1 1

+ + =X Y Z P2 3 (1.6)2 2 2 2

+ + =X Y Z P1.5 5 , (1.7)3 3 3 3

where X1, X2, X3, Y1, Y2, Y3, Z1, Z2, Z3 are variables and P1, P2 and P3 areconstants. The equations are linear equations and the values of P1, P2 and P3 may beany real values. The representation as shown by the three equations can also berepresented as a vector:

• X = [X1 × 2 × 3]; Y = [Y1 Y2 Y3]; Z = [Z1 Z2 Z3]; and P = [P1 P2 P3].• The coefficients used in the three equations can also be expressed as vectors:A1 = [1 1 1]; B = [2 2 1.5]; and C = [1.5 3 5].

We can also write the above equations as

⎡⎣⎢⎢

⎤⎦⎥⎥⎡⎣⎢⎢

⎤⎦⎥⎥

⎡

⎣⎢⎢

⎤

⎦⎥⎥=

XYZ

PPP

1 2 1.51 2 31 1.5 5

. (1.8)1

2

3

There are many other ways in which vector representation can replace the set oflinear equations. This representation of equations is more appropriately called amatrix representation. When matrix representation comes into the analysis of anyoptimization theory then all of its properties, such as determinant, rank, character-istic equation, eigenvalues, eigenvectors, etc, become very important to understand.Since the matrix and its properties at this level fall under the elementary knowledgeof matrices we will not discuss them further here. Thus, linear optimization canwork on:

• Linear equations.• Vector representations of linear equations.• Matrix representations.

Matrix representations provide an efficient way to express the operations and alsobetter help in the analysis of systems compared to the other approaches as far as theimplementation of linear optimization is concerned [16]. Linear representation basedoptimization includes:

• The steepest descent method.• The conjugate gradient method [17].• The normal equation method.

Modern Optimization Methods for Science, Engineering and Technology

1-14

1.6.2 Vector spaces

Vector space represents a set function whose elements are called vectors and vectorspace also includes another field which is a scalar quantity (generally a real number).Two important operations are characterized by vector space operation, plus (+) andmultiplication (·), which are vector addition and scalar multiplication, respectively.If x and y are vectors of vector space X then x + y will also belong to the vector spaceX. The following are the important principles of vector addition:

• Cumulative law: x + y = y + x, for all x, y in X.• Associative law: x + (y + z) = (x + y) + z, where x, y and z are vectors invector space X.

• Additive identity and additive inverse: If a vector is added with additiveidentity then it remains the same vector, x + 0 = x, and if the vector is addedwith additive inverse then it results in an additive identity: x + x′ = 0.

Similarly, scalar multiplication satisfies the following:• Distributive law: p·(x + y) = p·x + p·y and (p + q)·x = p·x + q·x where p ad qare real numbers.

• Associative law: p·(q·x) = (p·q)·x.• Unitary law: 1·x = x for all x in X.

Any vector space can also have its vector sub-spaces. The concepts of vector spaceand sub-spaces are utilized in the optimization methods for linear optimization inseveral applications [18–24]. Vector space includes vector addition which is acommon mathematical tool in vector algebra. This can be seen in figure 1.5.

1.6.3 Geometry, transforms, binary and fuzzy logic

Mathematical geometry and transforms are used in a number of optimizationproblems and solutions. Different types of operations such as rotation, scaling, shift,resizing and reformatting all come under the geometry and transforms that are used

x

y

x + y

Figure 1.5. Vector addition.

Modern Optimization Methods for Science, Engineering and Technology

1-15

some steps of optimization methods. In image processing, for example, rotation is animportant translation operation which can be performed to any extent; one suchexample is shown in figure 1.6.

There are several examples of operations which can be included in this section,but the idea of showing one example is to just highlight what transform andgeometry operations mean.

Now in the age of DSP, binary operations are essential components in all stagesof DSP. When it comes to decision making and path planning control then a newlogic is preferred in all STEM applications, which is called fuzzy logic. The conceptof fuzzy logic is now utilized in almost all applications, including optimizationmethods. Neural network and other machine leaning methods are used for variousfeature extraction and training methods for samples, but a hybrid approach isrecommended for better performance of neuro-fuzzy logic to reduce the uncertainty.Figure 1.7 shows the temperature of an object as hot and cold.

The problem arises when we want to know the degree of being hot or cold, whichmeans how hot or cold the object is. Here, the binary concept produces someamount of uncertainty which is addressed by using fuzzy logic. This logic ischaracterized by using some suitable membership function, for example in thecurrent case of temperature, if the membership function is defined properly then thedegree of hotness or coldness can be determined (see figure 1.8).

Figure 1.6. Rotation by a certain angle.

Hot

Cold

Figure 1.7. Two possible values of temperature as hot and cold.

Modern Optimization Methods for Science, Engineering and Technology

1-16

If we describe the temperature with a suitable value of membership function, saythe maximum temperature is 1100 °C and the minimum is 100 °C, then if themembership function (value) is 0.3 it means that the temperature of the object is 100(minimum) + 0.3 × (1100 – 100) = 400 °C. So, the object is hot but the certainty isthere as to what extent the object is hot. The concept of fuzzy logic is very popularamong scientists and researchers for various science and engineering applications.

References[1] Sinha G R, Raju K S, Patra R, Aye D W and Khin D T 2018 Research studies on human

cognitive ability Int. J. Intell. Defen. Supp. Syst. 5 298–304[2] Sinha H, Meshram M R and Sinha G R 2018 BER performance analysis of MIMO-OFDM

over wireless channel Int. J. Pure Appl. Math. 118 195–206[3] Patel B C and Sinha G R 2011 Comparative performance evaluation of segmentation

methods in breast cancer images Int. J. Mach. Intell. 3 130–3[4] Sinha G R 2015 Fuzzy Based Medical Image Processing Advances in Medical Technologies

and Clinical Practice (AMTCP) Book Series (Hershey, PA: IGI Global)[5] Patel B C and Sinha G R 2010 Early detection of breast cancer using self-similar fractal

method Int. J. Comput. Appl. 10 39–43[6] Sinha K and Sinha G R 2013 Comparative analysis of optimized K-means and C-means

clustering methods for segmentation of brain MRI images for tumor extraction Proc. of Int.Conf. on Emerging Research in Computing, Information, Communication and Applications(Amsterdam: Elsevier), pp 619–25

[7] Prepoka A 1980 On the development of optimization theory Am. Math. Month. 87 527–42[8] Patel B C and Sinha G R 2018 Mass segmentation and feature extraction of mammographic

breast images in computer-aided diagnosis PhD Thesis Chhattisgarh Swami VivekanandTechnical University Bhilai

[9] Chong E K P and Zak S H 2001 An Introduction to Optimization (New York: Wiley)[10] Cruz C, González J R and Pelta D A 2011 Optimization in dynamic environments: a survey

on problems, methods and measures Soft Comput. 15 1427–48[11] Labanada S R 2015 Mathematical programming methods for large-scale topology opti-

mization problems PhD Thesis Technical University of Denmark[12] Ahmad H A 2012 The best candidates method for solving optimization problems J. Comput.

Sci. 8 711–5[13] Luo Z Q and Yu W 2006 An introduction to convex optimization for communications and

signal processing IEEE J. Select. Area Commun. 24 1426–38[14] Bezhovski Z 2015 The historical development of search engine optimization Inform. Know.

Manage. 5 91–6

Hot

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Membership value (function), m

Figure 1.8. Membership function in defining a fuzzy variable.

Modern Optimization Methods for Science, Engineering and Technology

1-17

[15] Luenberger D G 1969 Optimization by Vector Space Methods (New York: Wiley)[16] Absil P A, Mahony R and Sepulchre R 2008 Optimization Algorithms on Matrix Manifolds

(Princeton, NJ: Princeton University Press)[17] Abrudan T, Eriksson J and Koivunen V 2009 Conjugate gradient algorithm for optimization

under unitary matrix constraint Signal Process. 89 1704–14[18] Ehrgott M and Gandibleux X 2003 Multiple Criteria Optimization: State of the Art

Annotated Bibliographic Surveys (New York: Kluwer/Academic)[19] Boumal N 2014 Optimization and estimation on manifolds PhD Princeton University https://

web.math.princeton.edu/~nboumal/papers/boumal_optimization_and_estimation_on_mani-folds_phd_thesis.pdf

[20] Hartley R I and Kahl F 2009 Global optimization through rotation space search Int. J.Comput. Vis. 82 64–79

[21] Gallier J and Quaintance J 2018 Fundamentals of linear algebra and optimization TechnicalReport University of Pennsylvania

[22] Nocedal J and Wright S J 2006 Numerical Optimization 2nd ed (Berlin: Springer)[23] Dubourg V 2011 Adaptive surrogate models for reliability analysis and reliability-based

design optimization PhD Thesis Université Blaise Pascal https://www.phimeca.com/IMG/pdf/these_dubourg_2011.pdf

[24] Simon D 2013 Evolutionary Optimization Algorithms: Biologically-Inspired and Population-Based Approaches to Computer Intelligence (Hoboken, NJ: Wiley)

Modern Optimization Methods for Science, Engineering and Technology

1-18