optimization methods aleksey minin saint-petersburg state university student of acophys master...
TRANSCRIPT
![Page 1: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/1.jpg)
Optimization methodsAleksey MininSaint-Petersburg State UniversityStudent of ACOPhys master program (10th semester)
1
Joint Advanced Students School
19.04.23
Applied and COmputational Physics
![Page 2: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/2.jpg)
What is optimization? 19.04.23Joint Advanced Students School
2
![Page 3: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/3.jpg)
Content:
3
Joint Advanced Students School
19.04.23
![Page 4: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/4.jpg)
Applications of optimization
• Advanced engineering designAdvanced engineering design• BiotechnologyBiotechnology• Data analysisData analysis• Environmental managementEnvironmental management• Financial planningFinancial planning• Process controlProcess control• Scientific modeling etcScientific modeling etc
4
Joint Advanced Students School
19.04.23
![Page 5: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/5.jpg)
Global or Local ?
19.04.23Joint Advanced Students School
5
![Page 6: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/6.jpg)
What is global optimization?
•The objective of global optimization is to find the globally best solution of (possibly nonlinear) models, in the (possible or known) presence of multiple local optima.
19.04.23Joint Advanced Students School
6
![Page 7: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/7.jpg)
19.04.23Joint Advanced Students School
7
![Page 8: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/8.jpg)
Branch and bound 19.04.23Joint Advanced Students School
8
![Page 9: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/9.jpg)
19.04.23Joint Advanced Students School
9
Branch and boundScientist are ready to carry out some experiments. But the quality of all of the varies depending on type of experiment according to next table:
Type of experiment
Scientist number
1 2 3 4
A 0.9 0.8 0.9 0.85
B 0.7 0.6 0.8 0.7
C 0.85 0.7 0.85 0.8
D 0.75 0.7 0.75 0.7
![Page 10: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/10.jpg)
19.04.23Joint Advanced Students School
10
Branch and bound
Root
![Page 11: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/11.jpg)
19.04.23Joint Advanced Students School
11
Branch and bound
RootAAAA0.55
![Page 12: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/12.jpg)
19.04.23Joint Advanced Students School
12
Branch and bound
RootAAAA0.55
AADCC0.42
BBAAA0.42
CCAAA0.52
DDAAA0.45
![Page 13: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/13.jpg)
19.04.23Joint Advanced Students School
13
Branch and bound
RootAAAA0.55
AADCC0.42
BBAAA0.42
CCAAA0.52
DDAAA0.45
BCBAA0.39
DCDAA0.45
ACABD0.38
![Page 14: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/14.jpg)
19.04.23Joint Advanced Students School
14
Branch and bound
RootAAAA0.55
AADCC0.42
BBAAA0.42
CCAAA0.52
DDAAA0.45 D
CDAA0.45
BCBAA0.39
ACABD0.38 A
CBAD0.37
BCDBA0.40
![Page 15: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/15.jpg)
19.04.23Joint Advanced Students School
15
Branch and bound
RootAAAA0.55
AADCC0.42
BBAAA0.42
CCAAA0.52
DDAAA0.45 D
CDAA0.45
BCBAA0.39
ACABD0.38
BCDBA0.40
ACBAD0.37
![Page 16: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/16.jpg)
19.04.23Joint Advanced Students School
16
Branch and bound
![Page 17: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/17.jpg)
Evolutionary algorithms
19.04.23Joint Advanced Students School
17
![Page 18: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/18.jpg)
Evolutionary algorithms
19.04.23Joint Advanced Students School
18
![Page 19: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/19.jpg)
Simulated annealing 19.04.23Joint Advanced Students School
19
Apply small
perturbation
Solution found!
If T=0
Repeat until good solution
not found
![Page 20: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/20.jpg)
19.04.23Joint Advanced Students School
20
Simulated annealingresults
![Page 21: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/21.jpg)
19.04.23Joint Advanced Students School
21
Simulated annealing
![Page 22: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/22.jpg)
Tree annealingdeveloped by Bilbro and Snyder [1991]
19.04.23Joint Advanced Students School
22
![Page 23: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/23.jpg)
19.04.23Joint Advanced Students School
23
Tree annealingdeveloped by Bilbro and Snyder [1991]
![Page 24: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/24.jpg)
Swarm intelligence
19.04.23Joint Advanced Students School
24
![Page 25: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/25.jpg)
Tabu Search
19.04.23Joint Advanced Students School
25
![Page 26: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/26.jpg)
Taboo search implementation
19.04.23Joint Advanced Students School
26
1
![Page 27: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/27.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
27
1
34
5 2
![Page 28: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/28.jpg)
Taboo search implementation
19.04.23Joint Advanced Students School
28
1
34
5 2
1
![Page 29: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/29.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
29
1
34
5 2
1
7
6
3
![Page 30: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/30.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
30
1
34
5 2
1
7
6
3
9
8
6
![Page 31: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/31.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
31
1
34
5 2
1
7
6
3
9
8
6
10
11
9
![Page 32: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/32.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
32
1
34
5 2
1
7
6
3
9
8
6
10
11
9
![Page 33: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/33.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
33
1
34
5 2
1
7
6
3
9
8
10
11
9 6
![Page 34: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/34.jpg)
Tabu search implementation
19.04.23Joint Advanced Students School
34
1
34
5 2
1
7
6
3
9
8
10
11
9 6
![Page 35: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/35.jpg)
19.04.23Joint Advanced Students School
35
Tabu Search
![Page 36: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/36.jpg)
What is Local Optimization?
•The term LOCAL refers both to the fact that only information about the function from the neighborhood of the current approximation is used in updating the approximation as well as that we usually expect such methods to converge to whatever local extremum is closest to the starting approximation.
•Global structure of the objective function is unknown to a local method.
19.04.23Joint Advanced Students School
36
![Page 37: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/37.jpg)
19.04.23Joint Advanced Students School
37
Local optimization
![Page 38: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/38.jpg)
Gradient descent
19.04.23Joint Advanced Students School
38
![Page 39: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/39.jpg)
Gradient descent
19.04.23Joint Advanced Students School
39
Therefore we obtained: F(x0)<F(x1)<…<F(xn )
![Page 40: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/40.jpg)
Quasi-Newton Methods
19.04.23Joint Advanced Students School
40
•These methods build up curvature information at eachiteration to formulate a quadratic model problem of the form:
where the Hessian matrix, H, is a positive definite symmetric matrix, c is a constant vector, and b is a constant.•The optimal solution for this problem occurs when the partial derivatives of x go to zero:
![Page 41: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/41.jpg)
Quasi-Newton Methods
19.04.23Joint Advanced Students School
41
![Page 42: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/42.jpg)
BFGS - algorithm 19.04.23Joint Advanced Students School
42
![Page 43: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/43.jpg)
19.04.23Joint Advanced Students School
43
BFGS - algorithm
![Page 44: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/44.jpg)
Gauss Newton algorithm
19.04.23Joint Advanced Students School
44
Given m functions f1 f2 … fm of n parameters p1 p2 .. Pn (m>n),and we want to minimize the sum:
![Page 45: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/45.jpg)
19.04.23Joint Advanced Students School
45
Gauss Newton algorithm
![Page 46: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/46.jpg)
Levenberg-Marquardt 19.04.23Joint Advanced Students School
46
This is an iterative procedure. Initial guess for pT = (1,1,…,1).
![Page 47: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/47.jpg)
19.04.23Joint Advanced Students School
47
Levenberg-Marquardt
![Page 48: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/48.jpg)
19.04.23Joint Advanced Students School
48
SQP – constrained minimization
Reformulation
![Page 49: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/49.jpg)
19.04.23Joint Advanced Students School
49
SQP – constrained minimizationThe principal idea is the formulation of a QP sub-problem based on a quadratic approximation of the Lagrangian function:
![Page 50: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/50.jpg)
19.04.23Joint Advanced Students School
50
SQP – constrained minimizationUpdating the Hessian matrix
![Page 51: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/51.jpg)
19.04.23Joint Advanced Students School
51
SQP – constrained minimizationUpdating the Hessian matrix
![Page 52: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/52.jpg)
Neural Net analysis
What is Neuron?
Typical formal neuron makes the elementary operation – weighs values of the inputs with the locally stored weights and makes above their sum nonlinear transformation:
y f u u w w xi ii , 0
x1 xn
y
u
y
u w w xi i 0
neuron makes nonlinear operation above a linear combination of inputs
19.04.23Joint Advanced Students School
![Page 53: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/53.jpg)
Neural Net analysis
What is training?
W – set of synaptic W – set of synaptic weightsweightsE (W) – error functionE (W) – error function
What kind of optimization to choose?
Joint Advanced Students School
19.04.23
![Page 54: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/54.jpg)
19.04.23Joint Advanced Students School
54
Neural Network – any architecture
1 2 3 4
Error back propagation
1 2
34
5
06
![Page 55: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/55.jpg)
19.04.23Joint Advanced Students School
55
How to optimize?
Objective function – is an Empirical error (should decay)Parameters to optimize - are weightsConstraints – are equalities (inequalities) for weights if exist
![Page 56: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/56.jpg)
19.04.23Joint Advanced Students School
56
Neural Net analysis and constrained and unconstrained minimization
NB! For unconstrained optimization I applied Levenberg-Marquardt methodFor constrained case I applied SQP method
![Page 57: Optimization methods Aleksey Minin Saint-Petersburg State University Student of ACOPhys master program (10 th semester) 1 Joint Advanced Students School](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649d745503460f94a54ea4/html5/thumbnails/57.jpg)
Thank you for your attention
19.04.23Joint Advanced Students School
57