performance assessment of multiobjective evolutionary
TRANSCRIPT
![Page 1: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/1.jpg)
Performance Assessment of
Multiobjective Evolutionary Algorithms
Lucas S. Batista
Universidade Federal de Minas GeraisDepartamento de Engenharia Elétrica
Belo Horizonte - MG, Brasil
November, 2018
![Page 2: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/2.jpg)
Contents
1 Introduction
2 Unary and Binary Quality Indicators
3 Performance Measures
Unary Quality Indicators
Binary Quality Indicators
4 Conclusions
![Page 3: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/3.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Current Section
1 Introduction
2 Unary and Binary Quality Indicators
3 Performance Measures
Unary Quality Indicators
Binary Quality Indicators
4 Conclusions
3 / 39
![Page 4: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/4.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Initial Presentation
State of the Art
Multiobjective Optimization Problem
minxxx
fff (xxx) ∈ Rm
subject to: xxx ∈ Ω
Ω =
gi(xxx) ≤ 0; i = 1, . . . ,phj(xxx) = 0; j = 1, . . . ,qxxx ∈ X
X =
xmin
k ≤ xk ≤ xmaxk
k = 1, . . . ,n
4 / 39
![Page 5: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/5.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Initial Presentation
State of the Art
The global Pareto front and Approximation Sets
P = xxx∗ ∈ Ω | ∄ xxx ∈ Ω : fff (xxx) ≤ fff (xxx∗) ∧ fff (xxx) 6= fff (xxx∗)
5 / 39
![Page 6: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/6.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Initial Presentation
State of the Art
A high-quality approximation set should:
1 Approach the true Pareto front as close as possible (con-
vergence); and
2 Be well-spread along the extension of the front found (diver-
sity).
How to evaluate the quality of estimation sets?
6 / 39
![Page 7: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/7.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Initial Presentation
State of the Art
Comparison methods:
1 If one algorithm is better than another, can we express how
much better it is?
2 If no algorithm can be said to be better than the other, are
there certain aspects in which respect we can say the for-
mer is better than the latter?
How to best summarize approximation sets when de-
signing quality measures?
7 / 39
![Page 8: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/8.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Current Section
1 Introduction
2 Unary and Binary Quality Indicators
3 Performance Measures
Unary Quality Indicators
Binary Quality Indicators
4 Conclusions
8 / 39
![Page 9: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/9.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
General Informations
Unary Quality Measures
The quality value assigned to an approximation set is inde-
pendent of other sets under consideration;
Unary quality indicators are most commonly used in the lit-
erature;
Assign a real number to an approximation set;
9 / 39
![Page 10: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/10.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
General Informations
Unary Quality Measures
In general, are not capable of indicating whether an approx-
imation set is better than another;
Existing unary indicators at best allow to infer that an ap-
proximation set is not worse than another;
With many unary indicators and also combinations of unary
indicators, no statement about the relation between the cor-
responding approximation sets can be made.
10 / 39
![Page 11: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/11.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
General Informations
Binary Quality Measures
The quality value assigned to an approximation set depends
on other sets under consideration;
Assign real numbers to ordered pairs of approximation sets;
The performance measure of MOEAs is also a multicriteria
task;
11 / 39
![Page 12: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/12.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
General Informations
Binary Quality Measures
Binary quality indicators can be used to overcome the diffi-
culties with unary indicators;
However, when we compare l algorithms using a single bi-
nary indicator, we obtain l(l − 1) distinct indicator values;
This renders the analysis and the presentation of the results
more difficult.
12 / 39
![Page 13: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/13.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Current Section
1 Introduction
2 Unary and Binary Quality Indicators
3 Performance Measures
Unary Quality Indicators
Binary Quality Indicators
4 Conclusions
13 / 39
![Page 14: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/14.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Spacing measure (Schott 1995)
Spacing focuses on relative distance between solutions;
A value S = 0 indicates all members of the estimated front
are equidistantly spaced;
However, if all the solutions are located in the same point,
then S = 0 and in fact the diversity in this case is the worst.
S =
√√√√ 1
|A| − 1
|A|∑
i=1
(d − di)2
di = minj
(∣∣f1(xi )− f1(xj )
∣∣+
∣∣f2(xi )− f2(xj )
∣∣)
14 / 39
![Page 15: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/15.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Error ratio (ER) (Veldhuizen 1999)
ei = 0 if vector i is in A∗ and 1 otherwise;
Lower values of the error ratio represent better nondomi-
nated sets;
ER is the proportion of non true Pareto points in A.
ER =
∑|A|i=1 ei
|A|
15 / 39
![Page 16: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/16.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Maximum Error with respect to A∗ (ME) (Veldhuizen 1999)
Measure the distance between the found solution set and
A∗;
The more distance the solution set is from A∗, the greater
the measure ME is;
The greatest minimal distance between vectors of the solu-
tion set and their closest neighbors in A∗:
ME = maxj∈A∗
( mini∈A
dij )
16 / 39
![Page 17: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/17.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
∆ measure (Deb et al. 2002)
∆ =
∑mi=1 de
i +∑|A|
i=1
∣∣di − d
∣∣
∑mi=1 de
i + |A| d
17 / 39
![Page 18: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/18.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
γ measure (Deb et al. 2002)
γ =
∑|A|i=1 di
|A|
18 / 39
![Page 19: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/19.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Generational distance (GD) (Nebro et al. 2006)
GD =
(∑|A|
i=1 d2i
)1/2
|A|
19 / 39
![Page 20: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/20.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Inverted GD (IGD) (Veldhuizen and Lamont 1998)
Represent average distance from solutions in the global Pa-
reto front to the nearest solution in an approximation set;
A value of IGD = 0 indicates that all the generated elements
are in the Pareto front and they cover all the extension of the
Pareto front.
IGD =
(∑|A∗|
i=1 d2i
)1/2
|A∗|
20 / 39
![Page 21: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/21.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Hierarchical Cluster Counting (HCC) (Guimarães et al. 2009)
Measure of both uniformity and extension of a set;
Identify the front with the best Pareto set shape;
It is an improved version of the sphere counting (SC).
21 / 39
![Page 22: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/22.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
Hyperarea and Ratio (HR) (Veldhuizen 1999)
Calculate the hypervolume of the multi-dimensional objetive
space enclosed by A and a reference point.
Consider both closeness and diversity.
HV (A) =
⋃
i
vol(yyy i) | ∀ yyy i ∈ A
HR =HV (A)
HV (A∗)
22 / 39
![Page 23: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/23.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Unary Quality Indicators
S-metric (HV) (Zitzler & Thiele 1999)
23 / 39
![Page 24: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/24.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Coverage of two sets (CS) (Zitzler & Thiele 1999)
CS(X ′,X ′′)=︷ ︸︸ ︷
|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|
|X ′′|
24 / 39
![Page 25: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/25.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Coverage of two sets (CS) (Zitzler & Thiele 1999)
CS(X ′,X ′′)=︷ ︸︸ ︷
|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|
|X ′′|
CS(X ′,X ′′) = (4/10) = 0.40
25 / 39
![Page 26: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/26.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Coverage of two sets (CS) (Zitzler & Thiele 1999)
CS(X ′,X ′′)=︷ ︸︸ ︷
|a′′ ∈ X ′′; ∃a′ ∈ X ′ : a′ a′′|
|X ′′|
CS(X ′,X ′′) = (4/10) = 0.40
CS(X ′′,X ′) = (3/10) = 0.30
26 / 39
![Page 27: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/27.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Coverage of Many Sets (CMS) (Batista et al.)
Quantify the domination of the final population of one algo-
rithm over the union of the remaining ones.
Consider both closeness and diversity.
CMS(Xi ,Ui) =|a′′ ∈ Ui ; ∃ a′ ∈ Xi : a′ a′′|
|Ui |
Ui =
k⋃
j=1j 6=i
Xj
27 / 39
![Page 28: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/28.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Coverage of Many Sets (CMS) (Batista et al.)
28 / 39
![Page 29: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/29.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Pareto Dominance Indicator (NR) (Goh and Tan 2009)
NR assesses the number of non-dominated solutions in the
set;
This metric measures the ratio of non-dominated solutions
that is contributed by a particular solution set Ai to the non-
dominated solutions provided by all solutions.
NR i(A1,A2, . . . ,Ak ) =|Ai ∩ B|
|B|
B =
bi | ∀ bi ,∄ aj ∈ (A1 ∪ A2 ∪ . . . ∪ Ak) ≺ bi
29 / 39
![Page 30: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/30.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
D1R (Czyzak and Jaszkiewicz 1998)
D1R measures the mean distance, over the vectors in a
reference set R, of the nearest solution in an approximation
set A.
D1R(A,Λ) =1
|R|
∑
rrr∈R
minaaa∈A
d(rrr ,aaa)
d(rrr ,aaa) = maxk
λk (ak − rk ), Λ = λ1, λ2, . . . , λm
λk = 1/range(f Rk ), k ∈ 1, . . . ,m
30 / 39
![Page 31: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/31.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
R1 (Hansen and Jaszkiewicz 1998)
R1 calculates the probability that approximation A is better
than approximation B over a set of utility functions U:
R1(A,B,U,p) =
∫
u∈U
C(A,B,u)p(u)du
C(A,B,u) =
1 if u∗(A) > u∗(B)1/2 if u∗(A) = u∗(B)0 if u∗(A) < u∗(B)
in which p(u) is an intensity function expressing the proba-
bility density of the utility u ∈ U, and u∗(A) = maxaaa∈Au(aaa).
31 / 39
![Page 32: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/32.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
Set of utility functions
A set Up of utility functions based on weighted Lp norm is a
parametric set composed of functions of the following form:
up(xxx , fff∗,Λ,p) =
( m∑
k=1
λk (fk (xxx)− f ∗k )p
)1/p
, p ∈ 1,2, . . . ,∞
Λ = λλλ ∈ Rm |∑
k
λk = 1 and λk > 0, k = 1,2, . . . ,m
32 / 39
![Page 33: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/33.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
R1R (Hansen and Jaszkiewicz 1998)
R1R is R1 when it is used with a reference set R.
R1R(A,U,p) = R1(A,R,U,p) =
∫
u∈U
C(A,R,u)p(u)du
C(A,R,u) =
1 if u∗(A) > u∗(R)1/2 if u∗(A) = u∗(R)0 if u∗(A) < u∗(R)
33 / 39
![Page 34: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/34.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
R2 (Hansen and Jaszkiewicz 1998)
R2 calculates the expected difference in the utility of an ap-
proximation A with another one B.
While R1 just uses the function C(A,B,u) to decide which
of two approximation is better on utility function u, without
measuring by how much, R2 takes into account the ex-
pected values of the utility.
R2(A,B,U,p) = E(u∗(A))−E(u∗(B)) =
∫
u∈U
(u∗(A)−u∗(B))p(u)du
34 / 39
![Page 35: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/35.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
R2R (Hansen and Jaszkiewicz 1998)
R2R is R2 when it is used with a reference set R.
R2R(A,U,p) = R2(A,R,U,p) = E(u∗(R))− E(u∗(A))
R2R(A,U,p) =∫
u∈U(u∗(R)− u∗(A))p(u)du
35 / 39
![Page 36: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/36.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Binary Quality Indicators
R3 and R3R (Hansen and Jaszkiewicz 1998)
In some cases, ratios of best utility values may be more
meaningful than their differences.
R3(A,B,U,p) = E
(u∗(B)− u∗(A)
u∗(B)
)
=
∫
u∈U
u∗(B)− u∗(A)
u∗(B)p(u)du
R3R(A,U,p) = R3(A,R,U,p) = E
(u∗(R)− u∗(A)
u∗(R)
)
R3R(A,U,p) =∫
u∈U
u∗(R)− u∗(A)
u∗(R)p(u)du
36 / 39
![Page 37: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/37.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Current Section
1 Introduction
2 Unary and Binary Quality Indicators
3 Performance Measures
Unary Quality Indicators
Binary Quality Indicators
4 Conclusions
37 / 39
![Page 38: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/38.jpg)
Introduction Unary and Binary Quality Indicators Performance Measures Conclusions
Conclusions
In MOOPs, the optimization result is a set of solutions;
The evaluation of obtained solution set consists of (in gen-
eral) two aspects: convergence and diversity properties;
There exists no unary quality measure that is able to indi-
cate whether an approximation A is better than an approx-
imation B, even when we consider a finite combination of
unary measure;
Since the performance assessment of MOEAs represents
a multicriteria task, the consideration of binary quality mea-
sures is very useful.
38 / 39
![Page 39: Performance Assessment of Multiobjective Evolutionary](https://reader034.vdocuments.us/reader034/viewer/2022052514/628bf1485737ed519a1b0640/html5/thumbnails/39.jpg)
Specialized Literature
Specialized Literature
M. P. Hansen & A. Jaszkiewicz (1998). Evaluating the quality of approximations tothe non-dominated set, Technical Report, pp. 1–30.
D. Van Veldhuizen & G. Lamont (2000). On Measuring Multiobjective EvolutionaryAlgorithm Performance, IEEE, pp. 204–211.
Joshua Knowles & David Corne (2002). On Metrics for Comparing NondominatedSets, IEEE, pp. 711–716.
Tatsuya Okabe, Yaochu Jin & Bernhard Sendhoff (2003). A Critical Survey of Per-formance Indices for Multi-Objective Optimisation, IEEE, pp. 878–885.
Zhenan He & Gary Yen (2011). An Ensemble Method for Performance Metrics inMultiobjective Evolutionary Algorithms, IEEE, pp. 1724–1729.
Title page
39 / 39