From Covariance Matricesto
Covariance OperatorsData Representation from Finite to Infinite-Dimensional Settings
Ha Quang Minh
Pattern Analysis and Computer Vision (PAVIS)Istituto Italiano di Tecnologia, ITALY
November 30, 2017H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 1 / 99
Acknowledgement
Collaborators
Marco San BiagioVittorio Murino
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 2 / 99
From finite to infinite dimensions
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 3 / 99
Part I - Outline
Finite-dimensional setting
Covariance Matrices and Applications
1 Data Representation by Covariance Matrices2 Geometry of SPD matrices3 Machine Learning Methods on Covariance Matrices and
Applications in Computer Vision
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 4 / 99
Part II - Outline
Infinite-dimensional setting
Covariance Operators and Applications
1 Data Representation by Covariance Operators2 Geometry of Covariance Operators3 Machine Learning Methods on Covariance Operators and
Applications in Computer Vision
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 5 / 99
From finite to infinite-dimensional settings
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 6 / 99
Part II - Outline
Infinite-dimensional setting
Covariance Operators and Applications
1 Data Representation by Covariance Operators2 Geometry of Covariance Operators3 Machine Learning Methods on Covariance Operators and
Applications in Computer Vision
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 7 / 99
Covariance operator representation - Motivation
Covariance matrices encode linear correlations of input featuresNonlinearization
1 Map original input features into a high (generally infinite)dimensional feature space (via kernels)
2 Covariance operators: covariance matrices of infinite-dimensionalfeatures
3 Encode nonlinear correlations of input features4 Provide a richer, more expressive representation of the data
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 8 / 99
Covariance operator representation
S.K. Zhou and R. Chellappa. From sample similarity to ensemblesimilarity: Probabilistic distance measures in reproducing kernelHilbert space, PAMI 2006M. Harandi, M. Salzmann, and F. Porikli. Bregman divergences forinfinite-dimensional covariance matrices, CVPR 2014H.Q.Minh, M. San Biagio, V. Murino. Log-Hilbert-Schmidt metricbetween positive definite operators on Hilbert spaces, NIPS 2014H.Q.Minh, M. San Biagio, L. Bazzani, V. Murino. ApproximateLog-Hilbert-Schmidt distances between covariance operators forimage classification, CVPR 2016
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 9 / 99
Positive Definite Kernels
X any nonempty setK : X × X → R is a (real-valued) positive definite kernel if it issymmetric and
N∑i,j=1
aiajK (xi , xj) ≥ 0
for any finite set of points {xi}Ni=1 ∈ X and real numbers{ai}Ni=1 ∈ R.[K (xi , xj)]Ni,j=1 is symmetric positive semi-definite
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 10 / 99
Reproducing Kernel Hilbert Spaces
K a positive definite kernel on X × X . For each x ∈ X , there is afunction Kx : X → R, with Kx (t) = K (x , t).
HK = {N∑
i=1
aiKxi : N ∈ N}
with inner product
〈∑
i
aiKxi ,∑
j
bjKyj 〉HK =∑i,j
aibjK (xi , yj)
HK = RKHS associated with K (unique).
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 11 / 99
Reproducing Kernel Hilbert Spaces
Reproducing property: for each f ∈ HK , for every x ∈ X
f (x) = 〈f ,Kx〉HK
Abstract theory due to Aronszajn (1950)Numerous applications in machine learning (kernel methods)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 12 / 99
Examples: RKHS
The Gaussian kernel K (x , y) = exp(− |x−y |2σ2 ) on Rn induces the
space
HK = {||f ||2HK=
1(2π)n(σ
√π)n
∫Rn
eσ2|ξ|2
4 |f (ξ)|2dξ <∞}.
The Laplacian kernel K (x , y) = exp (−a|x − y |), a > 0, on Rn
induces the space
HK = {||f ||2HK=
1(2π)n
1aC(n)
∫Rn
(a2 + |ξ|2)n+1
2 |f (ξ)|2dξ <∞}.
with C(n) = 2nπn−1
2 Γ(n+12 )
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 13 / 99
Feature map and feature space
Geometric viewpoint from machine learningPositive definite kernel K on X × X induces feature mapΦ : X → HK
Φ(x) = Kx ∈ HK , HK = feature space〈Φ(x),Φ(y)〉HK = 〈Kx ,Ky 〉HK = K (x , y)
Kernelization: Transform linear algorithm depending on
〈x , y〉Rn
into nonlinear algorithms depending on
K (x , y)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 14 / 99
Covariance matrices
ρ = Borel probability distribution on X ⊂ Rn, with∫X||x ||2dρ(x) <∞
Mean vector
µ =
∫X
xdρ(x)
Covariance matrix
C =
∫X
(x − µ)(x − µ)T dρ(x)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 15 / 99
RKHS covariance operators
ρ = Borel probability distribution on X , with
∫X||Φ(x)||2HK
dρ(x) =
∫X
K (x , x)dρ(x) <∞
RKHS mean vector
µΦ = Eρ[Φ(x)] =
∫X
Φ(x)dρ(x) ∈ HK
For any f ∈ HK
〈µΦ, f 〉HK =
∫X〈f ,Φ(x)〉HK dρ(x) =
∫X
f (x)dρ(x)= Eρ[f ]
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 16 / 99
RKHS covariance operators
RKHS covariance operator CΦ : HK → HK
CΦ = Eρ[(Φ(x)− µ)⊗ (Φ(x)− µ)]
=
∫X
Φ(x)⊗ Φ(x)dρ(x)− µ⊗ µ
For all f ,g ∈ HK
〈f ,CΦg〉HK =
∫X〈f ,Φ(x)〉HK 〈g,Φ(x)〉HK dρ(x)− 〈µ, f 〉HK 〈µ,g〉HK
= Eρ(fg)− Eρ(f )Eρ(g)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 17 / 99
Empirical mean and covariance
X = [x1, . . . , xm] = data matrix randomly sampled from Xaccording to ρ, with m observationsInformally, Φ gives an infinite feature matrix in the feature spaceHK , of size dim(HK )×m
Φ(X) = [Φ(x1), . . . ,Φ(xm)]
Formally, Φ(X) : Rm → HK is the bounded linear operator
Φ(X)w =m∑
i=1
wiΦ(xi), w ∈ Rm
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 18 / 99
Empirical mean and covariance
Theoretical RKHS mean
µΦ =
∫X
Φ(x)dρ(x) ∈ HK
Empirical RKHS mean
µΦ(X) =1m
m∑i=1
Φ(xi) =1m
Φ(X)1m ∈ HK
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 19 / 99
Empirical mean and covariance
Theoretical covariance operator CΦ : HK → HK
CΦ =
∫X
Φ(x)⊗ Φ(x)dρ(x)− µ⊗ µ
Empirical covariance operator CΦ(x) : HK → HK
CΦ(X) =1m
m∑i=1
Φ(xi)⊗ Φ(xi)− µΦ(X) ⊗ µΦ(X)
=1m
Φ(X)JmΦ(X)∗
Jm = Im − 1m 1m1T
m = centering matrix
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 20 / 99
RKHS covariance operators
CΦ is a self-adjoint, positive, trace class operator, with a countableset of eigenvalues {λk}∞k=1, λk ≥ 0 and
∞∑k=1
λk <∞
CΦ(X) is a self-adjoint, positive, finite rank operator
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 21 / 99
Covariance operator representation of images
Given an image F (or a patch in F ), at each pixel, extract a featurevector (e.g. intensity, colors, filter responses etc)Each image corresponds to a data matrix X
X = [x1, . . . , xm] = n ×m matrix
where m = number of pixels, n = number of features at each pixelDefine a kernel K , with corresponding feature map Φ and featurematrix
Φ(X) = [Φ(x1), . . . ,Φ(xm)]
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 22 / 99
Covariance operator representation of images
Each image is represented by covariance operator
CΦ(X) =1m
Φ(X)JmΦ(X)∗
This representation is implicit, since Φ is generally implicitComputations are carried out via Gram matricesCan be approximated by explicit, low-dimensional approximationof Φ
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 23 / 99
Part II - Outline
Infinite-dimensional setting
Covariance Operators and Applications
1 Data Representation by Covariance Operators2 Geometry of Covariance Operators3 Machine Learning Methods on Covariance Operators and
Applications in Computer Vision
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 24 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 25 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 26 / 99
Hilbert-Schmidt metric
Generalizing Frobenius inner product and distance on Sym++(n)
〈A,B〉F = tr(AT B), dE (A,B) = ||A− B||F
H = infinite-dimensional separable real Hilbert space, with acountable orthonormal basis {ek}∞k=1
A : H → H = bounded linear operator
||A|| = supx∈H,x 6=0
||Ax ||||x ||
<∞
Adjoint operator A∗ : H → H (transpose if H = Rn)
〈Ax , y〉 = 〈x ,A∗y〉 ∀x , y ∈ H
Self-adjoint: A∗ = A
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 27 / 99
Hilbert-Schmidt metric
Generalizing the Frobenius inner product 〈A,B〉F = tr(AT B)
Hilbert-Schmidt inner product
〈A,B〉HS = tr(A∗B) =∞∑
k=1
〈ek ,A∗Bek 〉 =∞∑
k=1
〈Aek ,Bek 〉
between Hilbert-Schmidt operators on the Hilbert space H
HS(H) = {A : ||A||2HS = tr(A∗A) =∞∑
k=1
||Aek ||2 <∞}
for any orthonormal basis {ek}k∈N
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 28 / 99
Hilbert-Schmidt metric
Generalizing the Frobenius norm ||A||FHilbert-Schmidt norm
||A||2HS = tr(A∗A) =∞∑
k=1
||Aek ||2 <∞}
A is Hilbert-Schmidt⇒ A is compact, with a countable set ofeigenvalues {λk}∞k=1
If A is self-adjoint
||A||2HS =∞∑
k=1
λ2k
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 29 / 99
Hilbert-Schmidt distance between RKHS covarianceoperators
Two random data matrices X = [x1, . . . , xm] and Y = [y1, . . . , ym]sampled according to two Borel probability distributions ρX and ρYon X give rise to two empirical covariance operators
CΦ(X) =1m
Φ(X)JmΦ(X)∗ : HK → HK
CΦ(Y) =1m
Φ(Y)JmΦ(Y)∗ : HK → HK
Closed-form expressions for ||CΦ(X) − CΦ(Y)||HS in terms of them ×m Gram matrices
K [X] = Φ(X)∗Φ(X), (K [X])ij = K (xi , xj),
K [Y] = Φ(Y)∗Φ(Y), (K [Y])ij = K (yi , yj),
K [X,Y] = Φ(X)∗Φ(Y), (K [X,Y])ij = K (xi , yj)
K [Y,X] = Φ(Y)∗Φ(X), (K [Y,x])ij = K (yi , xj)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 30 / 99
Hilbert-Schmidt distance between RKHS covarianceoperators
PropositionThe Hilbert-Schmidt distance between CΦ(X) and CΦ(Y)
||CΦ(X) − CΦ(Y)||2HS =1
m2 〈JmK [X],K [X]Jm〉F −2
m2 〈JmK [X,Y],K [X,Y]Jm〉F
+1
m2 〈JmK [Y],K [Y]Jm〉F .
The Hilbert-Schmidt inner product between CΦ(X) and CΦ(Y)
〈CΦ(X),CΦ(Y)〉HS =1
m2 〈JmK [X,Y],K [X,Y]Jm〉F .
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 31 / 99
Hilbert-Schmidt distance - Computational complexity
Requires the calculation of m ×m Gram matrices and theirpointwise multiplication. The computational complexity required isO(m2).For set of N data matrices {Xj}Nj=1, then for computing all thepairwise Hilbert-Schmidt distances/inner products between thecorresponding covariance operators {CΦ(Xj )}
Nj=1, the
computational complexity required is O(N2m2).
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 32 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 33 / 99
Infinite-dimensional generalization of Sym++(n)
The infinite-dimensional setting is significantly different from thefinite-dimensional settingThe functions log, det, different norms || || are defined for specificclasses of operatorsThe regularization (A + γI) is necessary both theoretically andempirically
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 34 / 99
Infinite-dimensional generalization of Sym++(n)
Motivation : Generalizing the Log-Euclidean distance
dlogE(A,B) = || log(A)− log(B)||F , A,B ∈ Sym++(n)
to the setting where A,B are self-adjoint, positive, boundedoperators on a separable Hilbert space HTwo issues to consider
1 Generalization of the principal matrix log function2 Generalization of the Frobenius inner product and norm (the
Hilbert-Schmidt norm is not sufficient)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 35 / 99
Infinite-dimensional generalization of Sym++(n)
Generalizing the principal matrix log(A) function
A ∈ Sym++(n), with eigenvalues {λk}nk=1 and orthonormaleigenvectors {uk}nk=1
A =n∑
k=1
λkukuTk , log(A) =
n∑k=1
log(λk )ukuTk
A : H → H self-adjoint, positive, compact operator, witheigenvalues {λk}∞k=1, λk > 0, limk→∞ λk = 0, and orthonormaleigenvectors {uk}∞k=1
A =∞∑
k=1
λk (uk ⊗ uk ), (uk ⊗ uk )w = 〈uk ,w〉uk
log(A) =∞∑
k=1
log(λk )(uk ⊗ uk ), limk→∞
log(λk ) = −∞
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 36 / 99
Infinite-dimensional generalization of Sym++(n)
First problem: unboundedness of log(A) since limk→∞ λk = 0
log(A) =∞∑
k=1
log(λk )(uk ⊗ uk ), limk→∞
log(λk ) = −∞
Resolution: positive definite operatorsStrict positivity, i.e. λk > 0 ∀k ∈ N is not sufficient. We need A tobe positive definite
〈Ax , x〉 ≥ MA||x ||2, MA > 0
The eigenvalues of A are bounded from below by MA > 0
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 37 / 99
Infinite-dimensional generalization of Sym++(n)
Finite-dimensional setting: {λk}nk=1, λk > 0
A is positive definite⇐⇒A is strictly positive
Infinite-dimensional setting: {λk}∞k=1, λk > 0, limk→∞ λk = 0
A is positive definite⇐⇒{
A is strictly positive andA is invertible
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 38 / 99
Infinite-dimensional generalization of Sym++(n)
First problem: unboundedness of log(A)
Resolution: positive definite operatorsRegularization: examples of positive definite operators areregularized operators of the form
{(A + γI) > 0 | γ ∈ R, γ > 0}
where A is self-adjoint, compact, positive. Then
log(A + γI)
is well-defined and bounded.
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 39 / 99
Infinite-dimensional generalization of Sym++(n)
Generalizing the Log-Euclidean distance
dlogE(A,B) = || log(A)− log(B)||F , A,B ∈ Sym++(n)
to the setting where A,B are self-adjoint, positive, boundedoperators on a separable Hilbert space HTwo issues to consider
1 Generalization of the principal matrix log function2 Generalization of the Frobenius inner product and norm (the
Hilbert-Schmidt norm is not sufficient)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 40 / 99
Infinite-dimensional generalization of Sym++(n)
Second problem: The identity operator I is not Hilbert-Schmidt:
||I||HS = tr(I) =∞
For γ 6= 1
|| log(A + γI)||2HS =∞∑
k=1
[log(λk + γ)]2 =∞
For γ 6= ν
d(γI, νI) = || log(γ/ν)I||HS = | log(γ/ν)| ||I||HS =∞
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 41 / 99
Infinite-dimensional generalization of Sym++(n)
Second problem: The identity I is not Hilbert-SchmidtResolution: Extended (unitized) Hilbert-Schmidt algebra(Larotonda, Differential Geometry and Its Applications, 2007)
HSX (H) = {A + γI : A ∈ HS(H), γ ∈ R}
Extended (unitized) Hilbert-Schmidt inner product
〈A + γI,B + νI〉eHS = 〈A,B〉HS + γν
i.e. the scalar operators γI are orthogonal to the Hilbert-Schmidtoperators
||A + γI||2eHS = ||A||2HS + γ2, ||I||eHS = 1
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 42 / 99
Infinite-dimensional generalization of Sym++(n)
Hilbert manifold of positive definite (unitized) Hilbert-Schmidtoperators (Larotonda 2007)
Σ(H) = {A + γI > 0 : A∗ = A,A ∈ HS(H), γ ∈ R}
Infinite-dimensional manifoldFor (A + γI) ∈ Σ(H)
|| log(A + γI)||2eHS =
∥∥∥∥log
(Aγ
+ I)∥∥∥∥2
HS+ (log γ)2 <∞
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 43 / 99
Infinite-dimensional generalization of Sym++(n)
Manifold Sym++(n) (SPD matrices) generalizes to
Manifold Σ(H) (positive definite operators)
Two major differences with the finite-dimensional setting
1 log(A) is unbounded for a compact operator A2 The identity operator I is not Hilbert-Schmidt
Resolutions1 The regularization form (A + γI), which is positive definite is
necessary both theoretically and empirically2 Extended (unitized) Hilbert-Schmidt inner product and norm
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 44 / 99
Infinite-dimensional generalization of Sym++(n)
Manifold Sym++(n) (SPD matrices) generalizes to
Manifold Σ(H) (positive definite operators)
The following are both well-defined on Σ(H)
Generalization of Affine-invariant Riemannian metricGeneralization of Log-Euclidean metric
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 45 / 99
Infinite-dimensional generalization of Sym++(n)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 46 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 47 / 99
Affine-invariant Riemannian metric
Affine-invariant Riemannian metric: Larotonda (2005), Larotonda(2007), Andruchow and Varela (2007), Lawson and Lim (2013)Larotonda, Nonpositive curvature: A geometrical approach toHilbert-Schmidt operators, Differential Geometry and ItsApplications, 2007
In the setting of RKHS covariance operatorsH.Q.M. Affine-invariant Riemannian distance betweeninfinite-dimensional covariance operators, Geometric Science ofInformation, 2015
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 48 / 99
Affine-invariant Riemannian metric
Tangent space
Finite-dimensional
TP(Sym++(n)) ∼= Sym(n), ∀P ∈ Sym++(n)
Infinite-dimensional: (Larotonda 2007)
TP(Σ(H)) ∼= HR ∀P ∈ Σ(H)
HR = {A + γI : A∗ = A,A ∈ HS(H), γ ∈ R}
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 49 / 99
Affine-invariant Riemannian metric
Riemannian metric
Finite-dimensional: For P ∈ Sym++(n)
〈A,B〉P = 〈P−1/2AP−1/2,P−1/2BP−1/2〉F
where A,B ∈ Sym(n)
Infinite-dimensional: (Larotonda 2007) For P ∈ Σ(H)
〈(A + γI), (B + νI)〉P= 〈P−1/2(A + γI)P−1/2,P−1/2(B + νI)P−1/2〉eHS
where (A + γI), (B + νI) ∈ HR
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 50 / 99
Affine-invariant Riemannian metric
Affine invariance
Finite-dimensional
〈CACT ,CBCT 〉CPCT = 〈A,B〉P
Infinite-dimensional: For any invertible C + δI, C ∈ HS(H)
〈(C + δI)(A + γI)(C + δI)∗, (C + δI)(B + νI)(C + δI)∗〉(C+δI)P(C+δI)∗
= 〈(A + γI), (B + νI)〉P
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 51 / 99
Affine-invariant Riemannian metric
Geodesics
Geodesically complete Riemannian manifoldUnique geodesic joining A,B ∈ Σ(H)
γAB(t) = A1/2(A−1/2BA−1/2)tA1/2
γAB(0) = A, γAB(1) = B
Exponential map ExpP : TP(Σ(H))→ Σ(H)
ExpP(V ) = P1/2 exp(P−1/2VP−1/2)P1/2
is defined for all V ∈ TP(Σ(H))
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 52 / 99
Affine-invariant Riemannian metric
Riemannian distance
Finite-dimensional
daiE(A,B) = || log(A−1/2BA−1/2)||F
Infinite-dimensional: Riemannian distance between (A + γI),(B + νI) ∈ Σ(H)
daiHS[(A + γI), (B + νI)]
= || log[(A + γI)−1/2(B + νI)(A + γI)−1/2]||eHS
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 53 / 99
Affine-invariant Riemannian distance
Proposition (H.Q.M. - GSI 2015)
d2aiHS[(A + γI), (B + νI)] dim(H) =∞
= tr
{log
[(Aγ
+ I)−1(B
ν+ I)]}2
+(
logγ
ν
)2
d2aiHS[(A + γI), (B + νI)] = d2
aiE[(A + γI), (B + νI)] dim(H) <∞
= tr
{log
[(Aγ
+ I)−1(B
ν+ I)]}2
+(
logγ
ν
)2dim(H)
− 2(
logγ
ν
)tr
{log
[(Aγ
+ I)−1(B
ν+ I)]}
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 54 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
For γ > 0, ν > 0,
daiHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )]
= daiHS
[(1m
Φ(X)JmΦ(X)∗ + γIHK
),
(1m
Φ(X)JmΦ(X)∗ + νIHK
)]has a closed form expression via the m ×m Gram matrices
K [X] = Φ(X)∗Φ(X), (K [X])ij = K (xi , xj),
K [Y] = Φ(Y)∗Φ(Y), (K [Y])ij = K (yi , yj),
K [X,Y] = Φ(X)∗Φ(Y), (K [X,Y])ij = K (xi , yj)
K [Y,X] = Φ(Y)∗Φ(X), (K [Y,x])ij = K (yi , xj)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 55 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
Theorem (H.Q.M. - GSI 2015)For dim(HK ) =∞, γ > 0, ν > 0,
d2aiHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )]
= tr
log
C11 C12 C13C21 C22 C23C11 C12 C13
+ I3m
2
+(
logγ
ν
)2
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 56 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
C11 =1νm
JmK [Y]Jm, C21 =1
√γνm
JmK [X,Y]Jm
C12 = − 1√γνm
JmK [Y,X]Jm
(Im +
1γm
JmK [X]Jm
)−1
C13 = − 1γνm2 JmK [Y,X]Jm
(Im +
1γm
JmK [X]Jm
)−1
JmK [X,Y]Jm
C22 = − 1γm
JmK [X]Jm
(Im +
1γm
JmK [X]Jm
)−1
C23 = − 1√γ3νm2
JmK [X]Jm
(Im +
1γm
JmK [X]Jm
)−1
JmK [X,Y]Jm
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 57 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
Theorem (H.Q.M. - GSI 2015)For dim(HK ) <∞, γ > 0, ν > 0,
d2aiHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )]
= tr
log
C11 C12 C13C21 C22 C23C11 C12 C13
+ I3m
2
+(
logγ
ν
)2dim(HK )
− 2(
logγ
ν
)tr
log
C11 C12 C13C21 C22 C23C11 C12 C13
+ I3m
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 58 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
Special case
For linear kernel K (x , y) = 〈x , y〉, x , y ∈ Rn
daiHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )] = daiE[(CX + γIn), (CY + νIn)]
This can be used to verify the correctness of an implementation
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 59 / 99
Affine-invariant Riemannian distance between RKHScovariance operators
For m ∈ N fixed, γ 6= ν,
limdim(HK )→∞
daiHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )] =∞
In general, the infinite-dimensional formulation cannot beapproximated by the finite-dimensional counterpart.
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 60 / 99
Affine-invariant Riemannian distance - Computationalcomplexity
Requires the multiplications and inversions of m ×m matrices,and the eigenvalue computation for a 3m × 3m matrix. Totalcomputational complexity required is O(m3).For a set of N data matrices {Xj}Nj=1, in order to compute all thepairwise distances between the corresponding regularizedcovariance operators, the total computational complexity isO(N2m3).
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 61 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 62 / 99
Log Hilbert-Schmidt metric
H.Q.Minh, M. San Biagio, V. Murino. Log-Hilbert-Schmidt metricbetween positive definite operators on Hilbert spaces, NIPS 2014H.Q.Minh, M. San Biagio, L. Bazzani, V. Murino. ApproximateLog-Hilbert-Schmidt distances between covariance operators forimage classification, CVPR 2016
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 63 / 99
Log-Hilbert-Schmidt metric
Generalization from Sym++(n) to Σ(H)
� : Σ(H)× Σ(H)→ Σ(H)
(A + γI)� (B + νI) = exp[log(A + γI) + log(B + νI)]
~ : R× Σ(H)→ Σ(H)
λ~ (A + γI) = exp[λ log(A + γI)] = (A + γI)λ, λ ∈ R
(Σ(H),�,~) is a vector space� acting as vector addition~ acting as scalar multiplication
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 64 / 99
Log-Hilbert-Schmidt metric
(Σ(H),�,~) is a vector spaceLog-Hilbert-Schmidt inner product
〈A + γI,B + νI〉logHS = 〈log(A + γI), log(B + νI)〉eHS
||A + γI||logHS = || log(A + γI)||eHS
(Σ(H),�,~, 〈 , 〉logHS) is a Hilbert spaceLog-Hilbert-Schmidt distance is the Hilbert distance
dlogHS(A + γI,B + νI) = || log(A + γI)− log(B + νI)||eHS
= ||(A + γI)� (B + νI)−1||logHS
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 65 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
The distance
dlogHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )]
= dlogHS
[(1m
Φ(X)JmΦ(X)∗ + γIHK
),
(1m
Φ(Y)JmΦ(Y)∗ + νIHK
)]has a closed form in terms of m ×m Gram matrices
K [X] = Φ(X)∗Φ(X), (K [X])ij = K (xi , xj),
K [Y] = Φ(Y)∗Φ(Y), (K [Y])ij = K (yi , yj),
K [X,Y] = Φ(X)∗Φ(Y), (K [X,Y])ij = K (xi , yj)
K [Y,X] = Φ(Y)∗Φ(X), (K [Y,x])ij = K (yi , xj)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 66 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
1γm
JmK [X]Jm = UAΣAUTA ,
1νm
JmK [Y]Jm = UBΣBUTB ,
A∗B =1
√γνm
JmK [X,Y]Jm
CAB = 1TNA
log(INA + ΣA)Σ−1A (UT
A A∗BUB ◦ UTA A∗BUB)Σ−1
B log(INB + ΣB)1NB
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 67 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
Theorem (H.Q.M. et al - NIPS2014)Assume that dim(HK ) =∞. Let γ > 0, ν > 0. The Log-Hilbert-Schmidtdistance between (CΦ(X) + γIHK ) and (CΦ(Y) + νIHK ) is
d2logHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )] = tr[log(INA + ΣA)]2 + tr[log(INB + ΣB)]2
− 2CAB + (log γ − log ν)2
The Log-Hilbert-Schmidt inner product between (CΦ(X) + γIHK ) and(CΦ(Y) + νIHK ) is
〈(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )〉logHS = CAB + (log γ)(log ν)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 68 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
Theorem (H.Q.M. et al - NIPS2014)Assume that dim(HK ) =∞. Let γ > 0.The Log-Hilbert-Schmidt norm of the operator (CΦ(X) + γIHK ) is
||(CΦ(X) + γIHK )||2logHS = tr[log(INA + ΣA)]2 + (log γ)2
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 69 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
Theorem (H.Q.M. et al - NIPS2014)Assume that dim(HK ) <∞. Let γ > 0, ν > 0. The Log-Hilbert-Schmidtdistance between (CΦ(X) + γIHK ) and (CΦ(Y) + νIHK ) is
d2logHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )]
= tr[log(INA + ΣA)]2 + tr[log(INB + ΣB)]2 − 2CAB
+ 2(logγ
ν)(tr[log(INA + ΣA)]− tr[log(INB + ΣB)])
+ (log γ − log ν)2dim(HK )
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 70 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
Theorem (H.Q.M. et al - NIPS2014)Assume that dim(HK ) <∞. Let γ > 0, ν > 0. The Log-Hilbert-Schmidtinner product between (CΦ(X) + γIHK ) and (CΦ(Y) + νIHK ) is
〈(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )〉logHS
= CAB + (log ν)tr[log(INA + ΣA)]
+ (log γ)tr[log(INB + ΣB)] + (log γ log ν)dim(HK )
The Log-Hilbert-Schmidt norm of (CΦ(X) + γIHK ) is
||(CΦ(X) + γIHK )||2logHS = tr[log(INA + ΣA)]2 + 2(log γ)tr[log(INA + ΣA)]
+ (log γ)2dim(HK )
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 71 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
Special case
For linear kernel K (x , y) = 〈x , y〉, x , y ∈ Rn
dlogHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )] = dlogE[(CX + γIn), (CY + νIn)]
〈(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )〉logHS = 〈(CX + γIn), (CY + νIn)〉logE
||(CX + γIHK )||logHS = ||(CX + γIn)||logE
These can be used to verify the correctness of an implementation
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 72 / 99
Log-Hilbert-Schmidt distance between RKHScovariance operators
For m ∈ N fixed, γ 6= ν,
limdim(HK )→∞
dlogHS[(CΦ(X) + γIHK ), (CΦ(Y) + νIHK )] =∞
In general, the infinite-dimensional formulation cannot beapproximated by the finite-dimensional counterpart.
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 73 / 99
Log-Hilbert-Schmidt distance - Computationalcomplexity
Requires the SVD and multiplications of Gram matrices of sizem ×m. The computational complexity required is O(m3).For a set of N data matrices {Xj}Nj=1, then the computationalcomplexity required for computing all pairwise distances betweenthe corresponding regularized covariance operators is O(N2m3).
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 74 / 99
Approximate methods for reducing computationalcosts
M. Faraki, M. Harandi, and F. Porikli, Approximateinfinite-dimensional region covariance descriptors for imageclassification, ICASSP 2015H.Q. Minh, M. San Biagio, L. Bazzani, V. Murino. ApproximateLog-Hilbert-Schmidt distances between covariance operators forimage classification, CVPR 2016Q. Wang, P. Li, W. Zuo, and L. Zhang. RAID-G: Robust estimationof approximate infinite-dimensional Gaussian with application tomaterial recognition, CVPR 2016
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 75 / 99
Geometry of Covariance Operators
1 Hilbert-Schmidt metric: generalizing the Euclidean metric2 Manifold of positive definite operators
Infinite-dimensional generalization of Sym++(n)Infinite-dimensional affine-invariant Riemannian metric:generalizing finite-dimensional affine-invariant Riemannian metricLog-Hilbert-Schmidt metric: generalizing Log-Euclidean metric
3 Convex cone of positive definite operatorsLog-Determinant divergences: generalize the finite-dimensionalLog-Determinant divergences
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 76 / 99
Log-Determinant divergences
Zhou and Chellappa (PAMI 2006), Harandi et al (CVPR 214):finite-dimensional RKHSH.Q.M. Infinite-dimensional Log-Determinant divergencesbetween positive definite trace class operators, Linear Algebraand its Applications, 2017H.Q.M. Log-Determinant divergences between positive definiteHilbert-Schmidt operators, Geometric Science of Information,2017
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 77 / 99
Summary - Geometry of Covariance Operators
Hilbert-Schmidt distance and inner productAffine-invariant Riemannian distanceLog-Hilbert-Schmidt distance and inner productExplicit formulas via Gram matrices in the case of RKHScovariance operators
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 78 / 99
Part II - Outline
Infinite-dimensional setting
Covariance Operators and Applications
1 Data Representation by Covariance Operators2 Geometry of Covariance Operators3 Machine Learning Methods on Covariance Operators and
Applications in Computer Vision
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 79 / 99
Kernels with Log-Hilbert-Schmidt metric
(Σ(H),�,~, 〈 , 〉logHS) is a Hilbert space
Theorem (H.Q.M. et al - NIPS 2014)The following kernels K : Σ(H)× Σ(H)→ R are positive definite
K [(A + γI), (B + νI)] = (c + 〈A + γI,B + νI〉logHS)d
c ≥ 0,d ∈ N
K [(A + γI), (B + νI)] = exp(− 1σ2 || log(A + γI)− log(B + νI)||peHS)
0 < p ≤ 2, σ 6= 0
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 80 / 99
Two-layer kernel machine with Log-Hilbert-Schmidtmetric
1 First layer: kernel K1, inducing covariance operators2 Second layer: kernel K2, defined using the Log-Hilbert-Schmidt
distance or inner product between the covariance operators
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 81 / 99
Two-layer kernel machine with Log-Hilbert-Schmidtmetric
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 82 / 99
Experiments in Image Classification
Multi-class image classification using multi-class SVM with theGaussian kernelEach image is represented by one covariance operatorFish recognitionMaterial classification
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 83 / 99
Fish recognition
(Boom et al, Ecological Informatics, 2014)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 84 / 99
Fish recognition
Using R,G,B channels
Method Accuracy
E 26.9% (±3.5%)
Stein 43.9% (±3.5%)
Log-E 42.7% (±3.4%)
HS 50.2% (±2.2%)
Log-HS 56.7% (±2.9%)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 85 / 99
Material classification
Example: KTH-TIPS2b data set (Caputo et al, ICCV, 2005)
f(x , y) =[R(x , y),G(x , y),B(x , y),
∣∣G0,0(x , y)∣∣ , . . . ∣∣G3,4(x , y)
∣∣]
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 86 / 99
Material classification
Method KTH-TIPS2b
E 55.3% (±7.6%)
Stein 73.1% (±8.0%)
Log-E 74.1 % (±7.4%)
HS 79.3% (±8.2%)
Log-HS 81.9% (±3.3%)
Log-HS (CNN) 96.6% (±3.4%)
CNN features = MatConvNet features
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 87 / 99
Object recognition
Example: ETH-80 data set
f(x , y) = [x , y , I(x , y), |Ix |, |Iy |]
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 88 / 99
Object recognition
Results obtained using approximate Log-HS distance
Method ETH-80
E 64.4%(±0.9%)
Stein 67.5% (±0.4%)
Log-E 71.1%(±1.0%)
HS 93.1 % (±0.4)
Approx-LogHS 95.0% (±0.5%)
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 89 / 99
Summary: Kernel methods with covariance operators
Two-layer kernel machine with Log-Hilbert-Schmidt metricSubstantial gains in performance compared to finite-dimensionalcase, but higher computational costApproximate methods for reducing computational costs
H.Q. Minh, M. San Biagio, L. Bazzani, V. Murino. ApproximateLog-Hilbert-Schmidt distances between covariance operators forimage classification, CVPR 2016
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 90 / 99
Summary: Infinite-dimensional setting
1 Data Representation by Covariance Operators2 Geometry of Covariance Operators
Hilbert-Schmidt distance and inner productAffine-invariant Riemannian distance and inner productLog-Hilbert-Schmidt distance and inner product
3 Machine Learning Methods on Covariance Operators andApplications in Computer Vision
Two-layer kernel machine with Log-Hilbert-Schmidt metricExperiments in image classification
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 91 / 99
Summary: Infinite-dimensional setting
Significantly different from finite-dimensional caseThe classes of operators, i.e. positive definite operators, extendedHilbert-Schmidt operators, must be defined carefullyLog-Euclidean metric, affine-invariant Riemannian metric,Bregman divergences can all be generalized toinfinite-dimensional settingCan obtain substantial gains in performance compared tofinite-dimensional case, but with higher computational costs
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 92 / 99
Summary: Infinite-dimensional setting
Finite-dimensional approximation methods can be applied undercertain conditions, but not in generalStill undergoing active development, both theoretically andcomputationally
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 93 / 99
Conclusion
1 Overview of the finite-dimensional settingCovariance matrix representationGeometry of SPD matricesKernel methods with covariance matrices
2 Generalization to the infinite-dimensional setting via kernelmethods
Covariance operator representationGeometry of covariance operatorsKernel methods with covariance operators
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 94 / 99
Some future directions
Applications of covariance operatorsComputer visionImage and signal processingMachine learning and statisticsOther fields
Covariance matrices and covariance operators in deep learningC. Ionescu, O. Vantzos, and C. Sminchisescu. Matrixbackpropagation for deep networks with structured layers, ICCV2015Q. Wang, P. Li, W. Zuo, and L. Zhang. RAID-G: Robust estimationof approximate infinite-dimensional Gaussian with application tomaterial recognition, CVPR 2016
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 95 / 99
Some future directions
Connection between the geometry of positive definite operatorsand infinite-dimensional Gaussian probability measures
Finite-dimensional setting
Affine-invariant Riemannian metric⇐⇒ Fisher-Rao metricLog-Determinant divergences⇐⇒ Reny divergences
Infinite-dimensional setting: upcoming work
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 96 / 99
References
M. Faraki, M. Harandi, and F. Porikli, Approximateinfinite-dimensional region covariance descriptors for imageclassification, ICASSP 2015M. Harandi, M. Salzmann, and F. Porikli. Bregman divergences forinfinite-dimensional covariance matrices, CVPR 2014Larotonda. Nonpositive curvature: A geometrical approach toHilbert-Schmidt operators, Differential Geometry and ItsApplications, 2007H.Q.Minh. Affine-invariant Riemannian distance betweeninfinite-dimensional covariance operators, Geometric Science ofInformation, 2015H.Q.Minh. Infinite-dimensional Log-Determinant divergencesbetween positive definite trace class operators, Linear Algebraand its Applications, 2017
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 97 / 99
References
H.Q.Minh. Log-Determinant divergences between positive definiteHilbert-Schmidt operators, Geometric Science of Information,2017H.Q.Minh, M. San Biagio, V. Murino. Log-Hilbert-Schmidt metricbetween positive definite operators on Hilbert spaces, NIPS 2014H.Q.Minh, M. San Biagio, L. Bazzani, V. Murino. ApproximateLog-Hilbert-Schmidt distances between covariance operators forimage classification, CVPR 2016Q. Wang, P. Li, W. Zuo, and L. Zhang. RAID-G: Robust estimationof approximate infinite-dimensional Gaussian with application tomaterial recognition, CVPR 2016S.K. Zhou and R. Chellappa. From sample similarity to ensemblesimilarity: Probabilistic distance measures in reproducing kernelHilbert space, PAMI 2006
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 98 / 99
Thank you for listening!Questions, comments, suggestions?
H.Q. Minh (IIT) Covariance matrices & covariance operators November 30, 2017 99 / 99