implementing kohonen's som with missing data in otb
DESCRIPTION
Implementing kohonen's som with missing data in OTB Gregoire Mercier; Institut Telecom; Telecom Bretagne Bassam Abdel Latif; Institut Telecom; Telecom BretagneTRANSCRIPT
![Page 1: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/1.jpg)
Implementing Kohonen’s SOM withMissing Data in OTBGregoire Mercier and Bassam Abdel LatifInstitut Telecom; Telecom BretagneCNRS UMR 3192 lab-STICC, team CIDTechnopole Brest-Iroise,CS 83818, F-29238 Brest Cedex 3, FranceIGARSS, 2009
![Page 2: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/2.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Contents
1 The Kohonen MapPropertiesDescription
2 Clouds and Shadows in times seriesModification to deal with missing valueModification to deal with erroneous value
3 Benefits in Generic Programming
page 2 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 3: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/3.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
The Kohonen Map
Properties
� Neural network of 1 fully connected layer to be trained by supervized orunsupervized approaches
� Corresponds to a transformation Rn =⇒ R, R2 or R3
• Visualization• Compression
� Neurons in their neighborhood give similar response# Self-Organizing Map (SOM)
• Clustering
� Often use when neighborhood of a class has significant meaning• Classification• Pattern recognition• Data mining• . . .
page 3 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 4: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/4.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
The Kohonen Map
Properties
� Neural network of 1 fully connected layer to be trained by supervized orunsupervized approaches
� Corresponds to a transformation Rn =⇒ R, R2 or R3
• Visualization• Compression
� Neurons in their neighborhood give similar response# Self-Organizing Map (SOM)
• Clustering
� Often use when neighborhood of a class has significant meaning• Classification• Pattern recognition• Data mining• . . .
page 3 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 5: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/5.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
The Kohonen Map
Properties
� Neural network of 1 fully connected layer to be trained by supervized orunsupervized approaches
� Corresponds to a transformation Rn =⇒ R, R2 or R3
• Visualization• Compression
� Neurons in their neighborhood give similar response# Self-Organizing Map (SOM)
• Clustering
� Often use when neighborhood of a class has significant meaning• Classification• Pattern recognition• Data mining• . . .
page 3 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 6: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/6.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
The Kohonen Map
Properties
� Neural network of 1 fully connected layer to be trained by supervized orunsupervized approaches
� Corresponds to a transformation Rn =⇒ R, R2 or R3
• Visualization• Compression
� Neurons in their neighborhood give similar response# Self-Organizing Map (SOM)
• Clustering
� Often use when neighborhood of a class has significant meaning• Classification• Pattern recognition• Data mining• . . .
page 3 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 7: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/7.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 8: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/8.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 9: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/9.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 10: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/10.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 11: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/11.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 12: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/12.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
PropertiesDescription
Training Illustration
� Neurons random initialization (cm)
� Select randomly a training sample x and find thewinning neuron cmx :
‖x− cmx‖ = minm∈{1,...,M}
‖x− cm‖
� Update the weight of the winning neuron and of itsneighborhood:
cm(t + 1) = cm(t) + hm,mx(t)[x(t)− cm(t)]
� Loop until convergence
page 4 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 13: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/13.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Contents
1 The Kohonen MapPropertiesDescription
2 Clouds and Shadows in times seriesModification to deal with missing valueModification to deal with erroneous value
3 Benefits in Generic Programming
page 5 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 14: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/14.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous data,clouds
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 15: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/15.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous Data,clouds
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 16: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/16.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Missing Data, sensor
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 17: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/17.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous Data,clouds
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 18: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/18.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous Data,clouds
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 19: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/19.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Clean Data
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 20: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/20.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Missing Data, sensor
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 21: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/21.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous Data,clouds
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 22: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/22.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Clean Data
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 23: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/23.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Problem: 10 dates in a MODIS time series over Brittany, France
MODIS time series (NIR band, 250m resol. 10dates). Images choosen with a cloud coverage
below 50%, zenithal angle below 20◦ overt centerof Brittany.
� 25-nov-2002
� 5-jan-2003
� 24-jan-2003
� 4-feb-2003
� 22-feb-2003
� 15-mar-2003
� 17-mar-2003
� 19-mar-2003
� 7-apr-2003
� 16-apr-2003
Erroneous Data,sensor
page 6 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 24: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/24.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
From Erroneous to Missing data
Type of errors
� Presence of clouds (± thickness)
� Presence of shadow
� Impulse noise
� outliers
Simple Cloud/Shadow Detector
Based on the Box and Whisker technique, rank statistics
let x = {x1, . . . , xk , . . .} be a time series
xk is erroneous if˛xk − α
`x3/4 − x1/4
´˛> |x1/2|, α = 1.5
Neuron definition
x = {NIR1, . . . ,NIRk , . . . ,NIR10}
where NIRi = stands for NIR band of MODIS at time i
page 7 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 25: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/25.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
From Erroneous to Missing data
Type of errors
� Presence of clouds (± thickness)
� Presence of shadow
� Impulse noise
� outliers
Simple Cloud/Shadow Detector
Based on the Box and Whisker technique, rank statistics
let x = {x1, . . . , xk , . . .} be a time series
xk is erroneous if˛xk − α
`x3/4 − x1/4
´˛> |x1/2|, α = 1.5
Neuron definition
x = {NIR1, . . . ,NIRk , . . . ,NIR10}
where NIRi = stands for NIR band of MODIS at time i
page 7 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 26: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/26.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
From Erroneous to Missing data
Type of errors
� Presence of clouds (± thickness)
� Presence of shadow
� Impulse noise
� outliers
Simple Cloud/Shadow Detector
Based on the Box and Whisker technique, rank statistics
let x = {x1, . . . , xk , . . .} be a time series
xk is erroneous if˛xk − α
`x3/4 − x1/4
´˛> |x1/2|, α = 1.5
Neuron definition
x = {NIR1, . . . ,NIRk , . . . ,NIR10}
where NIRi = stands for NIR band of MODIS at time i
page 7 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 27: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/27.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Kohonen Map Modification
Deal with missing value
In a series x = {x1, . . . , xk , . . .}
if xk is missing ⇐⇒ k ∈Mx
Where Mx is a set that contains indices of missing components
Winning Neuron Evaluation
‖x− cm‖2 =X
j /∈Mx
“x(j)− cm(j)
”2
Winning Neuron Update cmx
cm;k(t + 1) =
(cm;k(t) + hm,mx(t)
hxk − cm;k(t)
iif k /∈Mx,
cm;k(t) if k ∈Mx.
page 8 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 28: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/28.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
25-nov-2002
Associated neuron value
25-nov-2002
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 29: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/29.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
5-jan-2003
Associated neuron value
5-jan-2003
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 30: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/30.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
24-jan-2003
Associated neuron value
24-jan-2003
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 31: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/31.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
4-feb-2003
Associated neuron value
4-feb-2003
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 32: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/32.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
19-mar-2003
Associated neuron value
19-mar-2003
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 33: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/33.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Results
Initial NIR Band
16-apr-2003
Associated neuron value
16-apr-2003
page 9 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 34: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/34.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Validation
Comparison with compositing products of MODIS/Terra
Kohonen MOD13Q1 MOD09Q1
CV-MVC through 16 days CV-MVC through 8 days
18-jan-2003 a 02-feb-2003 18-jan-2003 a 25-jan-2003
page 10 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 35: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/35.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Appropriated Similarity Measure
Vanishing outiers impact
Euclidean Measure (no detection):
ED(x, y) =nX
i=1
(xi − yi )2
Euclidean Measure (with missing value):
ED(x, y) =X
i /∈Mx
(xi − yi )2
Sparse Measure:
D(x, y) =nX
i=1
|xai − y a
i |b, b > 0, a > 0
page 11 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 36: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/36.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Appropriated Similarity Measure
Vanishing outiers impact
Euclidean Measure (no detection):
ED(x, y) =nX
i=1
(xi − yi )2
Euclidean Measure (with missing value):
ED(x, y) =X
i /∈Mx
(xi − yi )2
Sparse Measure:
D(x, y) =nX
i=1
|xai − y a
i |b, b > 0, a > 0
page 11 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 37: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/37.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Appropriated Similarity Measure
Vanishing outiers impact
Euclidean Measure (no detection):
ED(x, y) =nX
i=1
(xi − yi )2
Euclidean Measure (with missing value):
ED(x, y) =X
i /∈Mx
(xi − yi )2
Sparse Measure:
D(x, y) =nX
i=1
|xai − y a
i |b, b > 0, a > 0
page 11 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 38: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/38.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
How to set parameters a and b
0
20
40
60
80
100
0 0.5 1 1.5 2 2.5 3
Cor
rect
Mat
chin
g %
a or b
a, b = 1b, a = 1
0
20
40
60
80
100
0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
Cor
rect
Mat
chin
g %
b
Heavy-tailed
Same distribution
0
20
40
60
80
100
0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
Cor
rect
mat
chin
g %
b
a = 1
a =0.1
� 0 < a, b < 1.
� fit ≈ 100% when b = 0.1.
� D(x, y) =P
i |xi − yi |0.1
page 12 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 39: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/39.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Comparison with other distances
a) Euclidean Distance ED(x, y) =pPn
i=1(xi − yi )2
b) Spectral Angle SA(x, y) = cos−1
„<x, y>
‖x‖ ‖y‖
«
c) Spectral Correlation SCorr(x, y) = E(xy−xyσxσy
d) Spectral Info Divergence SID(x, y) = D(x‖y) + D(y‖x)
e) Sparse Distance D(x, y) =P
i |xi − yi |0.1
page 13 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 40: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/40.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Modification to deal with missing valueModification to deal with erroneous value
Comparison with other distances
Statistiques des images de difference
a) ED b) SA c) SCorr d) SID e) Sparse D
SOM (no outlier detection at training)
µ -.03575 0.03665 0.03606 0.03878 -0.0002
σ .05594 0.13173 0.14953 0.1218 0.01607
SOM (with outlier detection at training)
µ -0.00736 -0.00233 0.00228 -0.00246 -0.00018
σ 0.03516 0.08345 0.09807 0.07718 0.01609
page 13 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 41: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/41.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Contents
1 The Kohonen MapPropertiesDescription
2 Clouds and Shadows in times seriesModification to deal with missing valueModification to deal with erroneous value
3 Benefits in Generic Programming
page 14 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 42: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/42.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map
� Periodic SOM for training
� SOM for classification
� SOM for segmentation
typedef itk::Statistics::EuclideanDistance< VectorType >DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::PeriodicSOM< SampleListType, MapType,
LearningBehaviorFunctorType, NeighborhoodBehaviorFunctorType >SOMType;
SOMType::Pointer som = SOMType::New();som->SetListSample( sampleList );som->Set...som->Update();
page 15 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 43: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/43.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map
� Periodic SOM for training
� SOM for classification
� SOM for segmentation
typedef itk::Statistics::EuclideanDistance< VectorType >DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::SOMClassifier<SampleType,SOMMapType,LabelPixelType>
ClassifierType;
ClassifierType::Pointer classifier = ClassifierType::New();classifier->SetSample(sample.GetPointer());classifier->SetMap(somreader->GetOutput());classifier->Update();
page 15 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 44: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/44.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map
� Periodic SOM for training
� SOM for classification
� SOM for segmentation
typedef itk::Statistics::EuclideanDistance< VectorType >DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::SOMbasedImageFilter<SampleType,SOMMapType,LabelPixelType>
FilterType;
FilterType::Pointer filter = FilterType::New();filter->SetInputImage( inputImage );filter->SetMap(somReader->GetOutput());filter->Update();
page 15 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 45: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/45.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
typedef itk::Statistics::EuclideanDistance<VectorType>DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::PeriodicSOM< SampleListType, MapType,
LearningBehaviorFunctorType, NeighborhoodBehaviorFunctorType >SOMType;
SOMType::Pointer som = SOMType::New();som->SetListSample( sampleList );som->Set...som->Update();
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 46: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/46.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
typedef otb::Statistics::EuclideanDistanceWithMissingValue<VectorType>DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::PeriodicSOM< SampleListType, MapType,
LearningBehaviorFunctorType, NeighborhoodBehaviorFunctorType >SOMType;
SOMType::Pointer som = SOMType::New();som->SetListSample( sampleList );som->Set...som->Update();
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 47: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/47.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
typedef otb::Statistics::EuclideanDistanceWithMissingValue<VectorType>DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::PeriodicSOM< SampleListType, MapType,
LearningBehaviorFunctorType, NeighborhoodBehaviorFunctorType >SOMType;
typedef otb::BoxAndWhiskerImageFilter<ImageType> CloudFilterType;
SOMType::Pointer som = SOMType::New();som->SetListSample( sampleList );som->Set...som->Update();
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 48: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/48.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
typedef otb::Statistics::EuclideanDistanceWithMissingValue<VectorType>DistanceType;
typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::SOMbasedImageFilter<SampleType,SOMMapType,LabelPixelType>
FilterType;
FilterType::Pointer filter = FilterType::New();filter->SetInputImage( inputImage );filter->SetMap(somreader->GetOutput());filter->Update();
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 49: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/49.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
typedef otb::Statistics::FlexibleDistanceWithMissingValue<VectorType>DistanceType;
DistanceType::SetAlphaBeta(a,b);typedef otb::SOMMap< VectorType, DistanceType, Dimension > MapType;typedef otb::SOMbasedImageFilter<SampleType,SOMMapType,LabelPixelType>
FilterType;
FilterType::Pointer filter = FilterType::New();filter->SetInputImage( inputImage );filter->SetMap(somreader->GetOutput());filter->Update();
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 50: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/50.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Generic Programming
The OTB code with Kohonen map for missing value
� Periodic SOM for training with erroneous data
� SOM for recovering missing data
� SOM for recovering erroneous data
� SOM for what particular application?
� Set your own distance
� Set the neurone type
� Set the kohonen map training procedure
� Set the Kohonen map way of using...
page 16 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 51: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/51.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Implementing Kohonen’s SOM with Missing Data in OTB
Conclusion
� Generic tool for using missing and erroneous data
� Thematic part validated by the COSTEL for land use purpose
� May be adapted for many kind of supervized/unsupervised learningprocedure based on the Kohonen map.
� Just use it!
page 17 G Mercier & B Abdel Latif Kohonen map in the OTB
![Page 52: Implementing kohonen's som with missing data in OTB](https://reader033.vdocuments.us/reader033/viewer/2022051610/54983f34b47959744d8b5497/html5/thumbnails/52.jpg)
The Kohonen MapClouds and Shadows in times series
Benefits in Generic Programming
Implementing Kohonen’s SOM with Missing Data in OTB
Conclusion
� Generic tool for using missing and erroneous data
� Thematic part validated by the COSTEL for land use purpose
� May be adapted for many kind of supervized/unsupervised learningprocedure based on the Kohonen map.
� Just use it!
page 17 G Mercier & B Abdel Latif Kohonen map in the OTB