Restoration of Missing Data due to Clouds on Optical Satellite Imagery Using Neural

Restoration of Missing Data due to Clouds on Optical Satellite Imagery Using Neural

Sergii Skakun1, Nataliia Kussul1, Ruslan Basarab2 1 Space Research Institute NAS and SSA Ukraine 2 National University of Life and Environmental Sciences of Ukraine Sentinel-2 for Science Workshop May 22, 2014, ESA/ESRIN, Frascati, Italy

S2 for

Science

2014 2

Content

• Objective of the study

• Methodology – Reconstruction of missing data using self-organising

Kohonen maps (SOM)

• Data used – Landsat-8

– Sich-2

• Results – Reconstruction of missing data in Landsat-8 and

Sich-2

– Large scale crop mapping using reconstructed imagery (Landsat-8)

• Discussion & conclusions

S2 for

Science

2014 3

Objective of the study

• Clouds and shadows – limiting factor in exploitation of optical

satellite imagery

– cause missing data

• Existing approaches (filling missing data) – inpainting-based (Lorenzi et al. 2011)

– multispectral-based • using MODIS (Roy et al. 2008)

– multitemporal-based • using SOM for MODIS time-series (Latif

& Mercier 2010)

• Objectives – to qualitatively assess the use of SOMs

for restoring missing data on time-series of high and medium resolution satellite images

– to provide crop mapping using restored images

Landsat-8 images over

JECAM Ukraine

02.05.2013

18.05.2013

02.05.2013

18.05.2013

02.05.2013

18.05.2013

19.06.2013

05.07.2013

S2 for

Science

2014 4

Methodology: SOM

• Self-organizing Kohonen maps (SOMs) – type of artificial neural network

– unsupervised learning

– produces a 2D, discretized representation of the input space

SOM architecture

SOM training process

lLl

i wxx ,1

minarg)(

Training sample Neuron

winner Updating weights

Llnnhnnn jilll ,1)),()(()()()1( )(, wxww x

S2 for

Science

2014 5

Methodology: Restoration

• Performed for each spectral band separately

• Training phase: – Training samples selected

automatically: on a regular grid

– Only pixels with all valid (i.e. non-missing) values considered

Restoration of missing values

Landsat-8 time-

series (Band 4)

X1 X2 X3 Nan X5 Nan

Input

SOM: selection of neuron winner

X1 X2 X3 Nan X5 Nan

wl1 wl2 wl3 wl4 wl5 wl6

Missing

Only valid components are considered for

finding a neuron winner

wi1 wi2 wi3 wi4 wi5 wi6

Missing components

are taken from

neuron winner

S2 for

Science

2014 6

Data used & experiment setup

• Landsat-8 – Resolution: 30 m

– Pre-processing:

• DN -> TOA -> SR

• Simplified Model for Atmospheric Correction (SMAC) (Rahman & Dedieu 1994)

– Dates:

• 16 April; 2 and 18 May; 19 June 2013

• Sich-2 – Resolution: 8 m

– DN used

– Dates:

• 3 June; 4, 14 and 19 September 2013

• Qualitative assessment – 2 ROIs selected that

were artificially assigned Nan values

– Metrics: RMSE and Relative RMSE (RRMSE)

ROI1

ROI2

Landsat-8

(true colour)

Sich-2 (false colour)

S2 for

Science

2014 7

Results (1)

Average RMSE error of reconstructing missing values for

ROI1 and ROI2 on Landsat-8 images. RMSE values are

shown depending on the number of missing values in the

time-series (M=1, 2, 3) and fraction of pixels taken for

training. X-axis is shown in a logarithmic scale.

S2 for

Science

2014 8

Results (2)

Average RRMSE error of reconstructing missing values for

ROI1 and ROI2 on Landsat-8 images. RRMSE values are

shown depending on the number of missing values in the

time-series (M=1, 2, 3) and fraction of pixels taken for

training. X-axis is shown in a logarithmic scale.

S2 for

Science

2014 9

Results (3)

Average RMSE and RRMSE error of reconstructing missing

values for ROI1 and ROI2 on Sich-2 images. RMSE and

RRMSE values are shown depending on the number of missing

values in the time-series (M=1, 2, 3) and fraction of pixels taken

for training. X-axis is shown in a logarithmic scale.

S2 for

Science

2014 10

Results (4)

Restoration of ROI1 for Landsat-8

S2 for

Science

2014 11

Crop mapping (1)

• Landsat-8 time-series: – Dates (6 images):

• 16 April, 02 May, 18 May, 19 June, 05 July, 06 August 2013

– Path/row:

• 181/24, 181/25, 181/26

– Pre-processing:

• TOA->DN->SR – SR: Simplified Model for

Atmospheric Correction (SMAC) (Rahman & Dedieu 1994)

– Aerosol optical depth: Aeronet station (Kyiv)

• Clouds & shadows detection – Fmask (Zhu & Woodcock

2012)

• Filling missing values – SOMs

TOA (top) and SR (bottom) for Landsat-8

acquired on 08 August 2013. A true color

composition of Landsat-8 bands 4-3-2. TOA and

SR reflectance are scaled from 0 to 0.15.

S2 for

Science

2014 12

Crop mapping (2)

Example of restoration for

Landsat-8 image

acquired on 05 July 2013

(true color composite)

S2 for

Science

2014 13

Crop mapping (3)

• Ground observations – Along the roads

• ~ 390 polygons

• Classes: No. LUCAS Description

1 Axx Artificial

2 B11 Winter wheat (and barley)

3 B32 Winter rapeseed

4 B12, B14 Spring crops (wheat, barley)

5 B16 Maize

6 B22 Sugar beet

7 B31 Sunflower

8 B33 Soybeans

9 B19, B39, B40

Other cereals, other annual crops, temporary grass

10 Cxx, B60 Forest, fruit trees

11 Exx Grassland

12 Fxx Bare land

13 Gxx Water

Location of along the

roads surveys within

Kyiv oblasts

Distribution of training (50%) and test (50%) data

Train

Test

S2 for

Science

2014 14

Crop mapping (4)

• Methodology – Inputs:

• Restored SR values (bands 2-7) for 6 multi-temporal images

• Total inputs: 36

– Ensemble of 6 neural nets:

• 30, 40, 50, 60, 70 and 80 hidden units

– Ensemble: • Max average a-

posteriori probability

Best single net

Ensemble

OA, % 83.90 85.07

Kappa 0.807 0.820

PA, % UA, %

1 Artificial 100.0 98.4

2 Winter wheat 95.4 91.6

3 Winter rapeseed 93.2 99.3

4 Spring crops 28.3 27.2

5 Maize 91.1 86.1

6 Sugar beet 94.7 90.2

7 Sunflower 84.2 86.4

8 Soybeans 68.8 77.5

9 Other cereals 71.3 75.7

10 Forest 97.0 92.7

11 Grassland 90.6 88.9

12 Bare land 86.9 98.8

13 Water 100.0 98.6

S2 for

Science

2014 15

Crop mapping (5)

Landsat-8 image of 05

July 2013 (true color

composite)

Restored Landsat-8

image from 05 July

2013 using SOMs

Crop map

S2 for

Science

2014 16

Conclusions

• SOMs for high & med resolution satellite images – encodes samples with non-missing values during

a training phase, and then reconstructs missing values from SOM weights

• Accuracy of reconstruction – Landsat-8: most accurate for NIR bands (11-15%)

comparing to visible bands (16-19%)

– Sich-2: relative error green (4.3%), red (5.8%), and NIR (8%)

• Efficient for large scale crop mapping

S2 for

Science

2014 17

Thank you!