This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEE GEOSCIENCE AND REMOTE SENSING LETTERS 1

Transferred Deep Learning for Sea Ice ChangeDetection From Synthetic-Aperture Radar Images

Yunhao Gao, Feng Gao , Junyu Dong , and Shengke Wang

Abstract— High-quality sea ice monitoring is crucial tonavigation safety and climate research in the polar regions.In this letter, a transferred multilevel fusion network (MLFN)is proposed for sea ice change detection from synthetic-apertureradar (SAR) images. Considering the fact that training data arelimited in the task of sea ice change detection, a large data setwas used to train the MLFN, and the deep knowledge can betransferred to sea ice analysis. In addition, cascade dense blocksare employed to optimize the convolutional layers. Multilayerfeature fusion is introduced to exploit the complementary infor-mation among low-, mid-, and high-level feature representations.Therefore, more discriminative feature extraction can be achievedby the MLFN. Furthermore, the fine-tune strategy is utilized tooptimize the network parameters. The experimental results ontwo real sea ice data sets demonstrated that the proposed methodachieved better performance than other competitive methods.

Index Terms— Change detection, deep learning, fine-tune,neural network, synthetic-aperture radar (SAR).

I. INTRODUCTION

THERE have been growing interests in the Arcticand Antarctic observations for shipping and climate

research. As a result, shipping activities begin to increaserapidly recently. However, many ships are not ice-strengthened, and then high-quality sea ice monitoringis very important for navigation safety. In order to providevaluable information for safe navigation, changed informationof the ice coverage is of great significance.

In this letter, we focus on sea ice change detection which isdefined as detecting changed areas between two images cap-tured at different times over the same geographical region [1].Due to the frequent cloud coverage and long periods of dark-ness in the polar regions, optical sensors can hardly generatecontinues observation. Instead, synthetic-aperture radar (SAR)sensors are used quite common to sea ice monitoring dueto their strong ability to generate observations regardless ofthe weather conditions (cloud or rain). However, the intrinsicspeckle noise makes the interpretation of SAR images chal-lenging. Currently, due to the lack of suitable automatic seaice change detection methods, SAR images are manually inter-preted by human experts working in national ice agencies [2].

Manuscript received November 1, 2018; revised February 20, 2019;accepted March 15, 2019. This work was supported in part by the NationalNatural Science Foundation of China under Grant 41606198, Grant 41576011,and Grant U1706218. (Corresponding author: Feng Gao.)

The authors are with the Qingdao Key Laboratory of Mixed Realityand Virtual Ocean, School of Information Science and Engineering, OceanUniversity of China, Qingdao 266100, China (e-mail: gaofeng@ouc.edu.cn).

Color versions of one or more of the figures in this letter are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/LGRS.2019.2906279

The manual interpretations are time-consuming and are highlydependent on the skills of the experts. Therefore, efficientautomatic sea ice change detection techniques need to bedeveloped.

There have been constant efforts in automating the seaice change detection task. Clustering methods have beenwidely used for change detection. Celik [3] proposed a changedetection method by partition pixel into two groups byk-means clustering. Gong et al. [4] presented a reformulatedfuzzy local information c-means clustering method whichcould reduce the influence of speckle noises by adding a fuzzyfactor to the objective function. Besides clustering methods,threshold methods are also very popular with the changedetection task. Bruzzone and Prieto [5] modeled the differenceimage (DI) by Gaussian distribution where the expectationmaximization (EM) algorithm was used to determine an opti-mal threshold. Xiong et al. [6] proposed a threshold methodfor SAR change detection which combined the features of twochange detection measures by the Markov random field model.Clustering and threshold methods are simple to implement andeasy to understand. However, these methods are prone to beaffected by the speckle noise, and the contextual informationaround each pixel is not fully exploited.

Sea ice image interpretation is a complex recognition taskthat requires machine learning algorithm with strong capabil-ities to learn the nonlinear relationship between multitempo-ral images. Recently, with the great breakthroughs by deeplearning methods in visual recognition tasks [7]–[9], remotesensing image interpretations also benefit from deep models.Hou et al. [10] proposed a change detection method based onmultilevel convolutional neural networks (CNN) and low-rankdecomposition. Wang et al. [11] presented a general end-to-end 2-D CNN framework for hyperspectral image changedetection. For remote sensing images, CNN can effectivelymodel local image structures at different scales, which isachieved by local connectivity between the adjacent layersand weight sharing within one layer. However, a large volumeof data is desired for a robust CNN model training, whichbecomes intractable when only limited samples are available.For the sea ice change detection task, the observation data ofice shelf breakup are scarce. One way to solve the problemis transferring a model pretrained on related tasks from alarge data set to sea ice change detection since it is widelyacknowledged that features in lower layers of CNN are lessspecific to the classification task.

In this letter, a multilevel fusion network (MLFN) wasestablished for sea ice change detection. We build a large data

1545-598X © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS

Fig. 1. Framework of the transferred MLFN. The LCCD data set is used totrain the MLFN, and deep knowledge is transferred to sea ice change detectiontask. Dense blocks and multilayer feature fusion are employed to extract morediscriminative features for the MLFN.

set to transfer deep knowledge from the data set to the limitedtraining data in sea ice change detection. As far as we know,it is the first time that transferred deep learning is appliedfor sea ice change detection. In the proposed MLFN, cascadedense blocks are employed to optimize the convolutionallayers. Multilayer feature fusion is introduced to exploit thecomplementary information among low-, mid-, and high-levelfeature representations. Therefore, more discriminative featureextraction can be achieved by the proposed MLFN. Experi-mental results on real SAR data sets demonstrated that theproposed MLFN has a better performance than several closelyrelated methods.

II. METHODOLOGY

A. Problem Statements and Overviewof the Proposed Method

Given two coregistered SAR images I1 and I2, which arecaptured over the same polar region at different times t1 and t2,respectively. Our goal is to highlight the changed regionsoccurring between t1 and t2. Finally, a change map will begenerated that represents the change information.

The general framework of the proposed method is illustratedin Fig. 1. A great number of image patch pairs are randomlyextracted from multitemporal images in the LCCD data set,and these image patch pairs are fed into the MLFN for training.After training, the trained weights are transferred to the inputsea ice images. A preclassification process as mentionedin [13] is employed to select reliable training samples fromthe sea ice images. Then, image patches around each trainingsample are extracted, and these patches are used to fine-tunethe MLFN. Features from all the layers of the MLFN areconsidered to be important, and all the layers are fine-tuned.After fine-tuning, all image patches from the input sea iceimages are fed into the MLFN for classification, and then thefinal change map can be obtained.

Fig. 2. Illustration of the dense block. ConvBN represents three operations:convolution, batch normalization and activation layer. The dense block hasfive layers, and each layer takes all preceding feature-maps as input.

B. Multilevel Fusion Networks

Densely connected convolution networks (DenseNets) [14]are known to be extreme cases of residual networks [15],where each convolution layer is connected by multipleshort-cut connections. DenseNets strengthen feature propaga-tion and encourage feature reuse.

Inspired by DenseNets, this letter proposes MLFN for seaice change detection. The proposed MLFN uses three denseblocks for feature extraction. The three dense blocks canextract low-level, mid-level, and high-level features, respec-tively. The first dense block can extract minor details of theimage such as lines or dots. The second block can extractmid-level features, which correspond to a combined outputof low-level features. Finally, the third block can capturethe structured information and semantic context of the inputimages.

As illustrated in Fig. 2, the dense block has four layers.The lth layer has l inputs, consisting of the feature mapsof all preceding convolutional layers. Its own feature mapsare passed to all 4 − l subsequent layers. This introduces4 × (4 + 1)/2 = 10 connections in the dense block.

Feature fusion can promote feature reuse and reduce thenumber of parameters to some extent. In vision recognitiontasks, feature fusion has been proven to be successful inmany visual recognition tasks. Zhao et al. [16] presented aperson reidentification methods which fused extracted fea-tures from different body regions with a competitive strategy.Chaib et al. [17] fused features from fully connected layersfor remote sensing scene recognition. In this letter, we intro-duced the feature fusion strategy to exploit the complementaryinformation among three dense blocks. Three dense blocksare used to extract the low-level, middle-level, and high-levelfeatures of the input image. The outputs of the three blocksare expressed as FL , FM , and FH , respectively. Outputs of thethree blocks are merged as follows:

F = pooling(h(FL) + h(FM ) + h(FH )) (1)

where F represents the fused features, and pooling is aglobal average function. h(·) is a dimension matching func-tion. Dimension matching is performed before feature fusion.64 kernels with the size of 1 × 1 are employed to convoluteFL , FM and FH . With such convolution, the number of featuresmaps of the three dense blocks all become 64. Therefore,the feature fusion can be achieved by elementwise summation.

The fused features are fed into a global pooling layer andthen processed by one fully connected layer. Finally, the fusedfeatures are transformed into one high-dimensional vector.

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

GAO et al.: TRANSFERRED DEEP LEARNING FOR SEA ICE CHANGE DETECTION FROM SAR IMAGES 3

The vector is used as the input of a softmax layer to cal-culate the possibility of being changed or unchanged. Thepossibility of being changed or unchanged can be representedas pc and pu , respectively. And if pc > pu , the input imagepatch pairs belong to the changed class, otherwise, they belongto the unchanged class.

C. Network Fine-Tuning

In this letter, a large data set named as Land CoverChange Detection (LCCD) data set was constructed. There are10 multitemporal SAR image pairs (256 × 256 pixels) togetherwith annotated ground truth change maps. There are twotypes of pixels in the ground truth change maps. Specifically,the changed class (the label is denoted as 1) and the unchangedclass (the label is denoted as 0). The images in LCCD dataset were captured by Radarsat, ENVISAT, and ERS satellites.

The goal of our work is to transfer deep knowledge fromthe LCCD data set to the limited training data in sea icechange detection. We randomly selected 30 000 image patchpairs for the changed class and 30 000 image patch pairs for theunchanged class from LCCD data set. These selected samplesare fed into the MLFN to train a model. Next, the modelis adapted to the input multitemporal sea ice images. Thereare two possible approaches for performing fine-tuning in thepretrained MLFN. The first approach is to fine-tune all thelayers. The second is to keep some of the low-level layers fixedand then fine-tune the high-level layers. In the first approach,the softmax layer is removed from the pretrained MLFN. In theother approach, the low-level layers are frozen to keep thefeatures already learned and the high-level layers are adjustedfor input multitemporal SAR images. In this letter, we fine-tuned all layers, and therefore, features from all layers areassumed to be crucial for sea ice change detection.

D. Training Samples Selection and Change Map Generation

The input multitemporal SAR images are first handled bythe log-ratio operator to generate the DI. The DI is computedas ID = | log(I2/I1)|. It is assumed that the multiplicativenoise can be transformed to additive noise by such operation.

After DI generation, a hierarchical fuzzy c-meansalgorithm [13] is performed on ID to classify pixels into threegroups: the changed class Gc, the unchanged group Gu , andthe intermediate group Gi . Pixels in Gc have high probabilityof being changed, while pixels in Gu have high probability ofbeing unchanged. Pixels in Gi will be further classified by theMLFN.

Image patches pairs centered at pixels from Gc and Gu

are extracted from the input multitemporal SAR images.Let R1

p represent the image patch centered at position p inimage I1, and R2

p represent the corresponding image patchin image I2. The size of each patch is r × r . We randomlyselect 10 000 samples from Gc and Gu . The selected samplesare used to fine-tune the pretrained MLFN. Finally, all thepixels from the Gi are further classified by the MLFN. In theclassification results, the changed class is labeled as 1, andthe unchanged class is labeled as 0. The classification results

TABLE I

IMPLEMENTATION DETAILS OF THE PROPOSED MLFN

together with Gc (labeled as 1) and Gu (labeled as 0) formthe final change map.

III. EXPERIMENTAL RESULTS AND ANALYSIS

In this section, to evaluate the performance of the proposedMLFN, experiments were conducted on two real sea ice datasets. The implementation details of the proposed MLFN arelisted in Table I. We first briefly describe the data set togetherwith the evaluation criteria. Then, the important parametersthat influence the change detection performance are discussed.Finally, comparisons with several excellent change detectionmethods are provided to demonstrate the effectiveness of theproposed MLFN.

A. Experiment Setup

In order to demonstrate the effectiveness of the proposedmethod, we apply the proposed method on two sea ice datasets. Both data sets were acquired at Sulzberger Ice shelfby the Advanced SAR (ASAR) on the ENVISAT satelliteand were provided by the European Space Agency. Theimages were acquired on March 11 and 16, 2011, respectively.The data sets show the calving of large icebergs from theSulzberger Ice Shelf as a result of the huge waves generated bythe tsunami in Japan. The original size of the two SAR imagesis 2263 × 2264 pixels. Both images are too huge to display thedetailed information. Therefore, we select two typical regions,denoted as Sulzberger I and Sulzberger II, respectively.

In this letter, the performance of the proposed MLFN willbe measured by the number of false positives (FP), falsenegatives (FN), overall error (OE), and percentage of correctclassification (PCC) and kappa coefficient (KC).

B. Analysis of Parameters

In order to obtain the contextual information, an image patchis extracted around each sample pixel. We choose a size ofr × r for each patch. The parameter r is a crucial parameterthat may affect the performance of sea ice change detection.Therefore, we set r as 5, 7, 9, 11, 13, and 15, respectively.The PCC values of different r are shown in Fig. 3.

As can be observed that, on both data sets, the PCC valuesachieve best when r = 9 or r = 11. There is a tendency

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS

Fig. 3. Relationship between different image patch size and the PCC.

TABLE II

CHANGE DETECTION RESULTS OF MLFN WITH TRANSFER

LEARNING AND MLFN WITHOUT TRANSFER LEARNING

of decline in PCC values when r > 11. The reason lies inthe fact that larger image patch may not be representativeof the central pixel, and the change detection is prone to beaffected by the speckle noise. When r ≤ 7, the PCC values arerelatively low since more contextual information needs to betaken into account. In this letter, r is set to 9 in our followingimplementations.

Next, we discuss the effectiveness of transfer learning.As shown in Table II, the MLFN combining transfer learn-ing obtains the better PCC. Specifically, there are 0.72%and 1.17% improvements in PCC by fine-tuning parameters ofall layers on two data sets, respectively. Furthermore, we canobserve that fine-tuning parameters of all layers have a betterperformance than fine-tuning parameters of the high-levellayers. The reason for this phenomenon lies in the fact that thefeatures from all layers are important for our sea ice changedetection task. Therefore, in our implementations, we fine-tuneparameters of all layers.

C. Experimental Results and Discussions

In this section, we compare our method with three closelyrelated methods, including PCAKM [3], NBRELM [12], andGaborPCANet [13]. It is worth mentioning that NBRELM,GaborTLC, and GaborPCANet are implemented by theauthors’ open source code with the default parameters. Thefinal change detection results of different methods are shown

TABLE III

CHANGE DETECTION RESULTS OF DIFFERENT METHODSON THE SULZBERGER I AND II DATA SETS

in figure form and the corresponding evaluation criteria arelisted in tabular form.

Fig. 4 shows the final change maps of different methods ontwo data sets, and Table III lists the corresponding evaluationcriteria. On the Sulzberger I data set, it can be observedthat many changed regions are missed by GaborPCANet, andtherefore, the FN value of GaborPCANet is high and the over-all performance is affected. PCAKM and NBRELM generatehigher OE values than the proposed MLFN. The proposedMLFN exploits rich feature representations from transferredknowledge from LCCD data set, and therefore achieves thebest performance. The KC value of the proposed MLFNis improved by 0.38%, 1.85%, and 0.57% over PCAKM,NBRELM, and GaborPCANet, respectively. Therefore, we canconclude that the proposed MLFN outperforms other methodson the Sulzberger I data set.

On the Sulzberger II data set, we can notice that manychanged pixels are missed by NBRELM. Therefore, NBRELMgenerates very high FN value. The FP values of PCAKM andGaborPCANet are extremely high, and it indicates that manyunchanged pixels are falsely detected as changed ones. Theproposed MLFN can draw a balance between the FP and FNvalue and achieves the best PCC and KC values. It should benoted that the proposed MLFN has powerful feature represen-tations by introducing residual learning and multilayer featurefusion. The PCC value of the proposed MLFN is improvedby 3.29%, 3.12%, and 2.59% over PCAKM, NBRELM, andGaborPCANet, respectively. As a result, we can draw theconclusion that the proposed MLFN is superior to the othermethods on the Sulzberger II data set. Visual and quantitativeanalysis on both data sets demonstrated that the proposedMLFN is effective in sea ice change detection from SARimages.

The computational complexity of the MLFN and otherclosely related methods is reported in Table IV. All experi-ments were implemented on the Intel Xeon E5-2620, NVIDIAGTX 1080 platform. The computational cost of the pro-posed MLFN is higher than PCAKM and NBRELM, due tothe fact that deep learning-based MLFN needs to optimize

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

GAO et al.: TRANSFERRED DEEP LEARNING FOR SEA ICE CHANGE DETECTION FROM SAR IMAGES 5

Fig. 4. Visualized results of different change detection methods on the Sulzberger I data set (first row) and Sulzberger II data set (second row). (a) Imagescaptured on March 11, 2011. (b) Images captured on March 16, 2011. (c) Ground truth change maps. (d) Results by PCAKM [3]. (e) Results by NBRELM [12].(f) Results by GaborPCANet [13]. (g) Results by the proposed MLFN.

TABLE IV

COMPUTING TIME OF DIFFERENT METHODSON BOTH DATA SETS (IN SECONDS)

much more parameters. However, if more powerful graphicscard is provided, the speed of the proposed MLFN can befurther improved. Moreover, compared with GaborPCANet,the proposed MLFN is superior in computational time, whichmeans that the speed of the proposed MLFN is computationalefficient than typical deep learning-based model.

IV. CONCLUSION

In this letter, we proposed a transferred MLFN for seaice change detection from SAR images. In the proposedmethod, a large data set (LCCD) is used for training. In theMLFN, cascade dense blocks and multilayer feature fusion areemployed to improve the classification performance. Further-more, the fine-tune strategy is utilized to optimize the networkparameters and better change detection results are achievedwith less computational cost. Experimental results on tworeal sea ice data sets demonstrated that the proposed MLFNoutperformed several excellent change detection methods.

REFERENCES

[1] R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam, “Image changedetection algorithms: A systematic survey,” IEEE Trans. Image Process.,vol. 14, no. 3, pp. 294–307, Mar. 2005.

[2] L. Wang, K. A. Scot, L. Xu, and D. A. Clausi, “Sea iceconcentration estimation during melt from dual-pol SAR scenesusing deep convolutional neural networks: A case study,” IEEETrans. Geosci. Remote Sens., vol. 54, no. 8, pp. 4524–4533,Aug. 2016.

[3] T. Celik, “Unsupervised change detection in satellite images usingprincipal component analysis and k-means clustering,” IEEE Geosci.Remote Sens. Lett., vol. 6, no. 4, pp. 772–776, Oct. 2009.

[4] M. Gong, Z. Zhou, and J. Ma, “Change detection in synthetic apertureradar images based on image fusion and fuzzy clustering,” IEEE Trans.Image Process., vol. 21, no. 4, pp. 2141–2151, Apr. 2012.

[5] L. Bruzzone and D. F. Prieto, “Automatic analysis of the differenceimage for unsupervised change detection,” IEEE Trans. Geosci. RemoteSens., vol. 38, no. 3, pp. 1171–1182, May 2000.

[6] B. Xiong, Q. Chen, Y. Jiang, and G. Kuang, “A threshold selectionmethod using two SAR change detection measures based on the Markovrandom field model,” IEEE Geosci. Remote Sens. Lett., vol. 9, no. 2,pp. 287–291, Mar. 2011.

[7] Q. Wang, J. Wan, and Y. Yuan, “Deep metric learning for crowdednessregression,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 10,pp. 2633–2643, Oct. 2018. doi: 10.1109/TCSVT.2017.2703920.

[8] Q. Wang, J. Gao, and Y. Yuan, “A joint convolutional neural networksand context transfer for street scenes labeling,” IEEE Trans. Intell.Transp. Syst., vol. 19, no. 5, pp. 1457–1470, May 2018.

[9] Q. Wang, J. Gao, and Y. Yuan, “Embedding structured contour and loca-tion prior in siamesed fully convolutional networks for road detection,”IEEE Trans. Intell. Transp. Syst., vol. 19, no. 1, pp. 230–241, Jan. 2018.

[10] B. Hou, Y. Wang, and Q. Liu, “Change detection based on deep featuresand low rank,” IEEE Geosci. Remote Sens. Lett., vol. 14, no. 12,pp. 2418–2422, Dec. 2017.

[11] Q. Wang, Z. Yuan, Q. Du, and X. Li, “GETNET: A general end-to-end 2-D CNN framework for hyperspectral image change detection,”IEEE Trans. Geosci. Remote Sens., vol. 57, no. 1, pp. 3–13, Jan. 2019.doi: 10.1109/TGRS.2018.2849692.

[12] F. Gao, J. Dong, B. Li, Q. Xu, and C. Xie, “Change detection fromsynthetic aperture radar images based on neighborhood-based ratio andextreme learning machine,” Proc. SPIE, vol. 10, no. 4, Dec. 2016,Art. no. 046019.

[13] F. Gao, J. Dong, B. Li, and Q. Xu, “Automatic change detectionin synthetic aperture radar images based on PCANet,” IEEE Geosci.Remote Sens. Lett., vol. 13, no. 12, pp. 1792–1796, Dec. 2016.

[14] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Denselyconnected convolutional networks,” in Proc. IEEE Int. Conf. Comput.Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 4700–4708.

[15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. IEEE Int. Conf. Comput. Vis. PatternRecognit. (CVPR), Jun. 2016, pp. 770–778.

[16] H. Zhao et al., “Spindle Net: Person re-identification with human bodyregion guided feature decomposition and fusion,” in Proc. IEEE Int.Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1077–1085.

[17] S. Chaib, H. Liu, Y. Gu, and H. Yao, “Deep feature fusion for VHRremote sensing scene classification,” IEEE Trans. Geosci. Remote Sens.,vol. 55, no. 8, pp. 4775–4784, Aug. 2017.