Efficient FPGA implementation of cloud detection for real-time remote sensing image processing

Na Shan, Xi-shuang Wang, Zhen-song Wang

Institute of Computing Technology, Chinese Academy of Sciences Graduate University of Chinese Academy of Sciences

Beijing, China Email: {shanna, wxs, zswang}@ict.ac.cn

Abstract—an optimized architecture is proposed for real-time automatic cloud detection system with Spectrum and Texture Analysis Combination (STAC) approach. The STAC approach is hard to be implemented in hardware, for the calculation of fractal dimension (FD) and angle second moments (ASM) of the approach consumes large computation time and hardware resources. By optimizing the calculation process of the FD and ASM, the proposed architecture efficiently improves the processing speed and dramatically reduces the logic resources and memories consumption compared to the automatic cloud detection architecture in standard STAC approach, with a 3 times running speed improvement and a more than 75% reduction of the hard resource consumption.

I. INTRODUCTION Cloud cover is a severe problem in optical remote sensing,

for its negative impact on the imaging capacity of the satellite observation system. Thus cloud detection plays an important role in remote sensing applications because it can effectively discard the “useless” images and then improve the imaging capacity. Furthermore, on board real-time cloud detection processing makes cloud detection more significant in satellite observation systems, for it can remove the invalid part of the image in the first place and efficiently obtain better utilization of expensive satellite resources [1] [2]. For this reason, real time cloud detection processing has become a hot research topic in the field of remote sensing.

So far, many real time cloud detection schemes have been proposed. Reference [3] [4] proposed a real time scheme that is based on spectrum analysis approach [5] [6]. Although this scheme is simple and easy to be implemented, it cannot provide high detection precision for single-spectrum images. In [7], a different real time method based on the Spectrum and Texture Analysis Combination(STAC) approach is presented, and it can gain high detection precision in both single-spectrum and multi-spectrum images. However, the scheme in [7] has a complex architecture which needs large logic and memory resources. This makes it difficult to implement the method of [7] in satellites.

In this paper, we aim our focus on the optimization of real time architecture for STAC approach. The analysis of STAC approach is presented in Section 2, which details profiles and analysis of the approach. Two optimized methods are proposed according to the analysis results. The proposed architecture exploiting these optimized ideas is described in Section 3. Compared with the architecture mentioned in [7], our architecture largely reduces consumption of the logic and memory resources, while provides a higher running speed. Experimental results and comparison results are given in Section 4. A short conclusion is given in Section 5.

II. SPECTRUM AND TEXTURE ANALYSIS COMBINATION APPROACH AND ANALYSIS

The STAC approach [8] adopted in this paper is a combination of two different analytical techniques: the spectrum threshold comparison and the texture analysis. Fig. 1 shows the block diagram of the STAC.

Sub-image

Spectrum threshold comparison

Probably is cloud

May be cloud, may be object Object

Calculate D value

Cloud Undetermined Object

CloudCloud Object Object

Calculate ASM

Cloud_percent>80% Cloud_percent<20%

1.9534�D�2.4500

0.9150�ASM�1.0000

1.9534�D�2.4500

0.9150�ASM�1.0000

D<1.953 or D>2.4500

ASM<0.9150 or ASM>1.0000

ASM<0.9150 or ASM>1.0000

D<1.9534 or D>2.4500

Cloud_percent: cloud threshold percentD fractal dimensionASM: angle second moments

Calculate D value

Undetermined

Calculate ASM

20%�cloud_percent�80%

Figure 1. The block diagram of STAC

By combining the spectrum threshold comparison and the texture analysis method, the STAC can provide high

performance and accuracy, especially for single-spectrum images. However, although the spectrum threshold comparison is simple for efficient real-time hardware implementation, the texture analysis approach comprised of the fractal dimension (FD) and angle second moments (ASM) calculation is a computationally intensive task and needs massive memory space; this makes it difficult to implement FD and ASM calculation in hardware. In this section, we will analyze the calculation process of the FD and ASM and propose the optimized methods below.

Because the texture analysis procedure is based on sub-graphs of 64*64 pixels rectangular blocks by convention, we will discuss the FD and ASM calculation on 64*64 pixels elementary blocks in the following sections.

A. Calculation of Fractal Dimension Fractal dimension is a parameter widely adopted to

describe the texture feature in texture analysis [9] [10]. The box-counting dimension is one of the most comprehensively used processes to obtain the fractal dimension parameter. Reference [8] presented an advanced box-counting dimension method which has simplified the equation and has reduced the elementary unit traversals; nevertheless it still requires large amount of memory resources and frequent memory access if the straightforward method is used.

Since the goal of the box-counting dimension process is to find out the minimum box number that covers the entire gray values in the elementary unit under different scale factors, the main operation of box-counting dimension process is to identify the maximum and minimum gray values in the sub-dimension of the elementary unit. Illuminated by this, we propose a method that abstracts a primary operation from the box-counting dimension process, which makes a big profit for implementing it in hardware.

Figure 2. The regular phases of the procedure

As illustrated in Fig. 2, because the scale factors (4 and 16) have specific inclusion relation in dimension, we partition the identifying procedure into four regular phases, in which phase 1 and 3 find the maxima/minima of every four pixels in row

direction, while phase 2 and 4 do the same operation in column direction.

The demarcation that changes the processing flow into regular phases abstracts a primary operation which is much easier to be implemented in hardware to find the maxima/minima of every four pixels,. Furthermore, because the data are processed only once and will never be used again, these four phases can overlap closely and only a few necessary data need to be stored. These technical measures bring three advantages: 1) improving computation speed due to the overlapping of the four phases; 2) saving the memory resources and 3) reducing memory accesses due to less memory space needed to store the data.

B. Calculation of Angle Second Moments ASM is another important parameter of the texture feature

in texture analysis [11] [12] [13]. Before calculating the value of ASM, a gray level co-occurrence matrix should be firstly formed, which is the most important part of the ASM calculation process. Due to the definition of the matrix generating procedure, all the elements of the matrix are updated according to the pixel-pair values of the image. For this reason, the matrix generation process is hard to be implemented in hardware, because large amount of memories are required to store the whole matrix and frequent memory access is needed to accomplish the update, besides a large number of clock cycles spent for the process, which is intolerable in actual applications. All these make it difficult to design efficient hardware architecture for calculating ASM, especially for real-time purpose.

By analyzing the definition of the matrix generation process, we propose a method that effectively facilitates the hardware architecture of ASM. The key idea is that we find the gray level co-occurrence matrix is a symmetrical matrix in size of 32*32, and we use this property to simplify the hardware design of the ASM.

As shown in Fig. 3, the symmetrical property indicates that the figures above the diagonal line are exactly the same as the figures below the diagonal line in the symmetrical location. Because of this, only the elements on and below the diagonal line need to be stored and updated, yet the information of the matrix keeps intact. By this way, almost 50% of the memories resource is saved and 50% of the updating operations are truncated.

32

32

Figure 3. The structure of gray level co-occurrence matrix

III. FPGA IMPLEMENTATION An efficient architecture for automatic cloud detection

processing unit is described in this section. The key ideas are based on the optimized operations described above.

A. System Fig. 4 illustrates the architecture of the optimized

automatic cloud detection processing unit based on STAC approach. Cal_threshold accomplishes the spectrum analysis and the result is expressed in cloud threshold percent (CTP); Cal_FD and Cal_ASM calculate the FD and ASM value; while the Cloud judge module makes the determination whether a sub-graph is cloud or not in terms of the three values. The two efficient methods for calculating FD and ASM parameters are applied to optimize the architecture of the Cal_FD and Cal_ASM sub-units, which will be detailed below.

Figure 4. Architecture of the optimized Automatic Cloud Detection

Processing Unit

B. Optimized architecture for Cal_FD As discussed in Section 2.A, a primary operation

abstracted from the four regular phases suitable for hardware implementation is proposed. Fig. 5 illustrates the hardware architecture of the primary operation. And Fig. 6 demonstrates how the four phases overlap in time, which means the running speed of the FD calculating process can be highly improved.

Figure 5. Architecture of the primary operation to find the maxima

Figure 6. Time-space graph of four phases overlapping

In Fig. 7, a memory arrangement is displayed to show how the memories are managed during the computation. Due to the single-use characteristic of the data, memories block A, B and C is reusable during the operation. So only a few memory resources are required to store the temporary results.

Figure 7. Memory arrangement of Cal_FD

C. Optimized architecture for Cal_ASM As analyzed in section2.B, the gray level co-occurrence

matrix generating process is the most important part of the ASM calculating process. Two methods are proposed to realize the real-time process of the gray level co-occurrence matrix generating in hardware.

One method is to apply the symmetrical matrix compact storage scheme in hardware to solve the problem of memory arrangement and access. As the calculating operation for the figures below the diagonal line is different from that on the diagonal line, a specific memory arrangement in which all the figures on the diagonal line are stored in succession followed by the figures below the diagonal line is adopted so that the memory access will be much easier. The address mapping is defined in (1). Furthermore, a lookup table is referred to speed up the address mapping operation.

,

* ( 1)32 ,2

* ( 1)32 ,2

i i ji iaddr j i j

j j i i j

�� =� +�= + + >��

+� + + <��

(1)

In order to further speed up the matrix generating process, a parallel processing method is introduced. Multi calculating units are set to concurrently generate the matrix. To avoid memory access conflict, 4 memory copies are set to store the result of each calculating unit respectively. The final matrix was formed by merging the results in the 4 memory copies into 1 result. The reason why 4 memory copies are set is a compromising result between the memory consumption and the processing speed.

IV. RESULTS Verilog HDL implementations are accomplished on STAC

with the proposed architecture. Until now we have not found any implementation result in previous literature for the real-time cloud detection implemented on FPGAs is a relatively new field of study, so we use the result in [7] as a comparison. The synthesis reports provided by Xilinx ISE show that [7] can only run at 57 MHz, while the proposed architecture runs at 178MHz, a more than three-fold increase in clock rate. This makes the proposed architecture gain a peak computing throughput of 5.7Gb/s. Fig. 8 displays the resources utilization of [7] and the proposed architecture on XC4VFX140 FPGA after synthesizing. It is clear that the resource consumption reduced dramatically, with more than 75% of reduction in all cases.

05

10152025303540

Res

ourc

e ut

iliza

tion

(per

cent

age)

S lices Slices FlipFlops

4 input LUTs bounded IO Bs RAMB 16s

[7]poroposed

Figure 8. Resource utilization on the XC4VFX140 FPGA after synthesizing

V. CONCLUSIONS In this paper, an optimized architecture is presented for

automatic cloud detection system with STAC approach. The optimizations are focused on the FD and ASM calculation process, by which the hardware architecture may gain a three-time speedup while cuts more than 75% hardware resources compared with the architecture in standard STAC approach described in [7].

REFERENCES [1] T. Algra. “On the effectiveness of cloud cover avoidance methods in

support of the Supper-spectral Mission for Land Applications,” Geoscience and Remote Sensing Symposium, 2002. IGARSS '02. 2002 IEEE International Volume 2, 24-28 June 2002 Page(s):982 - 985 vol.2

[2] T. Algra. “Real-time cloud sensing for efficiency improvement of optical high-resolution satellite remote sensing,” Geoscience and Remote Sensing Symposium, 2003. IGARSS '03. Proceedings. 2003 IEEE International Volume 7, 21-25 July 2003 Page(s):4311 - 4313 vol.7

[3] J. A.Williams,A. S. Dawood and S. J. Visser. “FPGA-based cloud detection for real-time onboard remote sensing,” Proceedings of IEEE International Conference on Field-Programmable Technology, 16-18 Dec. 2002, pp.110-116.

[4] E. A. Esam, T. Mohamed, E. G. Tarek, L. M. Jacqueline. “An efficient implementation of on-board cloud detection on a reconfigurable computer,” Geoscience and Remote Sensing Symposium, 2005. IGARSS '05. Proceedings. 2005 IEEE International. vol.3, pp.2066- 2069

[5] R. R. Irish. “Landsat 7 automatic cloud cover assessment,” http://landsathandbook.gsfc.nasa.gov/handbook/pdfs/ACCA_SPIE_paper.pdf

[6] G. C. Luis, C. V. Gustavo, A. Julia, et al. “Cloud detection for MERIS multispectral images,” Proceedings of the MERIS (A)ATSR Workshop 2005 (ESA SP-597). 26 - 30 September 2005 ESRIN, Frascati, Italy. Editor: H. Lacoste. Published on CDROM., p.53.1

[7] N. Shan, T. Y. Zheng, Z. S. Wang. “Onboard Real-Time Cloud Detection Using Reconfigurable FPGAs for Remote Sensing," Proceedings of IEEE International Conference on Geoinformatics, 12-14 Aug. 2009, pp. 1-5

[8] N. Shan, T. Y. Zheng, Z. S. Wang. “A High-speed and high-accuracy algorithm of cloud detection and its application," Journal of Remote Sensing. Volume 13, November 2009, pp.1147-1155

[9] H. Takayasu. “Fractals in the physical sciences,” Manchester and New York: Manchester University Press, 1990: 1-58, 99-116.

[10] B. H. Ding, W. C. Li, F. M. Wang. 1999. “Analysis of fractal image and design of fractal dimension calculation program,” Journal of University of Science Technology Beijing, 21(3) : 304-307.

[11] R. M. Haralick. “Statistical and structural approaches to texture,” Proceedings of the IEEE,67, pp. 786-804, 1979.

[12] J. Feng, C. B. Zhou, W. Y. Yu, H. Y. Xu, Y. Dong. “Implementation of surface roughness with image texture analysis,” Optics & Optelecnic Technologh, Vol.5, No.2, pp.39-41, April. 2007

[13] B. C. Li. “Texture feature retrieval and applications using gray co-occurrence matrices,” Journal of Kashgar Teachers College, Vo1.27, No.6, pp.35-37, November. 2006