peach flower monitoring using aerial multispectral imaging

Article

Peach Flower Monitoring Using Aerial Multispectral Imaging Ryan Horton 1, Esteban Cano 1, Duke Bulanon 1,* and Esmaeil Fallahi 2

1 Department of Physics and Engineering, Northwest Nazarene University, Nampa, ID 83686, USA; [email protected] (R.H.); [email protected] (E.C.)

2 Parma Research and Extension Center, University of Idaho, Parma, ID 83660, USA; [email protected] * Correspondence: [email protected]; Tel.: 1-208-467-8047

Abstract: One of the tools for optimal crop production is regular monitoring and assessment of crops. During the growing season of fruit trees, the bloom period has increased photosynthetic rates that correlate with the fruiting process. This paper presents the development of an image processing algorithm to detect peach blossoms on trees. Images of an experimental peach orchard were acquired from the Parma Research and Extension Center of the University of Idaho using an off-the-shelf unmanned aerial system (UAS), equipped with a multispectral camera (Near-infrared, Green, Blue). The orchard has different stone fruit varieties and different plant training system. Individual tree images (high-resolution) and arrays of trees images (low-resolution) were acquired to evaluate the detection capability. The image processing algorithm was based on different vegetation indices. Initial results showed that the image processing algorithm could detect peach blossoms and demonstrate good potential as a monitoring tool for orchard management.

Keywords: blossoms; digital image processing; machine vision; peaches; unmanned aerial system

1. Introduction

Although Idaho is popularly known for potatoes, the state grows other specialty crops which includes peaches. Peaches are grown in the southwestern part of Idaho, which is warmer as compared to other regions. The state produces about 5300 tons of peaches [1]. In addition to peaches, Idaho agriculture produces apples, pears, cherries, apricots, nectarines, plums and grapes. The specialty crop industry in Idaho is thriving. However, the industry is currently facing the challenges of labor shortage, increasing labor cost, and the pressure of a growing market. Because of these challenges, fruit growers are going to need to adopt new technologies that can aid in optimizing crop production.

One of these new technologies is precision agriculture. Precision agriculture is an agricultural management concept based on measuring and responding to the variability in the field [2]. Crop variability has both temporal and spatial components that need to be considered. The spatial component is facilitated by the use of the global positioning system (GPS), which enables the farmer to locate the precise location in the field. In combination with advanced sensors that could measure moisture levels, nitrogen levels, organic matter content, etc., it allows the creation of maps that show the spatial variability of the field.

Although precision agriculture has been used mostly for row crops such as corn and wheat, studies have shown that the technology has been adopted for specialty crops which include fruit trees [3]. One of the precision agriculture technologies that has been reported is remote sensing. Remote sensing can be implemented using satellite or aerial system [4]. The downsides of using satellites are the cost and the frequency of data collection, which could affect the temporal aspect of crop production [5]. On the other hand, aerial systems can be classified as manned or unmanned. Similar to satellites, manned aerial system is costly which may not be economically feasible for smaller fields. However, with the proliferation of cheap commercial unmanned aerial system such as

Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 7 November 2016 doi:10.20944/preprints201611.0034.v1

Peer-reviewed version available at J. Imaging 2017, 3, 2; doi:10.3390/jimaging3010002

http://dx.doi.org/10.20944/preprints201611.0034.v1

http://dx.doi.org/10.3390/jimaging3010002

2 of 9

the 3DR Iris and DJI Phantom series (Figure 1), remote sensing using unmanned aerial system can be very promising for fruit growers with small acreage.

Figure 1. Off-the-shelf unmanned aerial systems.

One of the examples of the use of unmanned aerial system (UAS) for fruit trees is the crop

monitoring and assessment platform (C-MAP) developed at Northwest Nazarene University [6]. The C-MAP is composed of an off-the-shelf UAS equipped with a multispectral camera. Figure 2 shows one of the C-MAP UAS flying over an experimental apple orchard with different watering methods, drip and sprinkler. The false color image clearly shows the variability of the field caused by the difference in water input [7].

Figure 2. Monitoring of apple orchard using C-MAP.

In this paper, the application of CMAP is extended to the detection of blossoms of peaches. It

has been reported that there is an increase of photosynthetic activity during the bloom period, which





3 of 9

correlates with the fruiting process. In addition, farmers scout the orchard during the blooming season and use the observed amount of bloom with other parameters to predict yield. Early prediction of yield help growers in marketing their products and the packing operations. The objectives of this study are: (1) to expand the use of CMAP to detect peach blossoms, and (2) to develop an image processing algorithm to detect peach blossoms.

2. Materials and Methods

2.1. Target Field

The target fields in this study are an experimental peach orchard located north of Parma Idaho at the University of Idaho Research and Extension Center and a peach orchard located north of Marsing Idaho owned by Symms Fruit Ranch. Both orchards are located in the western part of the state of Idaho. The Parma orchard contains a variety of peach types, whereas the Symms orchard contains one type of peach tree which is grown for produce. The Parma orchard was approximately two acres and although the orchard at Symms was much larger, approximately only two acres were observed for the study.

2.2. Image Acquisition System

Two UASs were used in this study, both of which were DJI Phantom Quadcopters [8]. A DJI Phantom 3 Professional quadcopter was used to capture peach images in the RGB color spectrum and a DJI Phantom 3 Advanced was used to capture peach images in the NIR region with the use of a modified camera. Both DJI Phantom quadcopters utilized a navigation controller which could control the drone either manually or autonomously if interfaced with a tablet. A tablet with the DJI Go application software was used to connect and interface with the controller in order to calibrate the DJI drones and to allow for GPS and waypoint following during flights. The captured image files were written on two SD cards inside the drones and then a computer with MATLAB software was used to access and download the image files from the cards in order to perform image processing and analysis.

2.3. DroneDeploy

The software used on the tablet to collect the images was DroneDeploy [9]. DroneDeploy is a cloud-based software compatible with DJI Phantom 3 drones which uses Google Maps and GPS to construct a flight plan. Figure 3 shows the operation of the UAS using DroneDeploy. Once a drone is calibrated with the DJI Go app, a flight plan can be created in the DroneDeploy app at any given place as long as the device has Wi-Fi or a flight be loaded without Wi-Fi if the flights were pre-synced to the device beforehand. Using the touch display of the tablet, DroneDeploy allows for the user to tap and drag the boundaries of the flight zone overlaid over the desired region shown on Google maps. The figure shows how DroneDeploy works with the DJI Phantom.





4 of 9

Figure 3. Operation of UAS using DroneDeploy.

Figure 4 shows a screen shot of DroneDeploy, where the enclosed region is the desired field.

DroneDeploy then plans the flight and calculates the position where to take the images to capture images in order to obtain pictures that cover the whole field, which is shown as blue dots on the figure. DroneDeploy also shows the coverage area, the number of images that will be taken, file size, and flight time. Once a flight plan is set, the DroneDeploy application allows for the user to adjust the altitude and the number of pictures the drone will take during the flight with Frontlap and Sidelap selections. Once the flight is initiated, DroneDeploy will autonomously fly the drone along the given path and capture images at the given way points. Though the drone flies autonomously, the drone can be immediately switched back into manual flight by flipping the fight state switch on the controller. Once the images are taken, a computer accessing the DroneDeploy website can be used to upload the images and create an orthomosaic picture of the captured images.

Figure 4. Tablet screenshot of DroneDeploy.





5 of 9

2.4. Image Acquisition The images collected for this study were taken between the dates of March 8, 2016 and April 20,

2016. The pictures were taken roughly every week apart from the two weeks of bloom in which images were taken multiple days in a week. All data collection flights were dependent upon weather and solar conditions due to the impact they might have on the flight ability of the drones. All of the flights were completed between the times of 8am and 2pm and were normally conducted with little to no wind. Although the conditions were clear skies, about half the images obtained were taken in cloudy weather. Sample images taken from the color camera and the modified camera are shown in Figure 5.

Figure 5. Sample images acquired at peach orchards.

2.5. Image Processing and Analysis

The image processing acquired images were processed and analyzed using Matlab and the Image Processing Toolbox. The focus of this paper is the image processing of images from the modified camera (Near-infrared, Green, Blue). The image processing involved the separation of the three bands and analyzing the color distribution. A contrast stretching operation was made on each band to improve the color difference between the blossom and the background [10].

3. Results

3.1 Image Analysis

Figure 5b shows a sample image acquired from the experimental field using the modified camera. With the modified filter of the camera (Figure 5b), objects with high chlorophyll will have high reflectance in the near-infrared and green, but low in blue. In the image, the weeds have red brown hue because of the high chlorophyll content as compared with the other objects in the image. The colors of the peach blossoms are composed of white and light pink hue. Some of the blossoms have similar hue with the branch and some part of the ground.

The color properties of the blossoms and the weeds were plotted to show their distribution. Figure 6 shows the color distribution of the blossoms and the weeds. As observed in the raw image, the light color of the blossom explains the high amount of near-infrared, green, blue values as compared with the weeds. Although we could easily draw a line and separate the blossom and weed cluster, there is still some overlap between them. Figure 6 shows the color distribution when the contrasts of each band were stretched. The near-infrared versus the blue band shows the separation between the two clusters. By using a very rudimentary thresholding process, the blossom could easily be segmented.





6 of 9

Figure 6. Color distribution of blossoms and weeds.

3.2 Peach Bossom Detection

Figure 7 shows the image processing algorithm to detect the peach blossom. The first step is to stretch the three bands individually and then combine them. A simple thresholding operation for the near-infrared and blue bands is used for the segmentation of the blossom from the background. This segmentation process detects the blossom from the image. Figure 8 shows the image processing results. The stretched image shows that the peach blossoms are enhanced from the background as compared with the original image. Because of this enhancement, it is easy for the thresholding operation to detect the peach blossom. The overlaid image demonstrates the high success rate of detecting the blossom from the image.

Figure 7. Image processing algorithm for blossom detection.

Original Image

Contrast stretching of 3 bands

Contrast stretched image

Thresholding

Filtering

Blossom overlay on original image





7 of 9

Figure 8. Image processing for blossom detection.

With the blossom detection algorithm result shown in Figure 8c, a binary image was obtained

containing only the peach blossoms. With this binary image, the density could be generally approximated by doing a series of calculations. Knowing the approximate height above the blossoms that the pictures were taken, the drone was then flown over a known 2-meter by 2-meter square PVC pipe at that height. Processing this image as shown in Figure 9, the number of square meters per pixel was found for that given height, which could then be applied to the binary peach blossom detection images, yielding an approximate density. Although this result would not be perfect, as long as the height of the images was consistent across all images, a correlation could be attempted.

Figure 9. Image processing process for pixel density calculation.

In order to begin a rough tree detection algorithm to obtain a blossom density per tree ratio,

MATLAB was used to create a basic grid over a peach image to separate an aerial image into boxes around each tree. Figure 10 shows the result of this grid as well as the resulting peach segmentation over the image.





8 of 9

Figure 10. Image processing results for tree grid separation.

Although such a process of tree segmentation could not be done for every image and would be

very inaccurate, future work of this study would involve the detection of individual trees by way of boundaries. Using the boundaries, the blossom density of each tree is then directly and accurately calculated. A blossom density map can then be produced, which could be used to aid yield estimation and other subsequent orchard management operation. The farmer could also use the blossom density map to provide a temporal analysis of the orchard blossoms.

4. Conclusions

An image processing algorithm was developed to detect blossoms on peach trees. The image acquisition system used an on-the-shelf UAS, DJI Phantom 3. The UAS camera was modified to allow near-infrared, green, and blue bands. Images from experimental and commercial peach orchards were used as target fields. The DroneDeploy software was used to plan the flight path, collect the images, and image mosaicking. The image processing analysis showed that contrast stretching of the images’ three bands enhanced the color of the blossoms from the background. A very basic thresholding segmentation method was used to segment the blossoms. Initial results showed that the blossoms can be detected using the thresholding operation. Future study will involve the improvement of blossom density calculation and the development of an algorithm for exact tree segmentation.

Acknowledgments: This research was supported by the Idaho State Department of Agriculture (Idaho Specialty Crop Block Grant 2014) and Northwest Nazarene University.

Author Contributions: Duke M. Bulanon conceived the study, created the literature review, designed the experiments, processed the collected data and wrote the paper. Ryan Horton and Esteban Cano acquired the images, developed the image processing algorithms, and processed the data. Esmaeil Fallahi designed the target orchard and its treatment, and supervised during the data collection.

Conflicts of Interest: The authors declare no conflict of interest.





9 of 9

Abbreviations

The following abbreviations are used in this manuscript:

UAS: Unmanned aerial system GPS: Global positioning system C-MAP: Crop monitoring and assessment platform NGB: Near infrared, Green, Blue NIR: Near infrared RGB: Red, Green, Blue UAV: Unmanned aerial vehicle

References

1. USDA-National Agriculture Statistics Service (2015). 2015 State Agriculture Overview (Idaho). Retrieved from: https://www.nass.usda.gov/Quick_Stats/Ag_Overview/stateOverview.php?state= IDAHO.

2. Downey, D., R. Ehsani, K. Giles, S. Haneklaus, D. Karimi, K. Panten, F. Pierce, E. Schnug, D.C. Slaughter, S. Upadhyaya, D. Wulfsohn. (2010). Advanced Engineering Systems for Specialty Crops: A Review of Precision Agriculture for Water, Chemical, and Nutrient Application, and Yield Monitoring: Landbauforschung – vTI Agriculture and Forestry Research, Special issue no. 340: 1-88.

3. Lee, W.S., Alchanatis, V., Yang, C., Hirafuji, M., Moshou, D., Li, C. (2010). Sensing technologies for precision specialty crop production: Computers and Electronics in Agriculture, 74(1), 2-33.

4. Pajares, G. (2015). Overview and current status of remote sensing applications based on Unmanned Aerial Vehicles (UAVs): Photogramm. Eng. Remote Sens. 81, 281-329.

5. Johnson, A.K.L., Kinsey-Henderson, A.E. (1997). Satellite-based remote sensing for monitoring Baath land use in sugar industry: Proc. Aust. Soc. Sugar Cane Technolo. 19, 237-245.

6. Bulanon, DM, Horton, M., Salvador, P., Fallahi, E. (2014). Apple Orchard Monitoring Using Aerial Multispectral Imaging. ASABE Paper No. 1913165. St. Joseph, MI: ASABE.

7. Bulanon, DM, Lonai, J., Skovgard, H., Fallahi, E. (2016). Evaluation of Different Irrigation Methods for an Apple Orchard Using an Aerial Imaging System: ISPRS Int. J. Geo-Inf, 2016, 5(6), 79; doi:10.3390/ijgi5060079

8. DJI (2016). Retrieved from: http://www.dji.com/products/phantom 9. DroneDeploy (2016). Retrieved from: https://www.dronedeploy.com/ 10. Gonzalez, R.C., & Woods, R.E. (2007). Digital Image Processing (3rd ed). New York City, New York: Pearson

© 2016 by the authors; licensee Preprints, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).





peach flower monitoring using aerial multispectral imaging

Documents