aiaa-2009-1980(real-time detection of fire hotspots from mini-uav based, thermal infrared vis-nir...

American Institute of Aeronautics and Astronautics

1

Real-Time Detection of Fire Hotspots from Mini-UAV Based, Thermal InfraRed / VIS-NIR Hyperspectral Image Data

F. Esposito1, G. Rufino2, and A. Moccia3

University of Naples “Federico II”, Napoli, Italy, I-80125

This paper presents activities being carried out at the Department of Aerospace Engineering of the University of Naples “Federico II”, in the field of airborne remote sensing for natural disaster monitoring and of autonomous navigation, from system design to realization and test. Results of a flight test campaign for experiments of fire recognition are discussed in detail in the paper.

I. Introduction ANY authors document the important role of Remote Sensing (RS) and related technologies for wildfire detection and management and highlight how processing and analysis of images acquired with Electro Optical

(EO) sensors installed onboard airborne or satellite platforms support prevention of fire risk, real-time mapping of fire extent and movement, and post-fire estimates of burnt area1-17. A growing interest is being directed towards airborne platforms as tactical systems, since they present the advantage of timely and updated information on fire evolution and offer the ability of continuous ground coverage and rapid accessibility to data acquired over fires4-7, 9.

Moreover, there are many teams of researchers and investigators worldwide, also from the academic community, which are concentrating their efforts to demonstrate the feasibility of using Unmanned Aerial Vehicles (UAVs) and advanced Electro Optical (EO) sensors to monitor and map wildfires9-17. The objective of these efforts is to develop tactical UAVs capable of long mission durations, equipped with technologies and methodologies to collect and distribute geo-referenced image data, which can be quickly deployed to begin operations during wildfire critical phases, as well as employed for flying a localized area for long periods, including night.

This paper presents activities being carried out at the Department of Aerospace Engineering (DIAS) of the University of Naples “Federico II” in the field of airborne remote sensing to natural disaster monitoring, in particular forest fires, and of autonomous navigation, from system design to realization and test. Concurrent projects and co operation with external research teams constitute the scenario of these activities. Main focus is on flight test activities carried out by DIAS in the framework of gathering multi-band, image data over test beds arranged to simulate fire-risk-concerned areas. The reference platform adopted for system implementation and operation is a mini-UAV prototype equipped with sensors, compact onboard CPUs for operation control, and auxiliary equipment for both airborne surveillance and autonomous flight. The results presented in the paper are relevant to a specific flight test session arranged for the experiment of real fire recognition. The aim was to gather image data over a controlled fire along with navigation data, in order to develop and test off line image processing algorithms for autonomous fire identification and algorithms for autonomous geo-referentiation of the observed scene.

The final objective of the presented study is to demonstrate the feasibility of realizing a compact UAV platform capable of autonomous operation in missions to support firefighting activities. Finally, it must be highlighted that the capability of enabling autonomous operation for both the RS payload and the navigation instrumentation has the potential to support firefighter suppression strategies and to improve fire management equipment safety.

II. Project “1st”-FIRST DIAS has been involved in UAV navigation for several years18, 19, and is presently carrying out a series of

projects in the field of airborne RS aimed at forest fire risk management20, 21. The major effort is being directed

1 PhD contract researcher, Department of Aerospace Engineering, Via Claudio 21, Napoli, AIAA Member. 2 Senior researcher, Department of Aerospace Engineering, Piazzale Tecchio 80, Napoli, AIAA Member. 3 Full professor, Department of Aerospace Engineering, Piazzale Tecchio 80, Napoli, AIAA Senior Member.

M

AIAA Infotech@Aerospace Conference <br>and <br>AIAA Unmanned...Unlimited Conference 6 - 9 April 2009, Seattle, Washington

AIAA 2009-1980

Copyright © 2009 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved.


2

towards project “1st”-FIRST (autonomous Flyer for Integration of Remote Sensing and guidance, navigation, and control Technologies aimed at monitoring environmental risk), a technology demonstrator of airborne surveillance and autonomous flight capabilities based on a mini-UAV prototype, born within the framework of a joint research activity carried out at DIAS in co-operation with the Centre of Competence for Analysis and Monitoring of Environmental Risk (AMRA). AMRA is a recently founded research facility for the development of innovative-technology application for environmental problems, also providing Small and Medium Enterprises (SMEs) and industries with innovative approaches, development of prototypes, and user friendly applications. DIAS-AMRA activities consist in the development of the Airborne Fire Imaging Spectrometer (AFIS), a cutting-edge RS system based on integrated EO sensors and focused on tactical forest fires monitoring by using compact platforms. It is based on four instruments, selected to get enhanced ability in forest fire detection and monitoring and described in detail in Ref. 20. They are:

--an ultra-compact, lightweight, thermal infrared imager (spectral response band 7.5 13µm); --a 3-band multispectral, high-resolution, camera operating in the Visible (VIS) spectrum; --two hyperspectral sensors, in the VIS-Near InfraRed (NIR) and NIR bands. All of these instruments are Commercial-Off-The-Shelf (COTS) devices or were realized by assembling COTS

components, as, in particular, the hyperspectral sensors. The goal of the DIAS-AMRA research project is to demonstrate the potentiality of the selected multi band EO

sensor for fire monitoring, from prevention to management of a fire, damage mitigation, and also post-event burnt area mapping. In its full operational capability, AFIS will have the potential to support regional wildland fire management and provide Forest Service and Civil Protection with detailed observations essential for real time firefighting, prompt selection of effective fire-suppression strategy, efficiency and safety of equipment and personnel during fire-management operations.

Project “1st” focuses on the integration of a compact UAV platform with a subset of the AFIS multi band EO sensors, the thermal-infrared imager and the VIS-NIR hyperspectral sensor, to develop a breadboard for test and validation of both RS payload and the Guidance, Navigation, and Control (GNC) onboard equipment. The goals of project “1st” are manifold:

--assessment of the potentiality of the integrated RS system for fire monitoring, fire risk mapping for

prevention, and post-fire damage determination support; --assessment of GNC techniques and procedures, including sensor fusion and autonomous flight; --integration of the RS payload, the GNC system, and the auxiliary equipment in terms of on-board, real-time

processing, data exchange, and geo referentiation of the observed scene; --development of autonomous operation routines, run by onboard CPUs and based on modular software

logic; --development of telemetry radio-links to remotely monitor and operate the payload from a Ground Station

(GS), to downlink payload and navigation data, and to uplink commands; --real-time identification of fire hotspots in the acquired geo-referenced images. Several flight test campaigns were carried out in the period 2006-2008, which demonstrated the feasibility of a

compact UAV platform capable of autonomous operations in RS missions to support wildland fire management activities.

III. Hardware Configuration The activities presented here mainly deal with the integrated VIS-NIR/infrared payload installed onboard “1st”

mini-UAV prototype. Hardware components adopted for flight test campaigns are described in the following sub-sections.

A. UAV Platform The mini-UAV platform “1st” is a scaled reproduction of the Dornier™ DO27™. The airframe, figure 1 (a), is a

lightweight, radio-controlled model. Its fuselage was customized to carry the RS payload with the EO sensors in nadir-pointing configuration for push-broom operation, the GNC system, and communication equipment, as shown in figure 1 (b).


3

Specific solutions were adopted to allow for quick installation and disinstallation of the above subsystems, in

particular the EO sensors. Installation of the latter ones required also new windows to be opened in the lower skin of the fuselage. A general reinforcement of fuselage landing gear was realized to guarantee safe operation in presence of the additional mass of the payload.

The mini-UAV”1st” is conceived to be operated in both remote-pilot and autonomous flight modes. It is able to fly payloads up to 3.5kg, providing about one-hour flight endurance at cruise speed (25m/s). Further key features of mini-UAV “1st” that have encouraged the research activities are easy of transportation to the flight test area, operability at almost any airstrip, and little maintenance required to keep it performing well. Table 1 summarizes main characteristics of the mini-UAV “1st” platform.

B. RS Payload The use of a mini-UAV configuration for this application imposed a selection of the DIAS-AMRA set of sensor

due to the limited onboard availability of mass, volume, power, and processing resources for the RS payload20. Nevertheless the selected sensor are fully adequate for project objectives, at least at a basic level. Indeed, they allow

(a)

(b)

Figure 1. (a) The mini-UAV "1st" high-winged aircraft platform. (b) The RS payload is integrated in the front section of the fuselage along with the RS radio-communication system, whereas the RS CPU is located in the rear section. The GNC unit, the telemetry radio-modem, and the batteries for power supply are integrated in the central space.


4

for hot-spot identification, equivalent radiant temperature estimation, and characterization of surface coverage and status over the observed scene. As shown in figure 2, the two sensors were integrated in the lower part of the mid-front section of the fuselage, aligned downwards for push-broom operation.

The thermal-infrared camera is a commercial product22. In brief, its spectral response lies in the far-infrared range of the spectrum, 7.5-13µm, and it is based on an advanced detector, a microbolometer in Vanadium Oxide (VOx) technology, that does not need focal plane cooling. Its detector has a resolution of 160x120 pixels and is equipped with a 11mm focal length Germanium optics. Reduced mass (about 300g including lens and I/O unit), size (35x37x50+30mm3 with lens), and low power consumption (<1.5W) make it suitable for applications of airborne RS based on very compact platforms. Installed onboard of the mini-UAV “1st”, this thermal imager offers a very cost-effective solution to survey a fire zone and to support firefighting operations. In fact, the thermal channel has the potential to provide visibility through smoke, even at night, and it allows for rapid identification of hotspots with respect to background features.

The VIS-NIR hyperspectral sensor was integrated at DIAS by coupling an imaging spectrograph23 to a standard

monochrome CCD camera equipped with a two-dimensional detector24. This produces a hyperspectral instrument

Figure 2. Downward pointing EO sensors and RS payload-dedicated radio-transceiver antenna.

Table 1. Mini-UAV”1st" characteristics. Dimensional specifications

Wingspan 2.75m Wing Area 0.46m2 Length 1.80m Airframe mass 3.5kg Payload mass 2.7kg Airfoil Semi-symmetrical

Engine Model OS BGX 35cc Swept volume 35cm3 Propeller 20x10cm

Standard controls Channels: 5 Servos: 5 Controls Throttle; Elevator; Rudder; Ailerons;

Front Wheel


5

that images a one-dimensional scene and analyzes the spectrum of the collected energy from each resolution element in a single acquisition. The imaging spectrograph captures a line image and disperses it to its spectrum in the direction perpendicular to the line image. Hence, it converts a matrix camera to a one-dimensional, spectral imaging system. The rows of each acquired frame contain images of the same line of targets but at different wavelengths. As a result, a high number of close spectral samples is acquired, producing a hyperspectral sensor. A detailed description of the VIS-NIR hyperspectral sensor is provided by authors in Ref. 25, whereas its main characteristics are summarized in tables 2 and 3.

C. Onboard Auxiliary Sub-Systems of the RS Payload The RS payload also includes a computer for system management and a radio transceiver for communications

with the GS, both based on COTS hardware and compatible with installation onboard a mini-UAV platform21. The onboard computer controls EO sensor operation as well as it stores and processes collected data, and it manages data downlink and command uplink. It is based on a pc-104 Single Board Computer (SBC) equipped with:

TABLE 2. Spectral features of the Hyperspectral sensor components and the assembled system.

Model Specim ImSpector V9 Spectral response Band (nm) 430 – 900 Spectral Resolution (nm) 2.7 Number of bands 175 Slit size (µm) 9900 x 25 Im

agin

g sp

ectro

grap

h

Focal plane image size (µm) 9900 x 4350 Camera Sony XC-ST70/CE Model Detector Sony ICX 423 AL

Spectral response Band (nm) 400 – 1000 Pixel size (µm) 11.6 x 11.2 Number of pixels 752 x 582 B

/w c

amer

a

Sensing area size (µm) 8720 x 6520 Spectral response band (nm) 430 – 900 Spectral resolution. (nm) 2.7 Number of bands 174

Size (µm / pixels)

8720 x 4350 / 752 x 390 Usable image

Total pixels 2.93x105 Spatial x spectral 752 x 195

Hyp

ersp

ectra

l se

nsor

( S

pettr

ogra

ph +

b/w

ca

mer

a )

Number of required samples Total pixels 1.47x105

TABLE 3. Main specification of the hyperspectral sensor. Detector CCD Frame rate Up to 25 Hz Electronic shutter 1/120 – 1/10000 s Focal length 4.8 mm FOV / IFOV 85° x 0.30° / 0.14° x 0.30° Output Analog: standard PAL Operating conditions temperature/humudity -5 - +45 °C / up to 95% Power consumption 2.1 W Mass (including ImSpector & lens) 590 g

Dimensions (including ImSpector & lens) 44 x 29 x 65 + 176 mm (camera body + ImSpector&lens)


6

--a 4-ch, multiplexed, analog frame grabber, to acquire sensor output; --4GByte solid state disk as mass memory; --a power conditioning unit, to supply the whole RS payload with power input regulated at various voltages. It is worth noting that the mass memory was selected to satisfy demanding operational constraints: GByte

capacity, high transfer rate in write mode to sustain the frame rate required by push-broom hyperspectral acquisitions without gaps, ability of operation with hard environment onboard a UAV mainly because of vibrations and shocks.

The SBC is connected to a bidirectional, high-data-tare, S-band, transceiver26 that provides the digital radio link with the GS that is required for the download of payload data and status information and for uplink of commands. It is worth noting that, due to the volume of data produced during operation of the sensors, only a selection of the acquired images can be downloaded in real-time and, typically the thermal images are transmitted to ground, because they can be easily interpreted and the whole flow can be sustained by the radio-link.

D. GNC sub-system of mini-UAV “1st” The GNC functionalities are of great importance because they provide autonomous flight capabilities as well as

they allow for spatial referentiation of the observed scene. The GNC system is based on four components: processing unit, navigation sensors suite, servomotors, and telemetry and communications hardware. All of them were selected among COTS devices.

The avionics and servo control system for navigation and autonomous flight is produced by Crossbow Technology™ and composed of µNAV™ Navigation and Servo Control Board and Stargate™ Single Board Computer27, 28. µNAV™ is a suite of calibrated digital sensors which provides the ability to control servos and, combined with the Stargate™ processing board, it forms a complete autopilot. The suite of sensors consists of a package that includes:

--an array of inertial sensors (three 1-axis accelerometers, three 1-axis angular rate sensors, and three

temperature sensors for biases compensation); --a 3-axis magneto-resistive magnetometer; --an Air Data System (ADS), which elaborates the measures from a static pressure sensor to compute altitude

and from a dynamic pressure sensor to compute airspeed; --a 16-channel GPS receiver module. GNC system’s characteristics and its installation onboard mini-UAV “1st” are extensively presented by authors

in Ref. 20, 21 and 29, along with a detailed description of software architecture. A radio modem operating at 433MHz frequency with data rate of 19900bps30, is used for communications between airborne and ground segments.

Finally, the RS and navigation sub-systems are connected via a 10/100Mbps Ethernet link and they exchange data through the UDP-based protocol.

E. GS In-flight operation of the RS payload and GNC sub-system are remotely monitored and controlled from the GS.

It is based on a portable, high-performance personal computer connected to both the ground transceivers of the two digital radio-links in use to manage the down-/up-link data flows of the RS payload and the GNC sub-system.

Operation at any site is possible thanks to a set of sealed lead batteries guaranteeing up to 2-hr of autonomous operation of the full GS.

A specific software has been developed to implement a virtual cockpit at the GS, along with window-based, user friendly I/O interfaces that visualize telemetry data, status, and diagnostics of onboard devices, and allow for easy input of the commands to be up-linked29 (figure 3). In addition, a moving map is displayed following the real-time position of the vehicle. In conjunction with attitude and navigation indicators of the virtual cockpit, it allows to monitor the respect of the mission profile.

Finally, a real-time visualization of the images down-linked from the RS payload is implemented in a dedicated window (figure 4).


7

Figure 3. The first-level-layout display interface of the GS software managing "mini-UAV “1st” operation. A virtual cockpit and a moving map display visualize the current flight parameters.

Figure 4. Received thermal-infrared images as visualized on the RS payload dedicated GS display window.


8

IV. Flight Test for Real Fire Detection Several flight campaigns were carried out, as an example see Ref. 21, to validate GNC sub-system, RS payload,

radio communications equipment, and their integration, and to test on-board and ground software as well as data exchange software procedures and protocols on board between the RS and GNC systems as well as between airborne and ground segments.

The results presented here are relevant to the flight arranged specifically for an experiment of real fire detection with the following goals:

--to collect aerial acquisitions of fire by means of each of the EO sensors installed onboard; --to validate data exchange between RS payload and GNC sub-system, in particular the implemented in-flight

tagging of acquired images with time and navigation reference data; --to acquire data for post-flight analysis and assessment of fire detection algorithms and procedures; --to operate remotely the payload and telemeter RS and GNC data from the UAV to the GS. The mission took place in January 2008 at the flight facility “Il Sagittario” in Cancello Arnone, Italy. This

facility, located at latitude 41°00.95' North and longitude 13°59.79' East, has a 120m×15m asphalt airstrip with an asphalt taxing area on one side. Figure 5 shows an aerial image of the runway, acquired during a preceding mission by means of the low-resolution color camera.

To carry out the experiment of real fire detection, a piloted ignition of wood was produced, creating a small fire spot with diameter of less than 1m and flames as high as about 1.5m (figure 6). The fire was ignited approximately 1m beyond the end line of the runway, about 3m from the left side.

A. Mission Profile The nominal flight profile, planned to acquire data over the simulated fire, consisted in short, leveled flight,

runway flyovers. The mini-UAV "1st" was piloted in RC mode, the RS payload was controlled from the GS where telemetry data from both the RS payload and the GNC sub-system were collected along with images from the thermal camera. During operation of the cameras, the pilot maneuvered finely on the basis real-time navigation telemetry displayed at the GS, in order to keep the actual platform status close to the nominal one.

Flight parameters and RS payload settings were selected on the basis of constraints related to payload operation, hardware performance limits, and airborne platform:

Figure 5. Aerial image of the runway acquired by the low-resolution color camera on board of 1st during a former flight mission campaign.


9

--correct pushbroom operation of the RS payload, with overlap between acquired images in sequence to

guarantee continuous coverage of the scene; --overall frame rate from the sensors within the performance sustainable by the RS computer. Laboratory

tests showed that acquisition and storage of a single channel can be sustained typically up to 18-20 fps (not a deterministic value because of the non-real-time operating system in use for the RS computer), whilst multiplexed acquisition and storage from more than one camera can be sustained at up to 12 fps, being the performance loss mainly due to the number of channel switching per second;

--flight parameters not simply within platform envelope, but adequate for fine control in RC piloting mode and good resolution of the acquired images. In particular, on the basis of the expertise gained after previous flight tests, it is advisable to keep altitude within 120m.

It is worth noting that continuous coverage and adequate resolution are of great importance for this experiment in

which the target to be observed, the above described fire spot, is quite small. Its detection could be easily missed because of gaps between the acquired images or poor resolution.

A tradeoff of the above requirements can be identified on the basis of the relation between flight parameters (forward speed, V, and altitude, H) and sensor characteristics (FOV size in along-track direction, θaz ) and settings (frame rate, FR, and image overlap, ε):

2tanH2)1(

VFRazθε−

= (1)

While solutions can be easily found for the two-dimensional imagers thanks to their wide FOV (table 4), planning acquisitions of the hyperspectral camera requires more care. In fact, the along-track side of its FOV is particularly narrow, only 0.30° in spite of the quite short focal length adopted, and, hence, a very high frame rate is needed for continuous coverage. As an example, typical flight parameters and survey settings (H=150m, V=20m/s, ε=0.2) produce FR≈32fps, which is a very high value, as expected, much larger than the sustainable rates. On the basis of this considerations, it was chosen to define the mission profile as follows:

--altitude of 110m and speed of 10m/s as flight parameters during RS payload operation; --mission consisting of three sessions: during the first and the second ones, acquisitions of only the thermal

imager and only the color camera, respectively, took place at 5fps per camera in both cases, while the hyperspectral

Figure 6. The fire spot in proximity of the airstrip used as target for the experiment.


10

camera was not operated; during the third session, there was operation of the hyperspectral sensor only, at the maximum frame rate that could be sustained;

- the payload was controlled remotely; RS and GNC data were down-linked from the UAV to the GS through the dedicated radio links.

The above solution results in the nominal acquisition parameters reported in table 4. The frame rate adopted for

the two-dimensional imagers are higher than the minimum required values, even if well within hardware limits; this was chosen to avoid missing image overlap due to high-frequency attitude fluctuation21.

B. Test Flight Operations and Collected Data Image acquisitions with the thermal imager, initiated approximately after 650s from the beginning of GNC

system operations and lasted approximately 524s. During this time interval, 2631 frames were acquired and stored in the mass memory of the RS CPU. The realized frame rate was exactly 5fps as planned. Table 5 lists the image sequences acquired by the thermal imager in the seven runway flyovers of the first flight. Fire was always clearly imaged except that for flyovers It and IIIt, when the pilot was not able to maneuver correctly the airframe while flying over the fire. Figure 7 shows a sequence of four images acquired consecutively in flyover IIt and composed as a mosaic to show the achieved scene coverage. Acquired images present about 80% of overlap, as expected. Figure 7 also shows that image translations and rotations are needed for registration as a consequence of attitude dynamics. In fact, it was not possible for the pilot to maintain a stable alignment of the airframe during the flyovers mainly because of wind gusts.

TABLE 4. Electro-optical sensor settings and survey parameters for the flight test. VIS-NIR Hyper. Thermal InfraRed Color VIS Focal length (mm) 4.8 11 6

cross track 42.28 40.70 53 FOV

(deg) along track 0.30 33.05 41

IFOV (deg) 0.14 x 0.30 0.27 x 0.27 0.14 x 0.14 Ground cell size from 110m (m) 0.24 x 0.52 0.47 x 0.47 0.24 x 0.24

Frame Rate (fps) 18 5 5 Image overlap for 10m/s forward speed

∼ 0.03 > 0.90 > 0.95

Pixels 384x390 160x128 400x300 Data Rate (Mbyte/s) 2.57 0.10 1.8

TABLE 5. Acquisition sequences with the thermal imager along straight flight passages over runway.

Runway Flyovers

Image Indexes

Acquisition time of first image (from GNC

clock)

Acquisition time of last image (from GNC

clock)

Frame Rate (fps)

It 299-315 708.539 s 711.749 s 5 IIt 1054-1069 859.539 s 862.539 s 5 IIIt 1716-1734 991.149 s 994.709 s 5 IVt 1956-1974 1038.749 s 1042.339 s 5 Vt 2045-2056 1056.539 s 1058.749 s 5 VIt 2201-2228 1087.739 s 1093.129 s 5 VIIt 2430-2452 1133.519 s 1137.919 s 5


11

Table 6 summarizes the acquisition sequences with the color VIS camera, corresponding to four runway

flyovers. The maneuvers were correctly conducted by the pilot and the fire was always detected by the sensor. The achieved typical frame rate (5fps) was in perfect accordance with the nominal value in table 4. A sequence of four images over fire flames and smoke, acquired along the runway flyover IVv and composed as a mosaic, is presented in figure 8. Correlation of attitude dynamics with image translations and rotations needed for mosaic is evident also in this case.

Figure 7. Mosaic of a 4-image sequence of the thermal infrared camera along runway flyover maneuvers. Fire flames and the surrounding hot air are evident.

TABLE 6. Acquisition sequences with the Color VIS camera along straight flight passages over runway.

Runway Flyovers Image Indexes

Acquisition time of

first image (from GNC

clock)

Acquisition time of last

image (from GNC

clock)

Frame Rate (fps)

Iv 16-34 1179.549 s 1183.149 s 5 IIv 196-219 1213.539 s 1218.139 s 5 IIIv 416-435 1255.529 s 1259.329 s 5 IVv 647-664 1301.159 s 1304.359 s 5


12

The VIS-NIR hyperspectral sensor was operated during the third flight session. Table 7 lists the image sequences

corresponding to the twelve runway flyover maneuvers executed by the pilot and identified by analyzing the acquisition log file of the camera. The analysis of the realized frame rate shows that the sensor was operated at the average value of approximately 17fps and that the nominal 25fps frame rate could not be sustained by the acquisition sub-system. Nevertheless, the realized values were enough to provide ground coverage of the observed areas, as will be shown in the following. Figure 9 plots the longitude-latitude flight profile of this mission session, based on on-board GPS measures. The runaway and, as an example, acquisition positions of flyover IIh are superimposed (bold rectangle and green stars, respectively).

Figure 8. Image mosaic for the sequence of four images from color VIS camera during runway flyover maneuvers. Fire flames and smoke can be identified within the images.

TABLE 7. Acquisition sequences with the VIS-NIR hyperspectral camera along straight flight passages over runway.

Runway Flyovers Image Indexes

Acquisition time of first image

(from GNC clock)

Acquisition time of last image (from

GNC clock)

Frame Rate (fps)

Ih 2650-2750 936.929 s 943.329 s ~16 IIh 3330-3430 980.319 s 987.129 s ~15 IIIh 4050-4150 1023.549 s 1029.749 s ~16 IVh 4750-4850 1066.539 s 1072.139 s ~18 Vh 5500-5600 1112.529 s 1118.529 s ~17 VIh 6300-6400 1160.519 s 1165.519 s ~20 VIIh 6900-7000 1198.749 s 1204.549 s ~17 VIIIh 7790-7890 1252.339 s 1258.529 s ~16 IXh 8450-8550 1294.919 s 1300.719 s ~17 Xh 9180-9280 1340.149 s 1345.749 s ~18 XIh 9870-9970 1385.939 s 1392.539 s ~15 XIIh 10650-10750 1442.329 s 1448.129 ~17


13

Visual examination of a single hyperspectral frame does not allow immediate recognition of the observed scene.

Indeed, multiple frame processing techniques (e.g. sub-band selection, frame mosaic, hypercube data organization). Synchronized navigation data are required to generate good-mosaic, two-dimensional images of the scene to take account of trajectory and attitude perturbations.

Figure 10, for example, shows the image mosaic of 101 acquisitions of runway flyover IIh generated from a single spectral sub band (line index 200, i.e. sub-band 685.6-686.8nm according to the spectral calibration function of this sensor20). Due to the small size of azimuth side of hyperspectral sensor FOV, image registration, which in figure 10 is only partial, is much more complex for the sequences of the hyperspectral cross track, strip like images and must be necessarily carried out on the basis of navigation status, mainly attitude and height over the scene. Figure 11 plots the measured platform attitude parameters corresponding to the acquisition instants of the sequence used to generate the mosaic of figure 10. A careful visual comparison of the two plots, allows to find out the correlation that exists between image translations and rotations and attitude angles, in particular roll which is plotted also aside the image mosaic.

13.994 13.995 13.996 13.997 13.998 13.999 14 14.001 14.002 14.003 14.00441.014

41.0145

41.015

41.0155

41.016

41.0165

41.017

41.0175

41.018

Longitude [deg.]

Latit

ude

[deg

.]

2D Path

II Runway FlyoverAirstripPath

Figure 9. Latitude-Longitude path profile computed based on GPS measures during flight session no. 3. Green stars represent acquisition positions of flyover IIh.

-50050978

980

982

984

986

988

990

Roll [deg.]

Tim

e [s

]

Swath width

Alo

ng-tr

ack

dim

ensi

on b

uilt

up b

y th

e m

otio

n of

the

airc

raft

Figure 10. hyperspectral image mosaic of sub-band no.200 for flyover IIh, and synchronized roll angle time-history of the platform during acquisition.


14

Also, details of the scene can be retrieved by combining several sub-bands. As an example, figure 12 shows the

“panchromatic-equivalent” and the “RGB-equivalent” combination of the same sequence as figure 10.

C. Fire Detection Test Results Thermal-infrared imagery represent a measure of the collected radiant energy in the spectral band where

emission from hot surfaces is very strong. Hence, pixel imaging a fire will exhibit high values of intensity and high contrast over the surrounding background that, of course, is much colder. The thermal infrared channel may be used for a preliminary step to separate the image pixels in a binary system of potential hotspots, or flames, and background.

At this regard, figure 13 shows an example to understand how a threshold may be exploited to produce a rough estimation of hotspots inside the observed scene. False colors are used to represent the different output levels corresponding to the different temperatures of the objects observed in the second image of the mosaic in figure 7. After adequate selection of a pixel-intensity threshold, the mentioned binary map of hot spots is obtained. Shape, size, and position of the fire, however, cannot be estimated precisely on the basis of thermal imagery due to the presence of hot smokes and heated air surrounding the fire.

978 980 982 984 986 988 990-20

0

20

40

Rol

l [de

g.]

978 980 982 984 986 988 990-60

-40

-20

0

Pitc

h [d

eg.]

978 980 982 984 986 988 990-140

-120

-100

-80

Hea

ding

[deg

.]

Time [s]

Figure 11. Platform attitude angles during runway flyover II.

Swath width

Alo

ng-tr

ack

dim

ensi

on b

uilt

up b

y th

e m

otio

n of

the

airc

raft

Swath width

Alo

ng-tr

ack

dim

ensi

on b

uilt

up b

y th

e m

otio

n of

the

airc

raft

Figure 12. Combination of hyperspectral acquisition of flyover IIh and mosaic: panchromatic- and RGB-equivalent images.


15

Unlike infrared data, color VIS images can be exploited to deduce information related to local conditions,

topography, and ground features. Unfortunately, when a fire is burning, the smoke spread around significantly reduces the visibility and often hides fire flames, thus preventing the use of color VIS images for reliable fire detection and fire feature estimation (front position, direction and speed of propagation, etc.). Nevertheless, based on the visible smoke, a rough estimation of the extent and movement of the fire could be provided.

Hyperspectral imagery, can be effectively exploited to identify a target on the basis of its spectral feature, independently of the intensity of the radiation. This aspect has been addressed in post-flight processing of the collected data of the presented field experiment. This produces an enormous amount of data, synthetically displayed for a portion of the scene as data hypercube in figure 14 (a). Different targets have been identified in the scene (fire, grass, dark/clear asphalt, white-paint strip on the runaway) and their hyperspectral image features were compared (figure 14 (b) and figure 15).

Figure 13. False-color representation of the second image of the sequence in figure 8. Temperature distribution in the imaged scene, the controlled fire, and the surrounding hot targets are clearly displayed.

Fire

Grass

White strip

Dark Asphalt

Clear Asphalt

(a) (b)

Figure 14. 190-band hypercube of the experiment test site scene based on the acquired VIS-NIR hyperspectral data (a), and restriction to selected targets (b)equivalent images.


16

For operational hyperspectral applications of fire detection, in particular real-time ones during acquisition, a

dimensionality reduction is necessary. Based on the published literature on this topic31 and on experimental data analysis, a three-dimensional space has been selected (λ1=540nm, λ2=657nm, λ3=745nm) in which satisfactory discrimination of the different targets, in particular the fire, can be achieved. Figure 16 and table 8 show that sample pixels of the same target in this space are clustered and quite well separated from other targets, which offers a viable solution for identification. In particular, fire reliable identification based on hyperspectral data match to the experimental model, could be carried out after hot-spot threshold mapping.

500 550 600 650 700 750 800 850 90020

30

40

50

60

70

80

90

100

110

Wavelength [nm]

Sen

sor o

utpu

t (B

est-f

it cu

rves

)

Grass Fire

Dark asphalt

Clear asphalt

White strip on runway pavement

Figure 15. Spectral sensor output of the sample ground targets as estimated from the collected hyperspectral data (10-order polynomial interpolation).

23

45

6

24

6

8104

6

8

10

12

14

16

λ1= 540nmλ2= 657nm

λ 3=

745n

m

Brightness at sensor

White strip on runway pavement

Clear asphalt

Dark asphalt

Fire

Grass

Figure 16. Projection into a three dimensional subspace of the wavelength dependent percent reflectance computed for some samples of ground targets.


17

V. Future Focus The results presented in the previous section highlighted that it was difficult to maintain a stable alignment of

instantaneous flight parameters during the test and in particular along the leveled flight runway flyovers, since the airframe was piloted in RC mode. A significant improvement to the presented system will be foremost in the next phase of the activities, where automatic multi mode flight control logic will be experimented, at least during RS sensor operations. Moreover, algorithms for autonomous geo referentiation of acquired images will be developed and tested along with image processing algorithms. The final goal will be to provide the system with real time, autonomous capability for fire detection and location.

VI. Conclusion Flight test campaigns carried out by DIAS in the framework of a research program in the field of forest fires

monitoring from airborne platforms, demonstrated the feasibility of employing a compact UAV in RS missions to support wildland fire management activities. The results presented in this paper are relevant to a specific flight test session arranged to gather image data over a real fire. Image acquisitions were referenced with navigation parameters and these data are being used in the post processing analysis to develop, implement, and test processing algorithms for fire detection, based on both thermal-IR imagery and VIS-NIR hyperspectral data. Future activities will deal with implementation of the described techniques for real time fire detection, and real-time geo-referentiation of the images being acquired, to realize an autonomous, mini-UAV-based remote sensing payload capable to support fire management operation.

Acknowledgments Dr. Esposito grant, for the period 2006-2007, was funded by AMRA.

References 1Li, Y.; Vodacek, A.; Kremens, R. L.; Ononye, A.; Tang, C., “A Hybrid Contextual Approach to Wildland Fire Detection

Using Multispectral Imagery,” IEEE Transactions on Geoscience and Remote Sensing, vol. 43, issue 9, pp. 2115-2126, Sep. 2005.

2A. Vodacek, R. L. Kremens, A. J. Fordham, S. C. Vangorden, D. Luisi, J. R. Schott, D. J. Latham, “Remote Optical Detection of Biomass Burning Using a Potassium Emission Signature,” International Journal of Remote Sensing, vol. 23, No. 13, pp. 2721-2726, July 2002.

3A. Barducci, D. Guzzi, P. Marcoionni, I. Pippi, “Infrared Detection of Active Fires and Burnt Areas: Theory and Observations,” Infrared Physics and Technology, vol. 43, No. 3, pp. 119-125, June 2002.

4J. Riggan, J. W. Hoffman, and J. A. Brass, “Estimating Fire Properties by Remote Sensing,” Proc. IEEE Aerospace Conference 2000, vol. 3, 2000, pp 173-179.

5N. El-Sheimy and B. Wright, “Real-time Airborne Mapping System for Forest Fire Fighting (F3) System,” PERS, vol. 70, No. 4, pp.381-383, 2004.

6D. A. Roberts and P. E. Dennison, “Hyperspectral Technologies for Wildfire Fuel Mapping,” Proc. 4th International Workshop on Remote Sensing and GIS Applications to Forest Fire Management, Ghent, Belgium, 2003, 10 pp.

TABLE 8. Data statistics on estimated percent reflectance evaluated for five different wavebands.

λ (nm): 540 657 686 745 765 Mean: 2.13 3.53 4.23 5.95 7.18 Grass: STD: 0.09 0.17 0.18 0.54 0.70 Mean: 2.74 4.53 5.47 8.53 10.00 Fire: STD: 0.17 0.39 0.51 0.42 1.08 Mean: 3.74 5.57 6.55 9.38 10.92 Dark asphalt: STD: 0.23 0.28 0.25 0.33 0.40 Mean: 5.12 7.45 8.79 12.26 13.97 Clear asphalt: STD: 0.23 0.36 0.29 0.70 0.73 Mean: 4.94 7.30 10.52 14.39 16.27 White-paint strip

on runway: STD: 0.32 0.39 0.67 0.86 0.92


18

7J.A. Brass, V. Ambrosia, J.D, Ricker jr., and A.C. Roberts, “Forest Firefighting from Real-Time Airborne Infrared Remote Sensing,” Proc. of Fifth International Airborne Remote Sensing Conference, San Francisco, 2001, pp 21.

8J. Verbesselt, S. Fleck, and P. Coppin, “Estimation of fuel moisture content towards Fire Risk Assessment: A review,” in Forest Fire Research & Wildland Fire Safety edited by Viegas, Millpress, Rotterdam (The Netherlands), 2002, 11 pp.

9V. G. Ambrosia, S. S. Wegener, D. V. Sullivan, S. W. Buechel, S. E. Dunagan, J. A. Brass, J. Stoneburner, and S. M. Schoenung, “Demonstrating UAV-Acquired Real-Time Thermal Data over Fires,” PERS, vol. 69, No. 4, pp. 391-402, April 2003.

10G. Zhou, C. Li, and P. Cheng “Unmanned Aerial Vehicle (UAV) Real-time Video Registration for Forest Fire Monitoring,” Proceedings of the International Geoscience and Remote Sensing Symposium, vol. 3, 2005, pp. 1803-1806.

11J. A. Brass, V. G. Ambrosia, R. S. Dann, R. G. Higgins, E. A. Hildum, J. McIntire, P. J. Riggan, S. M. Schoenung, R.E. Slye, D.V. Sullivan, S. Tolley, R.Vogler, and S. S. Wegener, “First Response Experiment (FiRE) Using An Uninhabited Aerial Vehicle (UAV),” Proc. of Fifth International Airborne Remote Sensing Conference, San Francisco, 2001, 9 pp.

12A. Restas, “Wildfire Management Supported by UAV Based Air Reconnaissance: Experiments and Results at the Szendro Fire Department, Hungary”, First international Symposium on Environment Identities and Mediterranean Area, Corte-Ajaccio, France, 2006, 10 pp.

13V. Ambrosia, B. Cobleigh, C. Jennison, and S. Wegener, “Recent Experiences with Operating UAS in the NAS,” Proc. AIAA Infotech@Aerospace 2007 Conference, California, USA, 2007, pp. 15.

14S. Dunagan, J. Eilers, B. Lobitz, and T. Zajkowski, “UAS Enabled Communications for Tactical Firefighting, ” Proc. AIAA Infotech@Aerospace 2007 Conference, California, USA, 2007, pp. 7.

15E. Hinkley, T. Zajkowski, V. Ambrosia, and S. Schoenung, “Small UAS Demonstration for Wildlife Surveillance,” Proc. AIAA Infotech@Aerospace 2007 Conference, California, USA, 2007, pp. 8.

16M. Tranchitella, S. Fujikawa, T. L. Ng, D. Yoel, D. Tatum, P. Roy, C. Mazel, S. Herwitz, and E. Hinkley, “Using Tactical Unmanned Aerial Systems to Monitor and Map Wildfires,” Proc. AIAA Infotech@Aerospace 2007 Conference, California, USA, 2007, pp. 7.

17P. J. Riggan, and J. W. Hoffman, “FireMapper™: A Thermal Imaging Radiometer for Wildfire Research and Operations,” Proc. of IEEE Aerospace Conference 2003, vol. 4, 2003, pp. 1863-1872.

18F. Esposito, D. Accardo, A. Moccia, U. Ciniglio, F. Corraro, and L.Garbarino, “Real-Time Simulation and Data Fusion of Navigation Sensors for Autonomous Aerial Vehicles,” in Advances and Innovations in Systems, Computing Sciences and Software Engineering, Springer Netherlands, 2007, pp. 127-136.

19D. Accardo, F. Esposito, and A. Moccia, “Low-cost Avionics for Autonomous Navigation Software/Hardware Testing,” Proc. IEEE Aerospace Conference 2004, Vol. 5, 2004, pp. 3016-3024.

20F. Esposito, G. Rufino, A. Moccia, P. Donnarumma, M. Esposito, and V. Magliulo, “An Integrated Electro Optical Payload System for Forest Fires Monitoring from Airborne Platform,” Proc. IEEE Aeroconf 2007, Big Sky, USA, pp. 13.

21F. Esposito, G. Rufino, and A. Moccia, “1st Mini UAV Integrated Hyperspectral/Thermal Electro Optical Payload for Forest Fire Risk Management,” Proc. AIAA Infotech@Aerospace 2007 Conference, California, USA, 2007, pp. 13.

22Omega Camera User Manual, Indigo system Corporation, doc. no. 412-0006-10, v. 110, 2002. 23ImSpector User Manual, Specim, v. 2.21, 2003. 24XC-ST70/70CE User’s Guide, Sony Corporation, v. 1.0, 1999. 25G. Rufino, and A. Moccia, “Integrated VIS-NIR Hyperspectral / Thermal-IR Electro-Optical Payload System for a Mini-

UAV,” AIAA 5th Aviation, Technology, Integration, and Operations Conference Arlington, Virginia, 2005, pp. 9. 26AC5124 2.4 GHz OEM Transceiver User Manual, Aerocomm inc., v. 4.4, 2003. 27MNAV100CA User’s Manual, Crossbow Technologies, Inc., doc. no.7430-0198-01 rev. A, 2005. Available:

http://www.xbow.com. 28Stargate Developer’s Guide, Crossbow Technologies, Inc., doc. no.7430-0317-13 rev. A, 2005. Available:

http://www.xbow.com. 29F. Esposito, D. Accardo, G. Rufino, and A. Moccia, “1st: a Fully Autonomous UAV Aimed at Monitoring Environmental

Risks,” Italian Journal on Composites and Nanotechnologies – Materials, Aerospace, Special Technologies, vol.2, n.2, pp. 61-69, October 2006.

30MC Light datasheet, Adcon RF Technology, Inc., doc. no. 10, 2002. Available: http://www.adcon.com. 31B. Gao, Y. J. Kaufman, and R. O. Green, “Remote Sensing of Smoke, Clouds, and Fire Using AVIRIS Data,” Proc. of the

Fifh Annual JPL Airbome Geoscience Workshop, JPL Publication 93-16, Pasadena, CA, 1993, pp. 61-64.

aiaa-2009-1980(real-time detection of fire hotspots from mini-uav based, thermal infrared vis-nir...

Documents