Detection, Localization and

Tracking of Wildfires using an

UAS

Sarthak Kukreti

Dr. Manish Kumar

Dr. Kelly Cohen

University of Cincinnati

1

Overview• Motivation

• Objective

• System Overview

• Approach

– Detection

– Localization

• Design and Results

• Conclusion

IntroductionEvery year, natural and

man-made disasters cause

loss of lives in addition to a

cost of approximately $52

billion in the form of impact

on economy and property

damage [1].

3

Motivation

• Interactions between Academia and an operational unit tasked with fighting forest fires enabling evaluation of UAVs for enhanced situational awareness.– No risk to human lives

– Range and capabilities of onboard sensors

– Real-time monitoring capabilities

– Growth in low-cost UAVs

– Easy deployment by local agencies

– Replaceable

– Potential benefits of autonomous flights

– Provide additional support to motivation

– Alleviate high costs of manned aircraft and reliance on third party for time-critical missions

4

Operational Applications• Forest/urban fire fighting

• Search and Rescue Missions

• Tornado/Flood disaster assessment, search, rescue, and recovery

missions

• HAZMAT related explosions (chemical detection sensor payload

required)

• Aerial monitoring of large public gatherings/events [2]

5

Prediction Algorithm

6

Flight Control ModuleGround Station: Control and

Data Processing Center

Video Data

GPS and INS Data Fire Propagation

Algorithm

Geographical

Information

Systems

Environmental

Conditions: Fuel,

Weather,

Topography

Overall Situational

Awareness

UAV

Data Processing Module

Real-World Scenario

Scenario #1: Search and Rescue of Lost Person

7

Scenario #2: Fire in a Forest

Detection and Localization

Localization

Main Objectives

• Developing an accurate Attitude Transformation representation Algorithm

• Defining Image depth and Target Location and Tracking Algorithm

Methodology

Filtered Attitude

and Altitude data

Pixel

Location

Fire Detection Using Genetic Fuzzy Logic

Objective

• Detect fire pixels from the visual and IR

camera feed obtained from the UAV.

Methodology

• Fuzzy logic schematic shown in the figure is used to

process the images.

• YCbCr color space is used. Fire pixel will satisfy the

following:

oY ≥C r ≥ Cb (for visual images)

oY(IR) should be high (For IR images)

• GA is used to tune the FIS in order to obtain the desired

characteristics.

Fuzzy Logic Overview

• Fuzzy refers to partial set membership

– Items are classified as belonging partially to a set

– Example: Given two sets, a set of red apples and a set of green

apples, to what set does the third apple belong?

• Third apple could belong 80% to green apples and 20% to red

apples

• Not to be confused with probability of finding third apple in the

set of red/green apples

Solution Methodology and Results

Identifying both fire and smoke pixels using colorinformation from the original (unfiltered) video.

• Fuzzy logic based image processing technique to identifyboth fire and smoke pixels simultaneously.

• For fire,– Y(x,y) > Cr(x,y) > Cb(x,y)

• When the fire has just started,– Smoke has low temperature

– Saturation as low as possible S < 0.1

• Image is analyzed in both YCbCr and HSV color space [3,4].

Solution Methodology and Results

Mamdani Type Fuzzy

Inference system [3,4]

3 Inputs

• Y-Cb

• Cr-Cb

• S

Output

• Fire

• Smoke

13

Courtesy: Anoop Sathyan

Solution Methodology and Results

Input 1: Y - Cb Input 2: Cr - Cb

Input 3: S

14

Courtesy: Anoop Sathyan

Solution Methodology and Results

Output 1 : Fire Output 2 : Smoke

15

Courtesy: Anoop Sathyan

Solution Methodology and Results

Original

Processed

16Original Image courtesy: https://plus.google.com/106083692624922066749/photos?banner=pwa

Original

Processed

Human Detection

Detection of humans from IR video

Localization Approach

Solution Methodology and Results

19

1. From INERTIAL to UAV vehicle frame

2. From UAV vehicle to UAV body

3. From UAV Body to Gimbal frame

4. Gimbal to Camera frame

R represents a (3×3) rotation matrix and dj

represents a (3 × 1) translation vector [5,6].

Matrix DescriptionTransformation from Inertial to UAV Vehicle frame

Transformation from UAV Vehicle to UAV Body frame

Transformation from UAV Body to Gimbal frame

Transformation from Gimbal to Camera frame

Synchronizing Video and Flight Data

Read the serial data coming

from APM/Pixhawk

Parse it into IMU/GPS data according to

MAVLINK protocol and

write to a text file in a given format (see next slide)

Read an image frame from camera and

save it as ‘frame xxxxx.jpg’,

where ‘xxxxx’ is the timestamp

coming from the message.

Challenges:

• Data from multiple heterogeneous sensors (such as cameras, IMU, and GPS) are

asynchronous and have often missing data.

Methodology:

• Data time-stamping achieved in programming using system clock

• Estimation in presence of missing and asynchronous data achieved via Kalman

filter that exploits UAV dynamics

Synchronizing Video and Flight Data

• Once the frames and data are saved, they can be matched according to the timestamp. The timestamp is the

timestamp on the MAVLINK message and is the system boot time (time since the APM/Pixhawk is initialized,

in milliseconds). The frames are saved on the disk and the local time is written into the EXIF data of the .jpeg

file. Hence, that can also be determined, if desired.

Localization

Localization

Localization

Based on the pixel location of the processed image the target is localized as follows:

Using the pixel location of the target in an image, with measurements of UAV position

and attitude, and camera pose angles, the target is localized in world coordinates.

The image processing algorithm will output the pixel locations helping us identify object

from the camera feed.

Using the processed pixels to find the depth of the image

Finding the target location relative to the UAV and transforming the location to the

global reference frame, so that the object can be located on a map.

Extended Kalman Filter (EKF)

The extended Kalman filter (EKF) to estimate the location, velocity and

heading of a mobile ground target. It works by linearizing the dynamics of a

system about the current best guess at the target’s state, then updating that

estimate using the linear Kalman filter equations.

To estimate positions and velocities of targets, we use a set of EKFs. We

assume that a UAV measures its pose in a world coordinate frame using a

Global Positioning System (GPS) signals and Inertial Measurement Unit

(IMU).The motion model for the target is

𝑥𝑖 = 𝑦𝑖(𝑥𝑖, 𝑢) + 𝑄

Extended Kalman Filter (EKF)

We assume that the ground mobile target moves with constant velocity but can change its

heading instantaneously and that a constant altitude flight is undertaken. The state for the

target is define

where xn, xe, vn and ve are the target’s position and velocity in the north and east

directions, respectively. L is the range between the target and the UAV

For the target x, the prediction step of the EKF in discrete time is given by

𝑥𝑖 = [𝑥𝑛, 𝑥𝑒, 𝑣𝑛, 𝑣𝑒, 𝐿]T

𝑥𝑘 = 𝑥𝑘−1 + 𝑇𝑠𝑦𝑖 𝑥𝑘−1 , 𝑢𝑘𝑃𝑘− = 𝑃𝑘−1

+ + 𝑇𝑠 𝐴𝑘−1𝑃𝑘−1+ + 𝑃𝑘−1

+ 𝐴𝑘−1𝑇 + 𝑄

Localization

The simulation of the UAV is quite simple. It is assumed that the UAV will fly about the flying

area at a constant low speed and altitude. It is also assumed that the UAV will be flying level

while over the target area and the target area has zero elevation.

• GPS error variance: [σn = 15 m; σe = 15 m; σd = 15 m] for north, east and down

directions

• UAV attitude error variance: σatt = 0.1 rad

• Target speed Vt = 5 m/s, UAV air speed Vuav = 20 m/s.

Sample Simulation Layout

Localization

In order to simulate noise in the vehicle’s GPS measurements, the fusion system only has

access to a noise-corrupted version of the vehicle’s position there are two results shown, first

when the object is stationary and second when a red mobile object/hot spot is tracked in the

line of sight of the UAV in a random obstacle field.

For the stationary target localization, over 100 runs of the simulation are done and the mean

value error for the runs was 1.5 m from the actual target location shown by a red cross.

Stationary Target Localization Time history of the Kalman Filter for stationary target

Localization

Figure below shows the actual and estimated trajectory of a ground moving target.

It can be seen that the estimated trajectory has the same behavior. The location

estimates are sensitive to the UAV attitude errors, more the errors in UAV attitude

error more uncertainty in location estimates of the target.

Trajectory Estimation of the Target

Sample Experimental Results

-79.798 -79.7975 -79.797 -79.7965 -79.79639.6334

39.6336

39.6338

39.634

39.6342

39.6344

39.6346

39.6348

39.635

39.6352

Longitude

Latit

ude

Conclusions/Future Work

• The SIERRA Team continues to work to develop and test algorithms for the enhancementof situational awareness.

• Experimentally testing out each sub-component of the algorithm in real time

• After integration doing flight test on a static/dynamic UAV platform and also developingand conducting a variety of flight test scenarios.

• Improving the reliability and accuracy of the geo-localization algorithm

• Conducting extensive field tests to validate all the algorithms after integration

• Further development with the infrared camera will allow for the autonomous detection offires and using image processing techniques for search and rescue operations.

• The team’s next goals are to combine the algorithms developed into a ground station.

31

Acknowledgments

• The SIERRA Project would like to thank the National Science

Foundation Award and the Ohio Space Grant Consortium for

their continued support.

• The SIERRA Project would also like to thank the West Virginia

Division of Forestry.32

Graduate Students

• Anoop Sathyan

• Bryan Brown

• Alireza Nemati

• Mohammad Sarim

Thank You!

References• [1] Garg, S., Balaji, R., Cohen, K., and Kumar, M., 2013. “A fuzzy logic based image processing method for automated fire and

smoke detection”. In 51st AIAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposition.

• [12] Davis, J., and Sharma, V., 2004. “Robust backgroundsubtraction for person detection in thermal imagery”. IEEE Int. Wkshp.

on Object Tracking and Classification Beyond the Visible Spectrum.

• [3] Dalal, N., and Triggs, B., 2005. “Histograms of oriented gradients for human detection”. In Computer Vision and Pattern

Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, Vol. 1, IEEE, pp. 886–893.

• [4] Wang, W., Zhang, J., and Shen, C., 2010. “Improved human detection and classification in thermal images”. In Image

• Processing (ICIP), 2010 17th IEEE International Conference on, IEEE, pp. 2313–2316.

• [5] J.A. Besada, J. M. Molina, J. Garcia, A. Berlanga and J. Portillo, “Aircraft identification integrated into an airport surface

surveillance video system,” Machine Vision and Applications, Springer-Verlag Heidelberg, Vol 15, No. 3, pp. 164-171, July 2004

• [6] J. M Molina, J. Garcia, F. J. Jimenez, J. R. Casar, “Cooperative Management of a Net of Intelligent Surveillance Agent

Sensors,” International Journal of Intelligent Systems, Vol. 18, no 3, pp. 279-307, 2003

• [7] R. Castaldo, C. Franck, A. Smith, “Evaluation of FLIR/IR Camera Technology for Airport Surface Surveillance,”

• [8] D. B. Barber, J. Redding, T. W. McLain, R. W. Beard, and C. Taylor, “Vision-based target geo-location using a fixed-wing

miniature air vehicle,” Journal of Intelligent Robotics Systems, vol. 47, pp. 361–382, December 2006.

• [9] M. J. Monda, C. A. Woolsey, and C. K. Reddy, “Ground target localization and tracking in a riverine environment from a UAV

with a gimbaled camera,” in Proceedings of AIAA Guidance, Navigation and Control Conference, pp. 6747–6750, August 2007.

Hilton Head, SC, USA.

• [10] P. DeLima, G. York, and D. Pack, “Localization of ground targets using a flying sensor network,” in Proceedings of the IEEE

international Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing, pp. 194–199, June 2006. Taichung,

Taiwan.

• [11] W. Whitacre, M. Campbell, M. Wheeler, and D. Stevenson, “Flight results from tracking ground targets using seascan UAVs

with gimballing cameras,” in Proceedings of 2007 American Control Conference, July 2007. New York, NY, USA.

• [12] I. H. Whang, V. N. Dobrokhodov, I. I. Kaminer, and K. D. Jones, “On vision-based target tracking and range estimation for

small uavs,” in Proceedings of AIAA Guidance, Navigation and Control Conference, August 2005. San Francisco, CA, USA.

• [13] V. N. Dobrokhodov, I. I. Kaminer, K. D. Jones, and R. Ghabcheloo, “Vision-based tracking and motion estimation for moving

targets using small uavs,” in Proceedings of 2006 American Control Conference, June 2006. Minneapolis, MN, USA.

• [14] R. Rysdyk, “Uav path following for constant line-of-sight,” in Proceedings of the 2nd AIAA Unmanned Unlimited Systems,

Technologies and Operations Aerospace, Land and Sea conference, September 2003. San Diego, California, USA.

Back up

35

Advancement in Unmanned

Systems• Recent advances in different

areas of cyber-physical and information systems including – Sensing

– Communication

– Computing Technologies

– Unmanned Systems

– Geographical Information Systems (GIS)

Have provided an unprecedented opportunity to revolutionize the generation of situational awareness for augmented management and control of a large-scale disaster.

36

Solution Methodology

37

Transformations

Solution Methodology

38

Transformations

Solution Methodology

39

Transformations

Solution Methodology

40

Transformations

Extended Kalman Filter (EKF)

The measurement model for the target is

ℎ𝑖 𝑥 = 𝑝𝑖 − 𝐿𝑖 TgcTb

gTvbTI

v 𝑙

Where 𝑙 is the desired direction of optical axis in the camera frame

𝐶𝑘 = 𝑃𝑘−𝐻𝑘

𝑇 𝑅 + 𝐻𝑘𝑃𝑘−𝐻𝑘

𝑇 −1

𝑃𝑘 = 𝐼 − 𝐶𝑘𝐻𝑘 𝑃𝑘−

𝑥𝑘 = 𝑥𝑘 + 𝐶𝑘 𝑧𝑘 − ℎ 𝑥𝑘 , 𝑢𝑘

It has been found that setting the diagonal elements of the covariance matrix to some

large value (on the order of 100), and the off-diagonal elements to some small value (on

the order of 10) works. Every time a new measurement comes in, the target’s state

estimate is updated using the extended Kalman filter equations.

Value Propositions

To understand what the firefighters needed in the field the SIERRA Program interviewed 103 potential users and collaborators, including high ranking fire Chiefs and industrial leaders. From these interviews two Value Propositions were discovered:• Save lives and property by enhanced real-time information and

decision making: The proposed UAV based platform provides an unprecedented opportunity to gather real-time information for fire-fighting and search and rescue operations.

• Ease of use by first responders: As opposed to other UAV platforms available in the market, this platform will provide push-button facility to operate the UAV and visualize the information.

42

43

Equipments Used:• Hexacopter equipped with

• APM Autopilot• GPS module• GoPro HD Video Camera• IR camera• Video transmitter• Telemetry• RC receiver to take over the control manually

• Laptop Computer for video feed• Ground station laptop computer to get IMU and GPS data in MATLAB.• Internet connectivity to get the Google Maps in MATLAB

Equipment Name

+3D Robotics 2014 DIY Quad kit

Zippy Flightmax 5000mAh LiPo battery

Edimax EW-7811 WiFi USB adapter

MB 1240XL Sonar

Futaba 6CH Radio

Raspberry Pi Model B

Raspberry Pi camera module

Infrared Camera

+ The 3D Robotics quad kit included 4-880 kV brushless motors, 4-20 A speed controllers, 4-10*4.7 propellers, and a PixHawk autopilot.

Goals of the Project

• The long-term goal of the SIERRA (Surveillance for Intelligent Emergency

Response Robotic Aircraft team) is to develop a unique and commerciallyviable human-robot system which would enable generation ofcomprehensive situational awareness during natural and man-made disasters.

• The objective is the generation of accurate situational awarenessby providing advantages in safety, cost, and ability to gather real-time data.

• Integrating real-time UAV sensory data into effective fire-predictor software will allow an incident commander to maketimely and informed decisions which can optimize the resourceallocation process and save lives.

44

The Endgame• Provide the operational fire-fighting unit with their own “mini-Air

Force” comprised of a squadron of 8-12 low cost UAVs

• Alleviate high costs of manned aircraft and reliance on third party for

collecting real-time intelligence in time critical missions

• Utilize an effective software package which runs mobile devices for

enhanced situational awareness, real time information sharing and

decision making

45

Objective:

• Detect humans from IR camera feed [6].

Methodology:

•Train a Convolutional Neural Network (CNN) consisting of a convolutional layer, a pooling layer

and two fully connected layers [5,6,7].

• Training is done using dataset of images of size 30 x 62 pixels each labeled 1 or 0 depending

on whether the image has a human or not.

•Using blob analysis and a Gaussian filter, the regions of interests (ROIs) where there is the

possibility of a human are identified.

•Each of the ROIs are passed through the trained CNN to check if it contains a human. If the ROI

size is smaller than the 30x62 window required, then the surrounding pixels are considered and if

it’s larger, then spline interpolation is used to resize the ROI to 30x62 pixels.

Human Detection Using Deep Learning

Convolutional layer

(12 kernels)Pooling layer

Input

image

of size

30x62

Fully connected

layer

Output label

(human or

not)

Courtesy: Anoop Sathyan

Localization

Sample Target

X position error v/s time Y position error v/s time

The localization error for test images is shown in the

figures below after being averaged by using a least

square fitting algorithm.

Localization percentage error estimates

Synchronizing Video and

Flight Data

• The IMU and GPS data are received at different frequencies.

Further, since it is a serial communication, the data is received

one message at a time.

• Hence, for a given timestamp, we just have a single message;

that is, when we are writing the IMU data, we don’t have values

of GPS coordinates, and similarly while writing GPS data, we

don’t have IMU data at that particular timestamp.

• A simple Kalman filter needs to be written to estimate the

missing data at those timestamps.

Synchronizing Video and

Flight Data

Fuzzy Logic Overview cont’d

• A FIS transforms inputs (1) to outputs (4) using If-Then rules– Membership functions define descriptive sets of inputs and outputs (i.e. “fast” or “slow” velocity)

– Crisp inputs are fuzzified (2)

– If-Then rules relate fuzzy inputs to fuzzy outputs (3)

– Fuzzy outputs are defuzzified into crisp outputs (4)

Fuzzy Logic Overview Cont’d

• FISs are universal approximators – can approximate any function

to any degree of accuracy

• FISs can be used to approximate control algorithms or create

unique control algorithms

Solution Methodology and Results

52

Original Image courtesy: https://plus.google.com/106083692624922066749/photos?banner=pwa

• Computational time requirement is approximately 10

seconds for a 640 x 480 resolution image on a 3.33GHz 64

bit processor

Original

Processed

SIERRA Dataset• 192 images taken from an IR camera at UC

▫ Used in training and validation sets for CNN

▫ 72 images containing humans, 120 containing space

▫ 62x30 pixel dimensions

▫ Grayscale

Two sample images from the SIERRA

dataset, left containing a human and

right containing empty space.

Modifications Implemented

• Lowering batch size to 8 images at once

• 168 images used for training

• 24 images used for validation

• 2 classifications instead of 10: human vs non-human▫ 3 unused dummy classes had to be created to run

• Final error of 5.89%

Graphs automatically produced by MatConvNet, updated after every training

epoch. These graphs are the objective and error graphs for training the SIERRA

set. After 40 epochs of training, the error stabilizes and ends close to 5% for both

training and validation.

Note on Classes:

0 = Empty Space

1 = Human