MACHINE VISION FOR INTELLIGENT

DRONES: AN OVERVIEW

APRIL 27TH 2016

KEVIN HEFFNER SAMUEL FOUCHER

RESEARCH ASSOCIATE, CRIM DIRECTEUR, VISION AND IMAGING TEAM PEGASUS RESEARCH & TECHNOLOGIES CRIM

2

Outline

Introduction to UAVs

Intelligent Drones

Machine Vision Technologies

Example Applications

Trends and Challenges

3

noun: drone; plural noun: drones a male bee in a colony of social bees, which does no work but can fertilize a queen.

Source: Steven Zaloga, Unmanned Aerial Vehicles: Robotic Air Warfare 1917-2007, Osprey Publishing, 2008

In 1935, U.S. Adm. William H. Standley saw a British demonstration of the Royal Navy's DH 82B Queen Bee. Commander D Fahrney was tasked to develop something similar for the US Navy. "Fahrney adopted the name ‘drone’ in homage to the Queen Bee”

DRONE TERMINOLOGY

4

TYPES OF DRONES BY SIZE

UAV ACRONYMS

Unmanned Aerial Vehicle (UAV)

Micro Aerial Vehicle (MAV)

Miniature Aerial Vehicle (MAV)

Small UAS (SUAS)

Medium Altitude Long Endurance (MALE)

High Altitude Long Endurance (HALE)

Remotely Piloted Aircraft Systems (RPAS)

5

Range (miles)

Flight d

uratio

n (h

ou

rs) M

ax O

per

atin

g A

ltit

ud

e (

feet

)

1 10 100 500 1K 10K

64

32

16

8

4

2

1

40K

20K

10K

5K

1K

Maximum Weight (lbs)

1 10 50 100 1K 2K 5K 10K 20K

UAV CLASSIFICATIONS

HOW FAR, HOW HIGH, HOW LONG, HOW HEAVY ?

6

Range (miles)

Flight d

uratio

n (h

ou

rs) M

ax O

per

atin

g A

ltit

ud

e (

feet

)

1 10 100 500 1K 10K

64

32

16

8

4

2

1

40K

20K

10K

5K

1K

Maximum Weight (lbs)

1 10 50 100 1K 2K 5K 10K 20K

UAV CLASSIFICATIONS: SOME EXAMPLES

7

The majority of private sector applications utilize drones as sensing platforms

FUNCTIONS AND MISSIONS

Site inspection (sensing)

Energy sector (sensing)

Geomatics (sensing)

Firefighting (sensing)

Search & Rescue (sensing)

Surveillance (sensing)

Agriculture (sensing)

Agriculture (intervention)

Transport (packages)

Recreational

PRIVATE SECTOR

Intelligence gathering

Weather

Weapons

Search & Rescue

Medical Evacuation

Communications

Transport

Target practice

Decoy

MILITARY

8

Canadian UAV Regulation

9

Which rules apply to me ?

http://www.tc.gc.ca/media/documents/ca-standards/Info_graphic_-_Flying_an_umanned_aircraft_-_Find_out_if_you_need_permission_from_TC.pdf

START

NO

I have read and can meet the exemption conditions

for UAVs < 2 kg

I have read and can meet the exemption conditions for UAVs from2 to 25 kg

You don’t need permission,

but you must meet the exemption

conditions

You must apply for a

Special Flight Operations Certificate

You don’t need permission, but

you do have to fly safely

You don’t need permission, but you must meet the exemption

conditions AND notify Transport Canada with:

1.Contact info 2.UAV model 3.Description of operation 4.Geographic boundaries of operation

NO

It weighs > 35 Kg

NO

I use my aircraft for WORK or RESEARCH

YES

YES NO

It weighs < 2 Kg

NO YES YES NO

YES YES

It weighs > 25 Kg

CANADIAN UAV REGULATION

10

http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html

• Fly your drone during daylight and in good weather (not in clouds or fog). • Keep your drone in sight, where you can see it with your own eyes – not only

through an on-board camera, monitor or SmartPhone. • Make sure your drone is safe for flight before take-off. Ask yourself, for

example, are the batteries fully charged? Is it too cold to fly? • Know if you need to apply for a Special Flight Operations Certificate • Respect the privacy of others – avoid flying over private property or taking

photos or videos without permission.

CANADIAN UAV REGULATION

• Closer than 9 km from any airport, heliport, or aerodrome. • Higher than 90 metres above the ground. • Closer than 150 metres from people, animals, buildings, structures, or vehicles. • In populated areas or near large groups of people, including sporting events,

concerts, festivals, and firework shows. • Near moving vehicles, highways, bridges, busy streets or anywhere you could

endanger or distract drivers. • Within restricted and controlled airspace, including near or over military bases,

prisons, and forest fires. • Anywhere you may interfere with first responders.

Do

Don’t fly

11

http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html

Currently there is no requirement for a Pilot or Operator License. Transport Canada has communicated their intent to institute new regulatory requirements that will address licensing and training of drone operators for drones up to 25 kg.

These requirements also establish how the UAV should be marked and registered.

CANADIAN UAV REGULATION

12

Intelligent Drones

13

DRONE EDUCATION

1. Make drones easier and safer to operate

2. To increase their autonomy and decrease human supervisory control requirements

3. So that they can perform complex functions

4. So they can collaborate with other systems (including drones)

Why do we want drones to become more intelligent ?

Making decisions

Reasoning

Prioritizing tasks

Detecting differences

Finding similarities

Learning from experience

Characteristics of Intelligence

14

*See Ingrand and Ghallab - http://homepages.laas.fr/felix/publis-pdf/aicom13.pdf

Cognitive Robotics – Breakdown of intelligence into functions

ROBOT INTELLIGENCE

Planning off-line prediction of feasible actions; Acting task execution; refines planned actions; Observing detects and recognizes; Monitoring comparison of predicted versus observed; Goal reasoning monitoring at mission level; Learning acquire, adapt and improve through experience;

DELIBERATION FUNCTIONS*

15

ENVIRONMENT

Task Execution

Low-level instructions Hardware Interface

Robot Platform

Sensory Functions

Motor Functions

ENVIRONMENT

Deliberation

Robot Platform

Task Execution

Low-level instructions Hardware Interface

Sensory Functions

Motor Functions

Robot Architecture Cognitive Robot Architecture

ROBOT ARCHITECTURES

16

ENVIRONMENT

Deliberation

Robot Platform

Task Execution

Low-level instructions Hardware Interface

Sensory Functions

Motor Functions

Symbolic reasoning

Sub-symbolic processing

Adjust speed to 40 knots. Change heading to 75 deg. …

Follow route A. Return to base. …

Robot Architecture Cognitive Robot Architecture

COGNITIVE ROBOT ARCHITECTURES

17

ENVIRONMENT

Deliberation

Robot Platform

Task Execution

Low-level instructions Hardware Interface

Symbolic reasoning

Sub-symbolic processing

Adjust speed to 40 knots. Change heading to 75 deg. …

Follow route A. Return to base. …

Robot Architecture Cognitive Robot Architecture

COGNITIVE ROBOT ABSTRACT MACHINE

CRAM Kernel

ROS Interface

Computable Predicates

Perception Navigation Manipulation Code OWL

LISP

SWI Prolog

CPL - CRAM Plans Belief State - Object designators - Entity designators - KnowRob knowledge base

KnowRob Reasoner

CPL Extension Modules

KnowRob Extension Modules

18

INTELLIGENT DRONE APPLICATION DEVELOPMENT

19

Machine Vision Technologies

20

COMPUTER VISION

Object or place recognition Visual odometry 3D reconstruction Target tracking Anomaly detection

Many applications

OpenCV PCL ArrayFire …

Open-source libraries

The main goal of Computer Vision is to enable a computer to analyze, process and understand one or more images taken by a vision system.

Development of MV Applications has exploded recently thanks to the advent of high-res digital cameras along with increased computing power & machine learning.

21

PLATFORM Ubiquity of low-cost prosumer

drones

SENSORS Smarter, Smaller,

Integrated Processors

PERFECT STORM OF TECHNOLOGIES

Size Weight and Power Price

(low) SWaPP

COMPUTING RESOURCES

Embedded, cloud Machine Vision

Libraries

22

ON-BOARD COMPUTER VISION

The combination of computer vision and IMU data (altitude, acceleration, etc.) can improve the precision and robustness of drone navigation.

Vision-based solutions are interesting for small drones (< 2.0 kg) for indoor navigation, obstacle avoidance and other problems.

Navigation Path Planning Localization Geo-fencing

Environment GPS-denied Obstacle avoidance Swarming

Automatic Landing and takeoff Toward increasing autonomy

Mission critical aspects

Environment mapping Object detection Anomaly detection Video enhancement:

De-hazing Image stabilization Super-resolution Contrast enhancement

Data compression

Application related processing

23

SIMULTANEOUS LOCALIZATION AND MAPPING (SLAM)

Example of an application for an on-board vision system

Precise drone localization Environment mapping around the drone Sensing with a sonar, a laser or a camera Object and place recognition

Problems to solve

Motion Initial

position prediction

Image Keypoint

Extraction Matching

Revised Prediction

24

KEYPOINTS DETECTION

Robustness Appearance (illumination, occlusion, etc.) Viewpoint

Local and distinct description Quantity Low computation cost

Ideal properties

Keypoints are sailient points in an image

Harris SURF SIFT ORBE FAST

Many different techniques

Detection

Description

Matching

25

VISUAL SLAM

Camera + SLAM = monocular visual SLAM (vSLAM) exploits visual information (keypoints) RGB-D or stereo cameras directly minimize the photometric error

Matching between successive images Mapping

26

VISUAL SLAM FOR GPS-DENIED ENVIRONMENTS

https://www.youtube.com/watch?v=VWWvjSHZCNo

High precision intertial navigation units are costly and heavy

Laser range finders (costly and heavy)

Cameras (visual odometry)

Possible solutions

Precision: +/- 2.5m Not accurate in urban environment Not possible to use indoors

GPS navigation limitations

27

Machine Vision Applications

for

Intelligent Drones

28

A LIST OF DRONE APPLICATIONS

Analysis of first 3136 exemption requests authorized by the FAA for small drones (<25 Kg)

29

NATURAL RESOURCE MANAGEMENT

RESEARCH APPLICATIONS

Source: Shahbazi, M., Théau, J. et P. Ménard (2014) Recent applications of unmanned aerial imagery in natural resource management. GIScience & Remote Sensing 51

Based on review of 150 articles about the use of UAV imagery

30

Improve water use Pest management Weed-identification Irrigation and water distribution Manned aircraft: ~$1,000/hour Satellite Imagery: $$$ Multispectral sensors:

• Visible (R,G,B) • Near-Infrared (NIR) • Thermal Infrared (TIR) • Resolution < 1 cm, > 12 Mpixels

Hyperspectral: 640 bands LIDAR

AGRICULTURE APPLICATIONS

31

PRECISION AGRICULTURE

Source: http://www.pole-pegase.com/documents/Actualites/6_-_presentation_Exametrics.pdf Source : http://www.icv.fr/conseil-viticulture-oenologie/oenoview

Re

solu

tio

n (

m)

10 1 0.1 0.01 0.001

$$$

$$

$

32

AGRICULTURE APPLICATIONS

Information gathering NDVI Multispectral Hyperspectral

Intervention Irrigation control Spraying Harvesting

AGRIBOTICS APPICATIONS

33

GLACIER MASS BALANCE APPLICATION

Source: Christophe Kinnard, Atelier sur la sécurité civile, AQT’2015

Orthomosaic (10 cm) + MNE

~US$10,600

34

ENERGY SECTOR INSPECTION & SITE SECURITY

Crack detection in dams and reactors

A security drone that will chase down intruders.

35

COFFEE BREAK DETECTION APPLICATION

36

HOTSPOT DETECTION USING OPTIMAL SEARCH PATTERN

Pegasus Research and Technologies Research Project

Rapid deployment Low-cost lightweight IR sensors Optimized search path Applications for

For missing persons Remnant forest fire detection Security applications

HOTSPOT DETECTION

37

OPEN-FIELD HOTSPOT DETECTION TRIAL

FOR LOCALIZED SEARCH AND RESCUE

1. Determine presence and approximate location of candidate “missing person”.

2. High resolution camera is then pointed at location and fed to SAR team.

3. System queries operator for next step 4. Future capability for automatic object

recognition.

HOTSPOT DETECTION

38

SEMI-AUTOMATED ROOF INSPECTION APPLICATION

Pegasus Research and Technologies Research Project

39

ROOF INSPECTION APPLICATION

1. Automated data collection a. 3D Model construction images b. High-resolution texture images

2. 3D Model construction 3. Roof extraction and zone detection 4. High-resolution texture generation 5. Inspection annotations 6. Report generation

ROOF INSPECTION PIPELINE

40

(define-policy start-execution()

"MISSION STARTED ?"

(:init (format t "Initializing policy to start the execution ~%") t)

(:check (setq execution_param (roslisp:get-param "/execute_mission"))

(print execution_param)

(cond ((eql execution_param 1))))

(:recover (format t "Running recovery mechanisms ~%"))

(:clean-up (format t "Running clean up ~%")))

(def-top-level-cram-function execute_misson ()

"MISSION EXECUTION"

(roslisp:start-ros-node "execute_mission")

(mav:init-ros-mav "hummingbird")

(roslisp:set-param "/execute_mission" 0)

(roslisp:set-param "/pause_execution" 0)

(roslisp:set-param "/roof_detected" 0)

(setf foo 1)

(setf counter 1)

(with-failure-handling

((policy-check-condition-met (f)

(print "starting execution") (return)))

(with-named-policy 'start-execution ()

(loop while (= counter 1)

do(print "waiting for execution")

(setq counter 1)

(sleep 3))))

(with-failure-handling

((policy-check-condition-met (f)

(print "test condition met")

(return)))

(with-named-policy 'my-policy ()

(defparameter *transform-listener* (make-instance 'cl-tf:transform-listener))

(sleep 5)

(setq trans (cl-tf:lookup-transform *transform-listener* :source-frame "/hummingbird/ground_truth" :target-frame "/world"))

AUTOMATED IMAGE COLLECTION USING DELIBERATION

41

SAMPLE IMAGES

42

3D MODEL RECONSTRUCTION

43

ROOF SEPARATION

44

ROOF ZONE DETECTION AND MEASUREMENT

45

HIGH RESOLUTION IMAGES FOR DETAILED INSPECTION

46

1

2

ANNOTATIONS AND REPORT GENERATION

47

Trends and Challenges

48

Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016

TRENDS AND CHALLENGES

49

TRENDS AND CHALLENGES

Source: Graves Presentation: NASA Aeronautics Strategic Thrust: Assured Autonomy for Aviation Transformation, Vision & Roadmap, March 9th 2016

50

?

Drone

? ?

?

TRENDS AND CHALLENGES

Airworthiness considerations

Platform health monitoring & failure prediction

Fault detection & failure recovery

Intelligent operator aids

Operator training

Navigation warnings and notifications

Obstacle avoidance

Night-time operations

Drone Safety

Rogue drones

Drone pirates

Drone data protection (Cybersecurity)

Privacy

Drone Security

51

Personal Drones

TRENDS AND CHALLENGES: DRONE HYPE CYCLE ?

Agrib

otics

Cin

em

a

Ge

om

atics

Pu

blic Safety

Ene

rgy Secto

r

Site In

spe

ction

52

http://www.tc.gc.ca/eng/civilaviation/standards/standards-4179.html

Advances in remote sensing, embedded processing and cloud computing technologies have created opportunities for new applications. The availability of low-cost prosumer drones adds another dimension of opportunity. Currently most drones are flying sensors. Adding intelligence to drones can improve safety and allow for drones to participate in data gathering and processing as part of an information ecosystem, e.g. Internet of Things (IOT). Regulation and safety aspects will be main factors in the adoption and practical use of drones for commercial purposes.

CONCLUSIONS

WWW.CRIM.CA

Suivez-nous

Dialoguez avec nous

Suivez-nous

#CRIM_ca

wwwCRIMca

Tous droits réservés © 2016 CRIM. 405, avenue Ogilvy, bureau 101, Montréal (Québec) H3N 1M3/514 840-1234/1 877 840-2746

Kevin Heffner Research Associate, CRIM President, Pegasus Research & Technologies K.Heffner@peretec.com

Samuel Foucher Équipe Vision et Imagerie CRIM – Centre de recherche informatique de Montréal Samuel.Foucher@crim.ca

Principal partenaire financier Le CRIM est un centre de recherche appliquée en TI qui développe, en mode collaboratif avec ses clients et partenaires, des technologies innovatrices et du savoir-faire de pointe, et les transfère aux entreprises et aux organismes québécois afin de les rendre plus productifs et plus compétitifs localement et mondialement. Le CRIM dispose de quatre équipes de recherche en TI de calibre mondial, d’un centre de tests et d’interopérabilité considéré comme une référence neutre au Québec ainsi qu’un centre de formations de pointe en TI. Le CRIM œuvre principalement dans les domaines des interactions et interfaces personne-système, de l’analytique avancée et des architectures et technologies avancées de développement et tests. Détenteur d’une certification ISO 9001:2008, son action s’inscrit dans les politiques et stratégies pilotées par le ministère de l'Économie, de l'Innovation et des Exportations (MEIE), son principal partenaire financier.