a five-camera vision system for uav visual attitude

A five-camera vision system for UAV

visual attitude calculation and collision

warning

Akos Zarandy*†, Zoltan Nagy*†,

Balint Vanek*, Tamas Zsedrovits*,

Andras Kiss†*, Mate Nemeth*

email: [email protected]

*Institute for Computer Science and Control, 13-17 Kende

Street, Budapest, H-1111, Hungary

†Pazmany Peter Catholic University, 50 Prater Street,

ICVS 2013, St. Petersburg, Russia on July 16-18, 2013

Outline

• Goals

• System requirements

• Vision system architecture

• Many core processor array implementation

• Algorithms

Goals

• Unmanned Aerial Vehicle (UAV) technology is maturing

– UAVs are ready for autonomous missions technically

– Autonomous missions are not authorized

• Missing redundancy

• Missing collision avoidance device

– Automatic collision avoidance device is needed even for

remotely piloted UAVs

• Remote aircraft detector and ego-motion calculator

sensor development is introduced in this presentation

Collision avoidance devices

• Radar based

– Applied on large airliners (Airbus)

• Radar and vision

– Applied on large remotely piloted UAVs (predator)

• Transponder based

– TCAS, ADS-B (all manned aircrafts and larger UAVs)

• Vision only

– Currently developed (small UAVs)

• Basic requirements

– Equivalent safety

– Probabilities of collision < 10-11 per flight hour

– Layered approach

Requirements

• Detect a 10m aircraft from 2km

• 0.1 degree/pixel resolution

• min. 220x60⁰ view angle

• Flyable size/weight/power figures

• On-board data storage

intruder

collision

volume separation

minima

collision

avoidance

threshold

traffic avoidance

threshold

our

track

Architecture

• High resolution camera

system in the visual range

– Elongated, (220x60⁰)

– Large view angle

– min 2Mpixel

• FPGA processing system

– High computational power

– Low power consumption

• Solid state disk

– Bandwidth

– Capacity

– Vibration

Solid State Drive

FPGA board

Camera system

to/from on-board control

220⁰

Camera selection • Single camera with wide angle optics

– Easy from architectural, algorithmic, and processing side

– Low distortion, ultra wide view angle optics are bulky

• 3 pieces of C-mount cameras

– Good image quality

– Relatively large size, volume, and power (1kg, 10W)

– High speed serial I/O (USB, Gige, fire-wire) difficult to connect to

embedded systems

• 5 pieces of miniature cameras (M12 lens)

– Max 1.2 megapixel with global shutter

– 50g, 200mW

– Poorer image quality

• Micro cameras

– Very advantages size/weight/power figures

– Rolling shutter only

5 camera vision system

• 5 pieces of wVGA micro cameras – Aptina MBSV034 sensor

– 5g

– <150mW

– 3.8mm megapixel objectives (M-12)

– 70 degrees between two cameras

– Total view angle: 220˚x78˚

• FPGA board with Spartan 6 FPGA

• Solid State Drive (128Gbyte)

8

Solid State Drive

FPGA board

Interface card

to/from on-board control

WVGA panel cameras

Mechanics

• Stable camera holder

– Alignment

– Avoids cross vibration

of the cameras

– 100g

– Aluminum alloy

– Electronics in the

middle

– Covered with

aluminum plates

9

Hardware system

• Sensing and processing system

– Field of view: 220˚x78˚

– Resolution: ~2250x752

– Frame-rate: 56 FPS

– Processor: Spartan6 L45 FPGA

– Storage: 128Gbyte (23 min)

– Size: 125x145x45mm (5”x6”x1,8”)

– Weight: ~450g (1lb)

– Power consumption: <8W

Vision system mounted to the airplane

View angle

Panoramic view (stitched images)

Perspective view

Many-core processor arrays implemented in

FPGA

• FPGA chips have the largest computational capability

nowadays

• In affordable medium sized FPGAs:

– Over 200 DSP cores

– 200 memory blocks

– 500 IO pins

– Low power consumption

– Special purpose processor arrays

• How to utilize this performance?

– Many-core architectures

– Specially optimized data and control paths

– Distributed control units

Image processing system • Input:

– DVI/HDMI

• 1920x1080@50Hz

– Native camera interface

• Three image processing

accelerators

• Parallel operation

• Optimized for the image

processing algorithm

Memory

controllerDRAM

Microblaze

processor

Image capture

Full frame

preprocessing

Gray scale

processor

Binary

processor

Full frame image preprocessor

• Adaptive threshold

– 3x3, 5x5, 7x7

neighborhood

– Programmable

threshold level

• Centroid computation

– 32x32 centroid

– Final 128x128 centroid

computed by

Microblaze

DVI Interface

Adaptive

threshold

Centroid

computation

Input Video

1920x1080@50Hz

FIFO

Microblaze FSL

R

FIFOG

FIFOB

FIFOBW

CE

CE

Memory

Interface

Memory

clock domainDVI clock

domain

Gray scale processor

• On-chip memories – 128x128 pixel images

– 8 RAMB18

– Separate port for fast Microblaze access

• Separate clock domain

• Specialized Processing Elements (PE)

• Supported operations: – Edge Average, Diffusion,

Threshold, Pixel wise arithmetic operations

– Global white, black, change

• Expected performance: – 150MHz, 27.3us/op,

1460op/frame

Mem_0

128x128

8bit

Mem_1

128x128

8bit

Mem_N

128x128

8bit

PE_0 PE_1 PE_M

Input_1 Buffer Input_2 Buffer

Port A Port A Port A

Port B Port B Port B

A DI DO A DI DO A DI DO

I1_sel I2_sel

O_sel

Microblaze

clock

domain

Image proc.

clock

domain

Binary processor

• On-chip memories – 128x128 pixel images

– 1 RAMB18



• One line of Specialized Processing Elements (PE)

• Supported operations: – Erosion, Dilation, Recall,

Pixel wise logic operations

– Global white, black, change

• Expected performance: – 150MHz, 0.85us/op,

46,800op/frame

Mem_0

128x128

1bit

Mem_1

128x128

1bit

Mem_N

128x128

1bit

Input_1 Buffer Input_2 Buffer

Port A Port A Port A

Port B Port B Port B

A DI DO A DI DO A DI DO

I1_sel I2_sel

Microblaze

clock

domain

Image proc.

clock

domain

128bit data

path

PE_0 PE_1 PE_M

O_sel

128x1 processor array

Vector processing unit for Kalman filter

• On-chip memories – Scratchpad memory


– 64bit wide 64 element vector registers

• Configurable vector length


• 64bit double precision floating point units

• Supported operations: – Independent multiplication

and addition

– Multiply and add (MADD)

64x64bit

Vect_0

*

+

D

A DI

DO

ReadAddrWriteAddr

64x64bit

Vect_1

A DI

DO

64x64bit

Vect_n

A DI

DO

Scratchpad

memory

DI DO

A DI DO

Microblaze clock

domain

Algorithmic components • Aircraft detection against sky background

– Preprocessing on the full frame

• Identifying candidate points

– Post processing

• Discarding non-relevant candidate points

– Tracking

– Multi-level global and local adaptivity

• Aircraft detection against terrain background

– Visual-inertial data fusion

– Motion based moving object detection

Preprocessing (full frame)

• Identifies the

candidate aerial

objects

• Finds numerous

false targets also

• Local adaptation

based on edge

density

• Global

adaptation

based on

number of

candidate points

contrast

calculation

locally adaptive

contrast

thresholding

candidate

points

regions of

interest

(ROIs)

threshold

adjustment

Too

many or

too few

points? y

n

Post processing (ROIs)

• Discard edges

of clouds

• Significantly

reduces the

number of

candidate

points

• Resulting few

targets can be

tracked

cutting the

perimeter of

each object

histogram

calculation

accept

candidate

point

Variance

high?

reject

candidate

point

y

n

Results: distant airplane

Results: large aircraft

Navigation aid • UAV was equipped with

INS and cameras

• Attitude was calculated

from the INS data (blue)

• Attitude was calculated

from 5 point (red) and 8

point (magenta) algorithms

• Visual-navigation data

fusion development is

going on

615 620 625 630 635-200

-150

-100

-50

0

50

100

150

200Absolute yaw angles

Time [s]

[

deg]

615 620 625 630 635-100

-80

-60

-40

-20

0

20

40

60

80Absolute roll angles

Time [s]

[

deg]

615 620 625 630 635-30

-20

-10

0

10

20

30

40

50

60Absolute pitch angles

Time [s]

[

deg]

Conclusion

• Vision system is under development

• Currently,

– We are replacing the FPGA board

– Finalizing the implementation of the image processing

subsystem

– Fusion of visual and navigation algorithms

– Replacing the wVGA cameras with 1.2 megapixel ones

• Expecting the system completion by the end of the year

Thank you for your attention!

FPGA, many-core processing

and UAV group

Ákos Zarándy:

Péter Szolgay

András Radványi

Zoltán Nagy

András Kiss

Pencz Borbála

Németh Máté

a five-camera vision system for uav visual attitude

Documents