Pázmány Péter Catholic University, Faculty of Information Technology and Bionics

UAV Visual SAA on Many CoreProcessors

Tamás Zsedrovits

zsedrovits.tamas@itk.ppke.huusers.itk.ppke.hu/~zseta

Outline

• Goals

• Sense and avoid problem

• System requirements

• Closed loop visual SAA system

• Vision system architecture

• Algorithms

• Many core processor array implementation

Goals

• Unmanned Aerial Vehicle (UAV) technology is maturing– UAVs are ready for autonomous missions technically

• Surveillance, firefighting, agriculture, etc.

– Autonomous missions are not authorized

• Missing redundancy

• Missing collision avoidance device

– Automatic collision avoidance device is needed even for remotely piloted UAVs

• Remote aircraft detector sensor development is introduced in this presentation

„When Will We Have Unmanned Commercial Airliners?”

IEEE Spectrum MagazineDecember 2011

Sense and avoid –key issue

Collision avoidance devices

• Radar based

– Applied on large airliners (Airbus)

• Radar and vision

– Applied on large remotely piloted UAVs (predator)

• Transponder based

– TCAS, ADS-B (all manned aircrafts and larger UAVs)

• Vision only

– Currently developed (small UAVs)

• Basic requirements

– Equivalent safety

– Probabilities of collision < 10-11 per flight hour

– Layered approach

Requirements

• Detect a 10m aircraft from 2km

• 0.1 degree/pixel resolution

• min. 220x60⁰ view angle

• Flyable size/weight/power figures (@25Hz, <1W, <500g)

• On-board data storage

intruder

collision

volume separation

minima

collision

avoidance

threshold

traffic avoidance

threshold

our

track

Closed loop SAA system

HIL simulator

Estimation

Estimation

algorithms

(FPGA)

Image

Rendering

and

Processing

FlightGear

(PC)

Image

processing

(PC)

y,z,a (USB)[X,V,b]+P

(USB)

True and Estimated

Trajectories,

Estimation error

Aircraft

cockpit A/C

Control

Ctrl. Defl.

(Serial link)

Flight control

(MPC 555)

Aircraft

Model (PC-

Simulink)

Ctrl. Defl

(PWM)

[X,V,b]+P

(Ethernet)

True+Est.

A/C info packet

(Ethernet)

Architecture

• High resolution camera system in the visual range

– Elongated, (220x60⁰)

– Large view angle

– min 2Mpixel

• FPGA processing system

– High computational power

– Low power consumption

• Solid state disk

– Bandwidth

– Capacity

– Vibration

Solid State Drive

FPGA board

Camera system

to/from on-board control

220⁰

Camera selection

• Single camera with wide angle optics– Easy from architectural, algorithmic, and processing side

– Low distortion, ultra wide view angle optics are bulky

• 3 pieces of C-mount cameras– Good image quality

– Relatively large size, volume, and power (1kg, 10W)

– High speed serial I/O (USB, Gige, fire-wire) difficult to connect to embedded systems

• 5 pieces of miniature cameras (M12 lens)– Max 1.2 megapixel with global shutter

– 50g, 200mW

– Poorer image quality

• Micro cameras– Very advantages size/weight/power figures

– Rolling shutter only

Sensor and computational system

• 5 pieces of wVGAmicro cameras – Aptina MBSV034

sensor

– 5g

– <150mW

– 3.8mm megapixel objectives (M-12)

– 70 degrees between two cameras

– Total view angle: 220˚x78˚

• FPGA board with Spartan 6 FPGA

• Solid State Drive (128Gbyte)

Solid State Drive

FPGA board

752x480 cameras

to/from navigation computer

Mechanics

• Stable camera holder

– Alignment

– Avoids cross vibration of the cameras

– 100g

– Aluminum alloy

– Electronics in the middle

– Covered with aluminum plates

11

Hardware system

• Sensing and processing system

– Field of view: 220°x78°

– Resolution: ~2250x752

– Frame-rate: 56 FPS

– Processor: Spartan6 L45 FPGA

– Storage: 128Gbyte (23 min)

– Size: 125x145x45mm (5”x6”x1,8”)

– Weight: ~450g (1lb)

– Power consumption: <8W

Vision system mounted to the airplane

View angle

Panoramic view (stitched images)

Perspective view

Algorithmic components

• Aircraft detection against sky background

– Preprocessing on the full frame

• Identifying candidate points

– Post processing

• Discarding non-relevant candidate points

– Tracking

– Multi-level global and local adaptivity

• Aircraft detection against terrain background

– Visual-inertial data fusion

– Motion based moving object detection

First picture?

Adaptive threshold ROI

Yes

ROI calculation and cut

Adaptive

threshold darker

OR

Recall

Output data

CutNo

Input images

Invert

Adaptive threshold

brighter

Closing

Detection against the blue sky

Y=1020 Z=255 Wingspan=42

Preprocessing (full frame)

• Identifies the candidate aerial objects

• Finds numerous false targets also

• Local adaptation based on edge density

• Global adaptation based on number of candidate points

contrast calculation

locally adaptive contrast

thresholding

candidate points

regions of interest

(ROIs)

threshold adjustment

Too many or too few

points? y

n

Preprocessing (full frame)

• Identifies the candidate aerial objects

• Finds numerous false targets also

• Local adaptation based on edge density

• Global adaptation based on number of candidate points

contrast calculation

locally adaptive contrast

thresholding

candidate points

regions of interest

(ROIs)

threshold adjustment

Too many or too few

points? y

n

Post processing (ROIs)

• Discard edges of clouds

• Significantly reduces the number of candidate points

• Resulting few targets can be tracked

n

cutting the perimeter of each object

histogram calculation

accept candidate point

Variance high?

reject candidate point

y

Post processing (ROIs)

• Discard edges of clouds

• Significantly reduces the number of candidate points

• Resulting few targets can be tracked

n

cutting the perimeter of each object

histogram calculation

accept candidate point

Variance high?

reject candidate point

y

Post processing (ROIs)

• Discard edges of clouds

• Significantly reduces the number of candidate points

• Resulting few targets can be tracked

n

cutting the perimeter of each object

histogram calculation

accept candidate point

Variance high?

reject candidate point

y

Post processing (ROIs)

• Discard edges of clouds

• Significantly reduces the number of candidate points

• Resulting few targets can be tracked

n

cutting the perimeter of each object

histogram calculation

accept candidate point

Variance high?

reject candidate point

y

Tracking on image plane

• Candidate objects:

– Separate objects

– Track objects

• Filtering according to:

– Velocity, acceleration

• InstantVisionTM 4.1 Multitarget Tracking Library (Eutecus Inc.)

Distance calculation

Measurements

Gating

State Estimation

Data Association

Target positions

and attributes

Example 1: Ground camera in hand

Pre- and post processing

Red points: all candidate objectsGreen point: allowed by post processing

Pre- and post processing + tracking

Red points: all candidate objectsGreen points: allowed by post processingBlue points: tracked objects

Example : Airborne camera

Virtual Cellular Machines

• Many core processors

• Virtual memory concept

• Hide the hardware details

– Build a more general atchitecture (FPGA, GPU…)

– Make easier for the programmer

– The hardware layer can be changed

Algorithm VCMManyCore

SAA algorithm VCM synthesisBGR

Algorithmic description Virtual machine Physical machine

2048x1024@25FPS

Noise reduction

Adaptive thresholds

Spatial-logic combination

ROI selection

Cloud analysis Areal feature

point analysis

Tracking

Streaming (pipe-line)

architecture

RISC

Coarse grain

architecture

Streaming (pipe-line)

architecture

2048x1024 cellular

processor array at

3700 Array_Op/sec,

150MPixel/sec

RISC

64x64 cellular

processor array at

65000 Array_Op/sec

RISC

• Topographic, vector and scalaroperations:

– @25 fps ~3700 pixel level operations

– Transformation to streaming processors

– ASIC or FPGA implementation

– Cellular processor array – small loadlarge area

• Scalar dominant operations

– ROI selection, tracking

– Implementation on RISC cores

• Topographic operations with smallframe size

– Only 64x64 pixels/window

– @25 fps ~65000 operations

– Coarse grain implementation (Xenon)

Many-core processor arrays implemented in FPGA

• FPGA chips have the largest computational capability nowadays

• In affordable medium sized FPGAs:

– Over 200 DSP cores

– 200 memory blocks

– 500 IO pins

– Low power consumption

– Special purpose processor arrays

• How to utilize this performance?

– Many-core architectures

– Specially optimized data and control paths

– Distributed control units

Image processing system

Memory

controllerDRAM

Microblaze

processor

Image capture

Full frame

preprocessing

Gray scale

processor

Binary

processor

• Input:

– DVI/HDMI

• 1920x1080@50Hz

– Native camera interface

• Three image processing accelerators

• Parallel operation

• Optimized for the image processing algorithm

Full frame image preprocessor

DVI Interface

Adaptive

threshold

Centroid

computation

Input Video

1920x1080@50Hz

FIFO

Microblaze FSL

R

FIFOG

FIFOB

FIFOBW

CE

CE

Memory

Interface

Memory

clock domainDVI clock

domain

• Adaptive threshold

– 3x3, 5x5, 7x7 neighborhood

– Programmable threshold level

• Centroid computation

– 32x32 centroid

– Final 128x128 centroid computed by Microblaze

Gray scale processor

Mem_0

128x128

8bit

Mem_1

128x128

8bit

Mem_N

128x128

8bit

PE_0 PE_1 PE_M

Input_1 Buffer Input_2 Buffer

Port A Port A Port A

Port B Port B Port B

A DI DO A DI DO A DI DO

I1_sel I2_sel

O_sel

Microblaze

clock

domain

Image proc.

clock

domain

• On-chip memories– 128x128 pixel images– 8 RAMB18– Separate port for fast

Microblaze access

• Separate clock domain• Specialized Processing

Elements (PE)• Supported operations:

– Edge Average, Diffusion, Threshold, Pixel wise arithmetic operations

– Global white, black, change

• Expected performance:– 150MHz, 27.3us/op,

1460op/frame

Binary processor

Mem_0

128x128

1bit

Mem_1

128x128

1bit

Mem_N

128x128

1bit

Input_1 Buffer Input_2 Buffer

Port A Port A Port A

Port B Port B Port B

A DI DO A DI DO A DI DO

I1_sel I2_sel

Microblaze

clock

domain

Image proc.

clock

domain

128bit data

path

PE_0 PE_1 PE_M

O_sel

128x1 processor array

• On-chip memories– 128x128 pixel images– 1 RAMB18– Separate port for fast

Microblaze access

• Separate clock domain• One line of Specialized

Processing Elements (PE)• Supported operations:

– Erosion, Dilation, Recall, Pixel wise logic operations

– Global white, black, change

• Expected performance:– 150MHz, 0.85us/op,

46,800op/frame

Vector processing unit for Kalman filter

64x64bit

Vect_0

*

+

D

A DI

DO

ReadAddrWriteAddr

64x64bit

Vect_1

A DI

DO

64x64bit

Vect_n

A DI

DO

Scratchpad

memory

DI DO

A DI DO

Microblaze clock

domain

• Tracking and 3D estimation• On-chip memories

– Scratchpad memory– Separate port for fast

Microblaze access – 64bit wide 64 element vector

registers

• Configurable vector length• Separate clock domain• 64bit double precision

floating point units• Supported operations:

– Independent multiplication and addition

– Multiply and add (MADD)

References

• T. Zsedrovits, A. Zarandy, B. Vanek, T. Peni, J. Bokor, and T. Roska, “Collision avoidance for UAV using visual detection,” in Proc. of 2011 IEEE Int. Sym. of Circuits and Systems (ISCAS), 2011, pp. 2173–2176.

• T. Zsedrovits, A. Zarandy, B. Vanek, T. Peni, J. Bokor, and T. Roska, “Visual Detection and Implementation Aspects of a UAV See and Avoid System,” in Proc. of 2011 20th European Conference on Circuit Theory and Design (ECCTD), 2011, pp. 472–475.

• Z. Nagy, A. Kiss, A. Zarandy, T. Zsedrovits, B. Vanek, T. Peni, J. Bokor, and T. Roska, “Volume and power optimized high-performance system for UAV collision avoidance,” in Proc. of the 2012 IEEE Int. Symp. on Circuits and Systems, 2012, pp. 189–192.

• A. Zarandy, T. Zsedrovits, Z. Nagy, A. Kiss, P. Szolgay, and T. Roska, “Cellular processor array based UAV safety system,” in Proc. of 2012 13th International Workshop on Cellular Nanoscale Networks and their Applications, 2012, pp. 1–2.

• A. Zarandy, Z. Nagy, B. Vanek, and T. Zsedrovits, “A five-camera vision system for UAV visual attitude calculation and collision warning,” Comput. Vis. Syst. Lect. Notes Comput. Sci., vol. 7963, pp. 11–20, 2013.

• A. Zarandy, M. Nemeth, Z. Nagy, A. Kiss, L. Santha, and T. Zsedrovits, “A real-time multi-camera vision system for UAV collision warning and navigation,” J. Real-Time Image Process., Sep. 2014.