cdr presentation final - university of arizona...3 requirements overview elliott liggett 7o 6f 7...

11/16/2010

1

Machine Vision and Autonomy Integration into an

Unmanned Aerial System

Critical Design Review

1 of 67Hans Hony Elliott Liggett Michael Palmer James Dianics Malcolm Gibson

November 18, 2011

Presentation Objectives

• Project Introduction

• Requirements Overview

• Design Overview

• High-level System Design

Strategic Test Plan

2 of 67

• Strategic Test Plan

• Bill of Materials

• Project Timeline

• Risk Analysis

• Review and Conclusion

Introduction Requirements Design Overview Strategic Planning Conclusion

The AIR Team

Project Introduction

The University of Arizona’s Aerial Robotics Club has sponsored us to help them design and build an aerial vehicle for competition in the

2011 AUVSI Student Unmanned Aerial Systems Competition

The competition vehicle is currently being developed by two teams:

The AVIONICS Team

3 of 67

• Composed of Aerospace Seniors

• Aerodynamic Design & Analysis

• Airframe Design

• Composites Manufacturing

• Flight Testing

• Avatar UAV Platform

• Composed of ECE, OSE, SYE, and ME seniors.

• Autopilot Integration into Aircraft

• Computer Vision System

• Image Acquisition System

• System Integration

(Avionics Module)


11/16/2010

2


The 2011 AUVSI Student Unmanned Aerial Systems Competition

Key Performance Parameter

Percent of Score

Autonomy Up to 30%

Imagery Up to 20%

Competition Overview:

• June 16-19th 2011

• Sponsored by industry leaders

• Oral Presentation

S t S f t O i

4 of 67

Imagery Up to 20%

Target Location Up to 20%

Mission Time Up to 10%

In-flight Re-tasking Up to 5%

2010 AUVSI SUAS Competition Breakdown of Flight-Test Score

• Systems Safety Overview

• Pre-Mission Briefing

• Flight Test Evaluation

• Mission Efficiency

• Performance



Mission Breakdown

5 of 67

Webster Annex Field in Maryland



Key Mission Objectives

Autonomy

• Take-Off and Landing

• GPS Waypoint Navigation

• Avoid designated no-fly zones

• In Flight Re Tasking

6 of 67

• In-Flight Re-Tasking

Machine Vision

• Accurately resolve 4’x 4’ targets

• Size, Shape, Location, Colors

• Alphanumeric Character

• Pop-up Target

• Operate autonomously

Camera System

• Remain orthogonal to ground

• Capture clear images

• Sufficient resolution

• Transmit data to ground station


11/16/2010

3

Requirements Overview

Elliott Liggett

7 of 67


Must:

Produce an aircraft and machine vision system.

• Aircraft must be capable of flight for 30 minutes under autonomous control

• Must positively identify all target characteristics:

8 of 67

o Color

o Shape

o Alphanumeric Character



9 of 67Introduction Requirements Design Overview Strategic Planning Conclusion

11/16/2010

4

Changes since PDR

1. After analyzing several lens and camera options: o Two cameras would be required for wide-

angle (120º) view.o Reduces lens distortions to imageo Increases GIFOV

Recent Changes

10 of 67

2. Since flight time and bandwidth are coupled to power:o Requirements increase dramatically with

slight changes to path. o Thus an optimum path algorithm will be

created.


Trade Studies

James Dianics

11 of 67

Trade Studies

Camera System


11/16/2010

5

Camera System

Trade Studies

13 of 67

Resolution vs. FOV


Camera System

Trade Studies

14 of 67

Resolution and FOV


Weight, Size and Power

Camera System

Trade Studies


11/16/2010

6

Field of View and Resolution

Camera System

Trade Studies

16 of 67

Low Weight and Power Consumption


Processing Speed and Compatibility

Onboard Computer

Trade Studies

17 of 67

Weight, Power Consumption, and Size


Software Design

Hans Hony

Malcolm Gibson

18 of 67

Malcolm Gibson

Elliott Liggett

Michael Palmer

11/16/2010

7

System Design Overview


Autopilot Concept of Operation

Control Loop Theory:

20 of 67

Control Loop Theory:

• Input – Transfer Function – Output

• Feedback Loop

• Poles and Zeros of the system

• Operational range of the various gains


OL

OLGain A

AK

1


21 of 67

Pros of Closed Loop Controls:

• Less Gain Optimizing and more programming

• Bonus!


11/16/2010

8





Using the GPSData_t Class:

• Access to Latitude, Longitude and Altitude

• Ground Speed and Direction

• Timestamp information

• Double Bonus!

Software Design

Hans Hony

Malcolm Gibson

24 of 67

Malcolm Gibson

Elliott Liggett

Michael Palmer

11/16/2010

9

Machine Vision Process

ma

ges

Gimbaled Camera System


FitPC Onboard ComputerOpenCV

Co

mb

ine

Im

USB

Filter Image


Read image into the Matlab

W k

26 of 67

Identify Object LocationCrop Region of Interest

Send Image and info to Ground Station

Workspace



C

B

1

A

me

ric D

ata

ba

se

Compare to characterdatabase

OUTPUTShape &

Character To USER

Determine Target Shape

Determine Target Character

27 of 67

Perform Analysis on Binary Image

Convert RGB Image to BW

Sh

ap

e D

ata

ba

se

Generate Shape Signature

Binary Im

age

Extract Alphanumeric Character

Compare to Shape database

2

3

Alp

ha

nu


11/16/2010

10


Target Area ( )ft3 Original Image

Target Background Color

OUTPUT Target Area, Color, and Location

To USER

Original Filtered Image

28 of 67

Remove large objects

Perform “blob” analysis on image

DeterminePixel Area (px)

Determine pxCentroid (X,Y)

AircraftGPS Coordinates(from autopilot)

length( ft) f (px, altitude)

Target GPS Location(orientation)

Extract RGB ValuesFrom Centroid


Software Design

Hans Hony

Malcolm Gibson

29 of 67

Malcolm Gibson

Elliott Liggett

Michael Palmer

System Software Integration

while (in_air) {

In-Air Image Acquisition Heartbeat

30 of 67

• Images and commands sent via HPN-SSH

}...


11/16/2010

11

System Software Integration

Ground Station Image Processing Flow


Software Design

Hans Hony

Malcolm Gibson

32 of 67

Malcolm Gibson

Elliott Liggett

Michael Palmer

Interface Design


11/16/2010

12

Interface Design


• Easy to Customize GUI

• TCP / IP Network Interface

• Consumer/Producer Queue

Interface Design

Ground Station Interface

35 of 67

Q

• Report Generating Toolkit

• Printer Drivers


• Shape Recognition

• Optical Character Recognition (OCR)

Interface Design

Machine Vision Station

36 of 67

• Target Matrix

• Confirmation Countdown

• Return Targets to Labview Ground Station


11/16/2010

13

• Device to World

o Lab View GUI

o Final Report

Interface Design

Hardware Level

37 of 67

• Device to Human

o Matlab GUI

o Target Recognition

o Confirmation Countdown


Analysis and Testing

Michael Palmer

Elliott Liggett

38 of 67

Elliott Liggett

Hans Hony

Malcolm Gibson

Camera Analysis

• Camera Calculations

o Field of View

o Pixel Area of Target

o Optical Overlap

o Resolution (in. per pixel)

LAARK Toolbox


( p p )

o Lens Error

o Resolution Error

• To Meet Specifications

o Greater than 5MP

o 4.5mm to 6mm C-Mount lens

11/16/2010

14

Camera Analysis



Michael Palmer

Elliott Liggett

41 of 67

Elliott Liggett

Hans Hony

Malcolm Gibson

Data-link Analysis

FSPLdB 20 log10 d 20 log10 f 36.58

Free-Space Loss:

Property Value

FSPL at 2.43GHz and 1 mile radius ‐104.0dB

Signal Strength

42 of 67

• More negative numbers indicate lower signal strength, thus lower speed.

Transmit Strength + 24.0dBm

Ground Station Antenna Gain + 10.0dBi

Aircraft Antenna Gain + 2.7dBi

Loss in Cables (total both ends) ‐ 8.0dB

Total System Losses ‐75.3dB

Express Range 2 performance ‐77.0dB @ 54Mbps


11/16/2010

15

Data-link Analysis

Signal Strength


Data-link Analysis

Bandwidth

• Must supply the ground with all photos within mission timeframe.

44 of 67

Property Value

Estimated Photos ~ 3MB @ 500 photo = 1.5GB

Transfer rate of 10Mbps ~ 20 minutes 17 seconds

Transfer rate of 54Mbps ~ 3 minutes 45 seconds

Total Mission Time Allowed 40 minutes

Decreasing Scores Result at 30 minutes or more


Completed Testing

Thus far…

• Tested machine vision options:o Compiled several test OpenCV appso Preliminary image target detection in Matlabo Image display in LabView

45 of 67

• Image transfer and remote commands:o Tested SSH session sharing (faster connections)o Tested remote command cuing at 20 commands per secondo Tested High Performance SSH (“HPN-SSH”)


11/16/2010

16


Michael Palmer

Elliott Liggett

46 of 67

Elliott Liggett

Hans Hony

Malcolm Gibson

Design vs. Requirements

Camera Design

Status Requirements

Yes 120° Field Of View (FOV).

Yes FPS faster rate than speed of aircraft and GIFOV between photos.

Yes USB or Ethernet Connected Device


Yes USB or Ethernet Connected Device

Yes Linux Drivers


Onboard PC

Status Requirements

Yes Small Form Factor.

Yes Onboard PC able to process images and transmit data efficiently.

Yes Onboard storage capable of meeting high demand of photos


Yes Onboard storage capable of meeting high demand of photos.

Yes Expandable to interface with peripherals not accounted previously.

11/16/2010

17


WiFi Card

Status Requirements

Yes WiFi capable of at least one mile transmission.

Yes Small form factor and compatible with onboard PC.

Yes Free (Although not required)


Yes Free… (Although not required)


Overall System

Status Requirements

Yes Machine Vision able to perform in under 30 minutes.

Yes Onboard payload less than 3lbs.

Yes Power consumption manageable


Yes Power consumption manageable.


Michael Palmer

Elliott Liggett

51 of 67

Elliott Liggett

Hans Hony

Malcolm Gibson

11/16/2010

18

Test Plan

Optics Test Plan

1. Finalize optics configuration

2. Purchase cameras and lenses

3. System compatibility test

4. Determine optimum overlap ratio

52 of 67

p p

a. Angle between cameras

b. Confirm 120 FOV requirement

5. Integrate into the Aero-dome

6. Measure and quantify distortion

7. Confirm weight and size requirements

8. Aerial Photography test


Test Plan

Machine Vision Test Plan

1. Test camera/software interface

2. Construct mock targets

3. Generate simulated images for machine vision testing

4. Prototype algorithms

53 of 67

yp g

a. Image stitching

b. Filtering

c. Object analysis

d. Information output

5. Confirm reliability in determining target characteristics

6. Determine computational power and processing speed requirements


Test Plan

Systems Integration Test Plan

1. Test camera/piccolo/computer interface

2. Test gimbal control loop

a. Test response speed

b. Tune sensitivity to eliminate noise

3. Ensure reliable connections between

54 of 67

components

4. Test for any RF interference

5. Test Battery life under various operating conditions

a. Full power draw

b. Estimated mission load

6. Test software integration

a. OpenCV > Matlab > Labview User 1Safety Pilot

User 2Target Analysis

User 3Autopilot Control


11/16/2010

19

Test Plan

Data-link Test Plan

1. Fly express range wifi card in sponsor’s aircraft to perform rigorous data-link test

2. Measure bandwidth

3. Measure range


4. Test repeatability / reliability

5. Test potential weather effects

6. Optimize antenna configuration

7. Develop failure protocols to ensure safety and recovery in the event of a connection loss

8. Perform range/reliability tests with Piccolo ground-station

Bill of Materials

Hans Hony

56 of 67

Bill of Materials


11/16/2010

20

Bill of Materials


Bill of Materials

Component Timeline Status

Fit‐PC2 (Onboard PC) 4 to 5 weeks Ordered

Express Range 2 WiFi Card and antennas 2 weeks Arrived

UI‐1495LE‐C (Camera) + 4.5mm Lens 2 weeks TBA

Component Lead Times


UI 1495LE C (Camera) 4.5mm Lens 2 weeks TBA

Gimbal mounts, servos, and dome 2 weeks TBA

Second Piccolo Autopilot TBA TBA

Additional Ground Computer TBA TBA

Risk Analysis and Future

James Dianics

60 of 67

11/16/2010

21

Risk Analysis

Identified Risks

Risk Severity Likelihood Risk Factor

Optical Dome 0.60 0.30 0.18

Data Link 0.40 0.20 0.08

61 of 67

System Stability 0.80 0.30 0.24

Physical Crash 0.75 0.20 0.15

Data Loss 0.10 0.50 0.05


0.4

0.5

0.6

od

Risk Analysis

Highest Risk Factors

Rank Risk

1 System Stability

2 Data‐link

3 O ti l D

Risk Analysis

62 of 67

0

0.1

0.2

0.3

0 0.2 0.4 0.6 0.8 1

Likelihoo

Severity

3 Optical Dome

4 Physical Crash


Risk Mitigation

• System Stability:o Testing, Testing, Testingo Watchdog Functionso Robustness

Data link

Risk Analysis

63 of 67

• Data-link:o Preliminary Testingo Reduce Data Transfer

• Optical Dome:o Testing

• Physical Crash:o Reduce Flight Time


11/16/2010

22

Gnatt Chart

Future Planning


Next Steps

Future Planning

• Purchase Specified Components

– Imaging System

– Piccolo II

• Initial Configuration Testing

• Software Development


p

– Open Source

• Optical Character Recognition

• Shape Detection

– Closed Source

• LAARK Server & Watchdog

• Autopilot Ground Station

• Machine Vision Station

• Test & Refine Algorithms

Conclusion

~ Thank you for your attention ~

Questions or Comments?

66 of 67

Presentation By:

Hans Hony Elliott Liggett Michael Palmer James Dianics Malcolm Gibson


cdr presentation final - university of arizona...3 requirements overview elliott liggett 7o 6f 7...

Documents