gm-carnegie mellon autonomous driving crl curb detector 1

GM-Carnegie Mellon Autonomous Driving CRL

Curb DetectorCurb Detector

1


Project Leader Name & Functional AreaWende Zhang (GM R&D / ECS Lab) David Wettergreen (CMU)

Timing Date______________ Initial May, 2014Midterm iMay, 2015Final May, 2016

Resources 2013 2014 2015Total Material Cost[US$] 100k 100k 100kTotal Headcount (GM) 0.1 0.1 0.1Total Headcount (CMU) 1 1 1

GM Confidential

Automated Image Analysis for Robust Detection of Curbs

DescriptionCurbs are important cues on identifying the boundary of a roadway. Drivers understand an appropriate parking spot as defined by the curbs when reverse or parallel parking. Detecting curbs and providing information to assist drivers is an important task for active safety. Curb location is also crucial to autonomous parking systems.Visual indications of curbs are widely various in the appearance. For example, under perspective imaging, projection of 3-dimensional curbs into 2-dimensional image plane distorts most of the curbs’ geometry properties, such as its angle, distance, and ratio of angles. Also, all curbs might be seen different because of age, wear, damage and lighting. Methods of detecting, localizing, and classifying curbs must address this diversity. This is to say, there is not a fixed template or set of templates that could be applied to reliably detect curbs through images.Nevertheless visual appearance is how human drivers successfully detect curbs. Although physical structure can be sensed with some ranging sensors it not distinctive (two offset planes) or diagnostic of the roadway edge. Therefore we choose to pursue visual appearance. This new project will develop an automated curb detection through :• Choosing appropriate features, learning those features to detect, and classifying the detected curbs• Utilizing the calibrated camera to fuse the 3D geometry information• One year development plan: detect, localize, and classify curbs using in-vehicle vision sensor with backward looking view with wide field of viewMotivation/Benefits• Identify the boundary of a road way in urban driving• Understand an appropriate parking spot as defined by the curbs when reverse or parallel parkingDeliverable / Technology Insertion into GM (What, When, Where)• Problem Definition: Survey of curbs• Data collection: Database of definite curb images and diverse curb images• Application: Detect curb features in perspective imagery• Experimental validation and performance analysis• Annual report

Detect and classify features using learning-based

method


Use Case SlidesUse Case Slides

3


AssumptionsAssumptions

• Color monocular camera

• Known camera motion

• Known intrinsic parameters

• Maximum speed dependent upon frame rate

4


Use CasesUse Cases

• Parking lots– Backward parking– Parallel parking

• Driveways

• Roadways– Single lane– Multi-lane

5


Parking lotsParking lots

• Scenario : Curbs exist behind of a vehicle; rear-view camera with wide field of view

• Success : Detect and localize curbs on images;(Optional) estimate the distance from a vehicle to curbs

6

GM

CURB

CURB


Parking lotsParking lots

• Scenario : Parking curbs exist behind of a vehicle; rear-view camera with wide field of view

• Success : Detect and localize parking curbs on images;(Optional) estimate the distance from a vehicle to parking curbs

7

GM

CURB

CURB


DrivewaysDriveways

• Scenario : Curbs exist at the side of the entrance of driveway; front-view camera with wide field of view

• Success : Detect and localize curbs on images and indicates driveways as traversable path

8

GM

CURB CURBDriveways


RoadwaysRoadways

• Scenario : Curbs exist at the side of the road; wide field of view camera

• Success : Detect and localize curbs on images and indicates curbs as the non-traversable path and the boundary of road

9

GM

CURB

CURB

GM


Flow ChartFlow Chart

10

Detection

Tracking

: Localize relevant curbs in each image

: Localize the detected curbs in remained images

Edge

Texture

Segmentation


Edge DetectionEdge Detection

11

UndistortedDistorted

Bird’s-eye view

Edge

Edge


SegmentationSegmentation

12


Texture ClassificationTexture Classification

13


Development PlanDevelopment Plan

• Develop and test simple features

• Train classifiers to detect and localize curbs

• Evaluate classifier performance

• Add complex features

• Test quantify detection and localization performance

• Train color classifiers to interpret appropriate parking spots

• Motion Stereo to exploit 3D geometry

14


SchemeScheme

15

ExtractFeatures

TrackingEdge

DetectionClassification

- Geometricconsideration

- Horizontal- Long features- Thin features- Color- Texture- Curvature

- Filters• Edges• Intensity

differences• Gradients

- Appearance based tracking


Data collectionData collection

16

• Using 180 degree field of view camera

• Install underneath the side mirror, tilt 45 degree down to the ground

Sample images


Camera CalibrationCamera Calibration

• Wider field of view, more distortion

• Camera calibration is necessary in order to find geometry constrains (e.g., edges…)

• Using OCamCalib (Omnidirectional Camera Calibration Toolbox) to calibrate camera

17

Sample undistorted images


Shape InformationShape Information

• Edge detection

• HOG feature

18



19

Input Image at t

Undistorted Image

Edge Detection

SequentialRANSAC

Extract Dominant

Edges



20

Input Image at t

Undistorted Image

Edge Detection

SequentialRANSAC

Extract Dominant

Edges



21

Input Image at t

Undistorted Image

Edge Detection

SequentialRANSAC

Extract Dominant

Edges

Two or Three parallel lines with small offsets are important cue for curbs


HOG featureHOG feature

22



23



24

Input image HOG map

Score map Output image


PrerequisitePrerequisite

• The maximum distance of ‘Curb Detection’ from a vehicle should be defined.

• Given extrinsic parameters and the maximum distance, the followings can be estimated.– Different size of HOG model– Region of interest

25

short medium long


Geometry calculationGeometry calculation

26

GM

CURB

Local Area2.7-3.6 m (9-12 feet)

GM

Maximum detect distance

GM-Carnegie Mellon Autonomous Driving CRL 27

http://www.cadillac.com/srx-luxury-crossover/features-specs/dimensions.html

Examples

http://www.cadillac.com/cts-sport-sedan/features-specs/dimensions.html

Cadillac SRX

Cadillac CTS


Geometry calculationGeometry calculation

28

CURBG

M

Maximum detect distance = 2.1 meter


1m 1m 1m

Image Sample with distance measure

- The center of the camera is 1.05m from the ground.

- The angle of the camera is 45 degree down from the horizontal.

- ROI will be reduced. (Red transparent rectangle)

- ROI will be changed based on the extrinsic parameter.


Lane MarkingsLane Markings

• Since lane markings have strong edges, we need to eliminate outputs from lane markings.

• Parts of images which contain lane markings can be removed by detecting white blobs.

30


Result VideoResult Video

31


Performance MeasurePerformance Measure

• Choose 300 testing images– Positive samples: images which contains full length of

curbs– Negative samples: images without curbs

• We consider curbs are detected when the horizontal length of the detected curbs are bigger than half of the horizontal length of image.– Since the size of image is 480 by 720, we consider

curbs are detected and the sum of the length of the detected curbs are bigger than 360.

32


Performance MeasurePerformance Measure

33

Groundtruth

Positive Negative

Positive 80 13

Negative 24 183

length of detected curb

total length of image> 0.5


Future WorksFuture Works

• Features of curb detection– Redundant information through multiple images

• Include tracking system to recover false negatives

– Continuity• Develop likelihood function to recover false negatives and

remove false positives

– Height

• Front-view camera– Mount 180 degree field of view camera on the front

bumper

34


Front-view Camera ConfigurationFront-view Camera Configuration

35

gm-carnegie mellon autonomous driving crl curb detector 1

Documents