gm-carnegie mellon autonomous driving crl titleautomated image analysis for robust detection of...
TRANSCRIPT
![Page 1: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/1.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Title Automated Image Analysis for Robust Detection of Curbs
Thrust Area Perception
Project Lead David Wettergreen, CMUWende Zhang, GMInna Stainvas, GM
Contributors JongHo Lee, CMU
1
![Page 2: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/2.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
ScheduleCurb location Sensor location Methodology Date
On the side Bottom of the side mirror Visual appearance ~ Jan. 2014
In front The front bumper Geometric structure ~ May. 2014
In front The front bumper Appearance + Geometry with production camera
~ Nov. 2014
DeliverablesDemonstration: In-vehicle curb detection
Annual reports
2
![Page 3: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/3.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Objectives
Develop reliable methods of detecting, localizing, and classifying features associated with curbs using in-vehicle, low-cost, monocular vision sensor
Localize curbs within a range of 5 meters with 99% accuracy
3
![Page 4: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/4.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Approaches for Curb Detection Using Mono Camera Images
• Appearance-based image analysis (~ Nov. 2013)• Extract features
• Evaluate performance
• Geometry-based image analysis (~ May. 2014)• Structure-from-motion to estimate camera motion
• Multi-resolution plane sweeping algorithm to create 3-D point cloud
• Plane fitting to detect curb
• Combine appearance and geometric analysis (This Review)
4
![Page 5: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/5.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
• Appearance-based image analysis (~ Nov. 2013)• Extract features
• Evaluate performance
• Geometry-based image analysis (~ May. 2014)• Structure-from-motion to estimate camera motion
• Multi-resolution plane sweeping algorithm to create 3-D point cloud
• Plane fitting to detect curb
• Combine appearance and geometric analysis (This Review)
Approaches for Curb Detection Using Mono Camera Images
5
![Page 6: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/6.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Appearance-based image analysis
6
![Page 7: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/7.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Edge Detection
7
![Page 8: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/8.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Detect Curb Using HOG* Feature
* Histogram of Oriented Gradients
Input image HOG imageCurb model
8
![Page 9: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/9.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
• Appearance-based image analysis (~ Nov. 2013)• Extract features
• Evaluate performance
• Geometry-based image analysis (~ May. 2014)• Structure-from-motion to estimate camera motion
• Multi-resolution plane sweeping algorithm to create 3-D point cloud
• Plane fitting to detect curb
• Combine appearance and geometric analysis (This Review)
Approaches for Curb Detection Using Mono Camera Images
9
![Page 10: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/10.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Geometry-based image analysis
Input image Depth image
Ground plane estimation
3-D point cloud
10
![Page 11: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/11.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Plane Fitting
11
![Page 12: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/12.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
• Appearance-based image analysis (~ Nov. 2013)• Extract features
• Evaluate performance
• Geometry-based image analysis (~ May. 2014)• Structure-from-motion to estimate camera motion
• Multi-resolution plane sweeping algorithm to create 3-D point cloud
• Plane fitting to detect curb
• Combine appearance and geometric analysis (This Review)
Approaches for Curb Detection Using Mono Camera Images
12
![Page 13: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/13.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Appearance at t+1
Schematic Overview
Input at t+1
Input at t Appearance at t
Geometry
Candidate regions
Annotate curb region
![Page 14: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/14.jpg)
GM-Carnegie Mellon Autonomous Driving CRL 14
Appearance
- For each image, divide intom x n grids- m: image height / grid
size (pixels)- n: image width / grid size
(pixels)
Image at t
![Page 15: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/15.jpg)
GM-Carnegie Mellon Autonomous Driving CRL 15
Appearance
- For each grid, classify among two classes (road, curb)- uniform Local Binary
Pattern (LBP)
Image at t
![Page 16: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/16.jpg)
GM-Carnegie Mellon Autonomous Driving CRL 16
Local Binary Pattern
64 63 52
85 152 227
189 167 205
0 0 0
0 1
1 1 1
threshold
Binary: 00011110Decimal: 30
0 1 2 3 255
![Page 17: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/17.jpg)
GM-Carnegie Mellon Autonomous Driving CRL 17
Appearance
- Once all the grids of two images are classified, get the intersection of them
Output at t+1Output at t Intersect
![Page 18: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/18.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Geometry
- Green lines shows the vectors from the interesting points of image at time t (blue dots) to those of image at time t+1 (red dots)
- Calculate the 3-D points using camera matrix
![Page 19: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/19.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Appearance + Geometry
- For each grid,- Fit the best plane using 3-D
points - Compute the normal vector- Determine the normal vector
is a road surface or a curb surface
![Page 20: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/20.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Appearance + Geometry
![Page 21: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/21.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Extend Curb Region
- If the appearances are similar, extend the curb region- Calculate the distance of LBPs using chi-square
![Page 22: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/22.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Extend Curb Region
![Page 23: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/23.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Track Curb Region
- For the next frames, tracking the appearance of the curbs- When tracking, keep checking the geometry constraint to
remove the false positives if exist
Input at t+2 Input at t+3 Input at t+4
![Page 24: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/24.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Curved curb case
Extend Curb RegionCombine Analyses
![Page 25: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/25.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Curb Detection using Production Camera
Image size : 480 by 640FOV: 180 degree
![Page 26: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/26.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Test Curb Detection
Image Size: 640 x 480 (pixels)ROI: 640 x 160 (pixels)Size of grid: 20 x 20 (pixels)Number of grids: 32 x 8
Output of the appearance-based curb detection
![Page 27: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/27.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Test Curb Detection
Remove outliers based on cluster size
Find edges using Canny operator inside candidate region
![Page 28: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/28.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Test Curb Detection
- Fit polynomials to each segments, and check lines for similar curvatures (blue), and remove high curvatures (red) Annotate curb region on the
original input image
![Page 29: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/29.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Future WorkApplication: Operate real-time curb detection in vehicle
~ May 2015
29
![Page 30: GM-Carnegie Mellon Autonomous Driving CRL TitleAutomated Image Analysis for Robust Detection of Curbs Thrust AreaPerception Project LeadDavid Wettergreen,](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e375503460f94b27037/html5/thumbnails/30.jpg)
GM-Carnegie Mellon Autonomous Driving CRL
Thank you
Questions ?
30