vision based landing site detection
DESCRIPTION
The Ppt uploaded here is a concise abstract and overview of the method and the theory involved in extracting 3D information about the terrain under consideration using 2D images which can be further used to detect a suitable landing site and thus guide the UAV (preferably a VTOL UAV) for landing purposes at a particular region of interest as per the mission requirement. The Ppt demonstrates the numerous methods of getting the disparity estimate and thus using it to get the range map of the same and then comparison of the methodology adopted is made with the online data-set of stereo vision available at the middlebury website. The derived 3D map is further analysed to detect planes (using RANSAC) and thus compare it with the desired plane as per the UAV requirements for landing. The coding for the same is done using Matlab, C/C++ and the image processing is carried out using the open source OpenCV library and the Point Cloud Library to acquire and display the 3D mapping respectively.TRANSCRIPT
Vision Based Landing Site Determination
Submitted By :
Manish Tripathi
Enrollment No: A47171209021
Amity University
Human Depth Perception
The Overall System
Literature Review
• Projective Geometry for 3D scene analysis Camera Projection
I. Pin Hole Camera Model (Ideal Projection)
II. Actual Camera Model (Distortions Due to Camera and sensors)
III. Coordinate Transformation
Epipolar Geometry I. Epipolar Constraint
II. Fundamental Matrix/ Homography Matrix / Essential Matrix
III. Stereo Camera Geometry
Correspondence Matching
Triangulation and 3D Reconstruction
• Planar Segmentation
• Landing Site Features
Projective Geometry for 3D scene analysisCamera Projection Pinhole Camera Model [ Ideal Projection ]
Zero Aperture
All rays follow a Straight Line and pass from one point only
Actual Camera Tangential And Radial Distortions due to non zero lens Aperture
Can be removed using good quality Camera and sensor
Or by software
Projective Geometry for 3D scene analysis Camera Projection Continued…
Coordinate Transformation
Epipolar Geometry
Projective Geometry for 3D scene analysis
• Epipolar Geometry Continued… Fundamental Matrix [ Relation In terms of World Coordinates]
Essential Matrix [ Relation In terms of Pixel Coordinates ]
Homography Matrix [ Planar Orientation ] [For Rectification]
• Triangulation for 3D construction3D Range Map of Every Pixel
Projective Geometry for 3D scene analysisCorrespondence Matching
Area Based Block Matching (Local Method)
Graph Cut ( Global Matching)
Based On Energy Minimization (Slower Method)
Planar Segmentation
• Equation of Plane in 3D
ax+by+cz+d=0; [a, b, c] are Normal coefficients
[d] is distance from origin (Camera centre)
• RANSAC Algorithm Iteratively selects 3 Points and tries to fit the plane model using its distance
from the fourth randomly selected fourth point within a certain Threshold and would join all the points within that range.
Used Extensively to fit Mathematical models( Planes, Lines etc).
Orientation -> Perpendicular to the View Axis within a certain Tolerance angle
Landing Site Detection
• Identify landing sites
Hazard free
Terrain is suitable
Large enough to fit UAV
• Assumptions:
The camera mounted perpendicular to plane of the ground, pointing straight down.
The vertical axis of the camera image plane is aligned with the principal axis of the UAV.
• General Approach:
Generate 3D terrain map from consecutive images
Determine surface roughness and slope
Choose area that fits footprint of helicopter and minimizes roughness and slope using Plane Fitting on 3D Point cloud
Analysis and Method Used
• Stereo Setup Selection of Baseline
Too small: large depth error
Too large: difficult search problem, increase in outliers
Project analysis and Block Level Design
Disparity Estimation ResultsSequence of images captured by one camera
Two Cameras with non-planar placement
Exact Co-planar positioning of Camera
• Exact Co-planar positioning of Camera
High Accuracy
Disparity Estimation• Most important part of the whole process.
• Comes Under The Process of Correspondence Matching
• Area Based Block Matching based on minimisation of Sum of Absolute Differences (SAD) i.e.
disparity(δ) = arg[min(∑w(IL(i,j)-IR(x+i,y+j)]- (x,y)L
Faster than the Graph Cut method.
• Window Size Selection:
Small : Sensitive to noise; Thus less accurate but faster calculation
Large : Less sensitive to noise but Finer details get lost .
Thus I selected window size as 7X7 to get better results
• Disparity Range should be optimum to get balance between time for execution and Accuracy as each gray level value corresponds to one range
Disparity Estimation
• Result using stereo dataset available at Vision.Middlebury website
• Result using Raw images captured by my setup
CALIBRATION• Converting Disparity to Range requires the Projection
Matrix(f{focal length, Sx,y,Ox,y,relative orientation}
• Calibration is done to find out:Focal length of lens along X axis and Y axis
Lens displacement along X axis and Y axis
3 numbers that describe radial distortion
2 numbers that describe tangential distortion
The position and orientation of the camera in the real world (x, y and z) to calculate real world distance from pixel values in m,cm,mm etc.
CALIBRATION
• For Calibration we capture many images of a standard pattern like a Chessboard Pattern at different orientations from the two lenses :
Straight line pattern and all individual corners can be easily detected.
Independent of nonlinearity in the image acquisition.
Rectification
• To remove Image Distortions from the values calculated in Calibration
• To achieve epipolar constraint that is all conjugate pairs should lie on same rows so as to reduce search domain for point correspondences to a single row .
• We remap to remove distortions and find the new Camera Matrix.
Z-Coordinate Reconstruction
• Input :Disparity at each pixels of the overlapping region
X,Y coordinate of each pixel
Projection matrix ‘Q’ calculated from calibration to convert disparity to distance
• Output:Z coordinates at each pixel.
Formula :Z=T*ƒ/d (=δ)
3D Reconstruction
• Creating a Range Map of the image under consideration
• Done using Point Cloud Library
Planar Segmentation
• Objective :Finding planes that are perpendicular to the view angle that is normal along
z-axis.
Uses RANSAC Algorithm
Calculate Distance to the plane detected.
Contour Generation• Outline representing the shape or form of the image.
Software Tools
C/C++ in Visual Studio 2010 Compiler.
OpenCV 2.4.7 Library to do computer vision like image acquisition , calibration, Rectification, Block Mathing
Point Cloud Library 1.6.0 to do 3D Reconstruction and planar segmentation
The whole process can be implemented in MATLAB 2013a in its computer vision toolbox but is slow comparatively.
MATLAB for plot analysis and contour generation.
Inferences• 3D Range Map of the view under interest is generated.
• Plane perpendicular to the z-axis and its distance is also calculated.
• The Contour of the Image is also created.
Future scope for research • Comparing the footprint of landing strip suitable for landing
with plane detected in the image.
• Working on Multi-baseline Stereo imagery using single camera to get higher accuracy in Depth Perception
• Using Monocular cues to improve the depth.