artificial vision-based tele-operation for lunar exploration students aaron roney, albert soto,...
TRANSCRIPT
Artificial Vision-Based Tele-Operation for Lunar Exploration
Students Aaron Roney, Albert Soto, Brian Kuehner, David Taylor, Bonnie Stern,
Nicholas Logan, Stephanie Herd
NASA JSC Mentors Dr. Bob SavelyDr. Mike Goza
Project Mentor Dr. Giovanni Giardini
Project AdvisorProf. Tamás Kalmár-Nagy
Project Members
Electrical Engineering
Computer Engineering
Nuclear Engineering
Mechanical Engineering
Mechanical Engineering
Aerospace Engineering
Aerospace Engineering
Nicholas Logan
Stephanie Herd
Aaron Roney
Albert Soto
Bonnie Stern
Brian Kuehner
David Taylor
Freshman
Freshman
Sophomore
Sophomore
Junior
Senior
Senior
OutlineMotivation and ObjectivesEgo-Motion TheoryCode FlowCalibration and RectificationHardwareTesting ResultsFuture Work
Motivation
Lunar surface exploration
Human perspectiveIn safetyWith low risk
3D environment reconstruction
Self location with artificial vision system
Objectives
Vision SystemEgo-Motion estimationEnvironment reconstruction
Visual Feedback System for Tele-Operations
Tele-Operation SystemRemote control mobile unit
Hardware and Mechanical Implementation
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Ego-Motion Theory
3D Reconstruction Theory
Left image Right image
uleftp uright
p
vleftp vright
p
uleftp
It is impossible to compute the 3D coordinates of an object with a single image
Solution: Stereo Cameras
Disparity computation
3D reconstruction
Image
Disparity map computation: Given 2 images, it is a collection of pixel disparities Point distances can be calculated from disparities
Environment can be reconstructed from disparity map
Left Image Right Image Disparity Map
Environment Reconstruction
Perspective Projection Equation
Main goal: Evaluate the motion (translation and rotation) of the vehicle from sequences of images
Ego-Motion Estimation
Solving will give change in position of the vehicle
Optical Flow Example Optical Flow is related to vehicle movement through the Perspective Projection Equation
Least Square solution
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Code Flow
Sony VAIO - Pentium 4
Logitech QuickCam Deluxe
Image Processing CodeCalibration ParametersCalibration Parameters
Acquire ImagesAcquire Images
Rectify ImagesRectify Images
Ego-Motion Estimation
Ego-Motion Estimation
Wireless 802.11 Network
Wireless 802.11 Network
Ground StationGround Station
Mobile Unit Detailed Code
Calibration ParametersCalibration Parameters
Snapshot Image Matrix
Snapshot Image Matrix
Image Parameters: Gray Scale (640x480)
…
Image Parameters: Gray Scale (640x480)
…
Acquire Image Rectify Images
Rectified Image Matrix
Rectified Image Matrix
Save ImageSave
Image
T = 0.15 sec T = 0.5 sec
Ego-Motion EstimationEgo-Motion Estimation
Apply Distortion Coefficient to Image Matrix
Apply Distortion Coefficient to Image Matrix
Wireless 802.11 Network
Wireless 802.11 Network
Ground StationGround Station
Ego-Motion Estimation OverviewFind Features
in Right ImageFind Features
in Right Image
Calibration ParametersCalibration Parameters
Right ImageRight Image
Left ImageLeft Image
Track Right Image Features in Left Image
Track Right Image Features in Left Image
New Right Image
New Right Image
New Left Image
New Left Image
Find Features in New Right
Image
Find Features in New Right
Image
Find Features in Left ImageFind Features in Left Image
Find Features in New Left
Image
Find Features in New Left
Image
Track Right Image Features in New
Right Image
Track Right Image Features in New
Right Image
Track Right Image Features in New
Left Image
Track Right Image Features in New
Left Image
Discard All non-Identical Points in
All images
Discard All non-Identical Points in
All images
Displacement Vector
(X, Y, Z, X-Rot, Y-Rot, Z-
Rot)
Displacement Vector
(X, Y, Z, X-Rot, Y-Rot, Z-
Rot)
T = 3 sec
Image Feature Matrix
Image Feature Matrix
Wireless 802.11 NetworkWireless 802.11 Network
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Calibration and Rectification
Calibration and Rectification
Calibration: Utilizes Matlab tools to determine image distortion associated with the camera
Rectification: Removes the distortion in the images
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Hardware
Hardware
Mobile Unit
TROPOS Router
Laptop
Web CamerasMobile Unit Base Station
Linksys Router
Operator Computer
Command Computer
Wireless802.11
Wireless802.11
Improvements Implemented in the System
Improved robustness of the softwareImplemented a menu driven system for the operator
using Matlab’s network handling protocolAllowed pictures to be takenRun Ego-motionSending all the results to the operatorGraphic displaying of optical flow
Reduced crashing
Achieved greater mobile unit control
Mobile UnitVehicle Courtesy of Prof. Dezhen Song
Baseline
D
L
FOV1 FOV2
α
Horizontal View
Camera support system3-DOF mechanical neck:
Panoramic rotationTilt rotationTelescopic capability
Controlled height and baseline length
Visual System (onboard the Vehicle)
Ground Station
Vehicle Hardware
Wireless 802.11 Network
Wireless 802.11 Network
Testing Result
Test EnvironmentLight to simulate solar exposure
Black background to eliminate background features
Lunar Environment
Walls to eliminate stray light and side shadows
Measured displacements
Test Setup25 pictures taken from each location (0, 5, 10 and 15
cm) in the Z direction (perpendicular to camera focal plane), unidirectional movementSet 1 25 images located at Z=0Set 2 25 images located at Z=5Set 3 25 images located at Z=10Set 4 25 images located at Z=15
The distances are measured using a tape measureThe cameras are mounted using a semi ridged fixture
Determining the Number of Features
The standard deviation decreases with the more features
But the accuracy of the results decrease
100 Features were selected
Results for 5 cm displacement
Used all 100 images
Compared each set to the previous
Ego-Motion: ExampleOptical Flow Left Image Optical Flow Right Image
RANSACdegree 5 cm Std Dev 10 cm Std Dev 15 cm Std Dev
5 5.31 3.65 9.43 4.39 13.00 3.55
15 4.86 2.09 8.30 3.40 13.65 6.39
30 4.35 1.66 8.21 4.03 15.27 6.01
Problems
Images were not rectifiedPossible motion of cameras between imagesNo image filteringCamera mounting is misalignedImages acquired from the right camera appear
blurry
Conclusions andFuture Work
Demonstrated:Ego-motion estimationEnvironment ReconstructionVehicle control and movementSystem integration
Future Developments:Filtering and improving resultsIncrease the robustness of the vision systemCreate a visual 3D environment map