from rgb-d to low-resolution tactile: smart sampling and early testing
DESCRIPTION
FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING. Hao Tang 1,2 , Margaret Vincent 3 , Tony Ro 3 and Zhigang Zhu 2 1 Department of Computer Information System, Borough of Manhattan Community College 2 Department of Computer Science, City College & Graduate Center - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/1.jpg)
FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING
Hao Tang1,2, Margaret Vincent3, Tony Ro3 and Zhigang Zhu2
1 Department of Computer Information System, Borough of Manhattan Community College2 Department of Computer Science, City College & Graduate Center
3 Department of Psychology and PhD Program in Cognitive Neuroscience, City College & Graduate Center
City University of New York
July 15, 2013
![Page 2: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/2.jpg)
Outline of the Talk Background & Motivation Related Work Sampling and Rendering Methods Experimental Results and Testing Conclusions
![Page 3: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/3.jpg)
Background & Motivation Background & Motivation Related Work Sampling and Rendering Methods Experimental Results and Testing Conclusions
![Page 4: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/4.jpg)
Background and Motivation
Visual prostheses Retinal prostheses (RP): partially restore vision Tongue stimulator (TS): transduce the shape, size, and
location of objects Challenges and limitations
Low resolutions RP: <10x10 channels TS: 20x20 channels
Rendering difficulties Omitting small objects Highlighting important objects
![Page 5: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/5.jpg)
Our Solutions Smart sampling Background removal Parallax simulation Object highlighting Path directions
![Page 6: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/6.jpg)
Related Work Background & Motivation Related Work Sampling and Rendering Methods Experimental Results Conclusions
![Page 7: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/7.jpg)
Related Work Computer Vision algorithms
Use cell phone to find machine-readable signs (Coughlan et al. 06, 08)
Depth maps from stereo cameras to aid navigation (Audette et al. 00, Gonz´alez-Mora et al. 09)
Staircase (Se and Michael 00, Lu et al. 05, Pradeep et al. 08) and zebra-crossing detection (Se 00)
Obstacle detection using stereo vision (McCarthy et al. 2011)
![Page 8: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/8.jpg)
Sampling and Rendering Methods
Background & Motivation Related Work Sampling and Rendering Methods Experimental Results Conclusions
![Page 9: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/9.jpg)
System Pipeline
![Page 10: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/10.jpg)
RGB-D Image Acquisition and Segmentation
Input: stereo images Algorithms: Patch-based stereovision (Tang & Zhu
TCSVT’12)
Patch and interest point extraction Color segmentation and feature extraction
Three-step stereo match Global match – match of a group of feature points Local match – match of individual feature point Plane fitting – RANSAC
Plane merging and parameter refinement Output: RGB-D + Segmentation
![Page 11: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/11.jpg)
Sampling and Rendering Methods
Smart-Sampling
![Page 12: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/12.jpg)
Sampling and Rendering Methods
Background removal based on scene labeling
Motion parallax simulation using 3D Dynamic object re-illumination and
highlighting
![Page 13: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/13.jpg)
Sampling and Rendering Methods
Path Directions 3D locations of obstacles are available Discriminating orientations of straight
lines is easy
Turn Left Go Straight Turn Right
![Page 14: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/14.jpg)
Experimental Results Background & Motivation Related Work Sampling and Rendering Methods Experimental Results Conclusions
![Page 15: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/15.jpg)
Tongue stimulator –Brainport by Wicab, Inc.
BrainPort Vision Device Non-surgical assistive visual prosthetic device Translates information from a digital camera to your
tongue Includes: a tongue array, res: 20x20; size: 3cmx3cm, a
digital video camera and a hand-held controller (zoom and contrast inversion)
Functionalities: recognize high-contrast objects and their locations
Original image Sampled image Inversed Sampled image
![Page 16: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/16.jpg)
Experimental Results RGB-D image data
Depth imageReference image
![Page 17: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/17.jpg)
Result of Smart Sampling
![Page 18: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/18.jpg)
Highlight Object of Interest Background removal based on scene
labeling Motion parallax simulation using 3D Dynamic object re-illumination and
highlighting
Sampling results using motion parallax simulation
Sampling results of highlighting objects in a closed range
![Page 19: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/19.jpg)
EARLY TESTS USING BRAINPORT
Line Orientation 3 pixel & 5 pixel widths / 45° diagonal
![Page 20: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/20.jpg)
Turn Left
Go Straight
Turn Right
EARLY TESTS USING BRAINPORT Navigation using line orientation
![Page 21: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/21.jpg)
EARLY TESTS USING BRAINPORT
Trajectories
![Page 22: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/22.jpg)
Conclusions Background & Motivation Related Work Sampling and Rendering Methods Experimental Results Conclusions
![Page 23: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/23.jpg)
Conclusions Transduce image and 3D scenes
RGB-D image acquisition and segmentation Smart sampling Different rendering methods
Early tests on Brainport Line orientation Navigation using line orientation
![Page 24: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/24.jpg)
Future work Conduct more experiments on
Brainport Navigation using line orientation
Use Brainport with RGB-D sensor Object localization/manipulation
![Page 25: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/25.jpg)
Acknowledgments US National Science Foundation
Emerging Frontiers in Research and Innovation Program under Award No. EFRI-1137172
National Science Foundation Award, No. BCS-0843148
City Seeds City College 2011 President Grant for Interdisciplinary
Scientific Research Collaborations. PSC-CUNY Research Award
Grand Round 44
![Page 26: FROM RGB-D TO LOW-RESOLUTION TACTILE: SMART SAMPLING AND EARLY TESTING](https://reader036.vdocuments.us/reader036/viewer/2022062801/5681442c550346895db0c525/html5/thumbnails/26.jpg)
Thank you!&Questions?