bryan willimon clemson university 2013 committee: stan birchfield (committee chair)
DESCRIPTION
Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems. Bryan Willimon Clemson University 2013 Committee: Stan Birchfield (committee chair) Ian Walker (co-advisor) Adam Hoover Damon Woodard. Robotic systems are divided into two groups. Industrial Recycling Mail - PowerPoint PPT PresentationTRANSCRIPT
Bryan WillimonClemson University 2013
Committee:Stan Birchfield (committee chair)
Ian Walker (co-advisor)Adam Hoover
Damon Woodard
Sensing Highly Non-Rigid Objects with RGBD Sensors for Robotic Systems
Robotic systems are divided into two groups• Industrial
• Recycling• Mail• Food• Laundry
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry
Why Domestic Robots?• robotic systems are able to accomplish
chores around the house• Domestic robotic systems are getting more
attention in the news
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• Laundry
• Domestic• Outdoor
• Lawn care• Pool cleaning• Gutter cleaning• Window cleaning
• Indoor• Floor cleaning• Pet assistance• Picking up objects• L
Laundry is an Important Problem
Laundry
Laundry is an Important Problem• Industries and research institutes are making
attempts to solve the process
PR2 at UC Berkeley
NEDO Laundry Handling System
Laundry
Difficulties in the Laundry Problem• What are current problems that make a laundry
system difficult to automate?
• Highly deformable objects• Infinitely large number of configurations• Lots of possible grasp points
• The laundry problem is still in a research stage
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
Kita et al. use a humanoid system to recognize the state of clothes using a cloth model with 22 out of 27 attempts correctly classified
Previous Work on Classification for Robotics
Cusumano-Towner et al. were aimed at classifying the category of seven articles with a success rate of 20 out of 30 trials
Our system does not use predefined cloth models or dual manipulators
IsolationGraph-based SegmentationStereo MatchingDetermining Grasp Point
Classification Hanging Position
Binary Silhouettes Visual-based shape and
appearance informationLying Position
Low level features Characteristics Selection Masks
Classification Framework
Classification in a …
Hanging Position Lying Position
Experimental Results With 6 categories, 5 items per category, and 20 images per item,
the dataset collected by the extraction / isolation procedure consists of 600 images
This dataset was labeled in a supervised learning manner so that the corresponding category of each image was known
Two experiments were conducted:
1) Extraction and isolation process
2) Interaction vs. Non-interaction
Extraction and isolation process:
The image taken by one of the downward-facing stereo cameras
The result of graph-based segmentation
The object found along with its grasp point (red dot)
The image taken by the side-facing camera
The binary silhouettes of the front and side views of the isolated object.
Experimental Results
Interaction vs. Non-interaction:
The process of interacting with each article of clothing provided the system with multiple views using various grasp locations, allowing the system to collect 20 total images of each object.
Experimental Results
Two articles were compared by examining the 400 match scores between their pairs of images (20 images per article).
Classification in a …
Hanging Position Lying Position
Classification in a Lying Position
Experimental Results The proposed L-C-S-H approach is applied to a laundry
scenario.
Two different scenarios involved using: 3 categories {shirts, dresses, socks} 7 categories {shirts, dresses, socks, cloths, pants, shorts,
jackets}
Each scenario is run through three experiments:
1) Baseline System (L-H)
2) L-C-H approach
3) L-C-S-H approach
Experimental Results Scenario 1 → 3 categories {shirts, dresses, socks}
Local Global Both0.00
10.0020.0030.0040.0050.0060.0070.0080.0090.00
100.00
BaselineL-C-HL-C-S-H
Experimental Results Scenario 2 → 7 categories {shirts, dresses, socks, cloths, pants,
shorts, jackets}
Local Global Both0.005.00
10.0015.0020.0025.0030.0035.0040.00
BaselineL-C-HL-C-S-H
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
Cusano-Towner et al. were aimed at flattening a piece of crumpled clothing by implementing a disambiguation phase and a reconfiguration phase.
Previous Work on Unfolding for Robotics
Our system does not use a dual manipulator or a predefined model of the clothing
First PhaseRemove any major wrinkles
and / or folds Pulling the cloth at individual
corners every d degrees
Second PhaseDefine cloth modelCalculate various components
needed for the cloth model
Model to Unfold Laundry into a Flat Canonical Position
Experimental Results The proposed approach was applied to a 3D simulated cloth to
determine the results of the first and second phase.
Two experiments were conducted:
1) Experimental Test of Algorithm
2) Test to Fully Flatten the Cloth
Experimental Test of Algorithm :
This experiment tested the first phase of the proposed algorithm and monitored the process from eight iterations of pulling the cloth.
The models continually change configurations in a manner that flattens and unfolds larger areas of the cloth as the iterations increase.
Eventually, the cloth is mostly flattened out to a more recognizable shape in the final iteration.
Experimental Results
Test to Fully Flatten the Cloth :
This experiment tested the proposed algorithm in determining if this approach would completely flatten a piece of clothing.
The test used the first and second phase of the algorithm to grasp the cloth at various locations and moved the cloth at various orientations until the cloth obtained a flattened percentage greater than 95%.
Experimental Results
Clothing ClassificationB. Willimon, S. Birchfield, and I. Walker, “Classification of clothing using interactive perception,” in ICRA 2011.B. Willimon, I. Walker, and S. Birchfield, “A new approach to clothing classification using mid-level layers,” in ICRA 2013.B. Willimon, I. Walker, and S. Birchfield, “Classification of clothing using midlevel layers,” in ISRN Robotics, 2013.
Unfolding ClothingB. Willimon, S. Birchfield, and I. Walker, “Model for Unfolding Laundry using Interactive Perception,” in IROS 2011.
Pose EstimationB. Willimon, S. Hickson, I. Walker, and S. Birchfield, “An energy minimization approach to 3D non-rigid deformable surface estimation using RGBD data,” in IROS 2012.B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation,” Submitted to Autonomous Robotics (AURO). B. Willimon, I. Walker, and S. Birchfield, “3D non-rigid deformable surface estimation without feature correspondence,” in ICRA 2013.
Research PathFocus of thesis
Previous Work on Pose Estimation for Robotics
• Elbrechter et al. (IROS 2011) use a soft-body-physics model with visual tracking to manipulate a piece of paper.
• Bersch et al. (IROS 2011) describe a method to bring a T-shirt into a desired configuration by alternately grasping the item with two hands, using a fold detection algorithm.
Both approaches require predefined fiducial markers.
The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:
Energy Minimization Approach
Correspondence term
Depth term
Boundary term
Smoothness term
The purpose of this approach is to minimize the energy equation of a mesh model that involves 4 terms:
Energy Minimization Approach
Energy Minimization Approach
Smoothness term
Correspondence term
Depth term
Boundary term
Smoothness term
Energy Minimization Approach
Energy Minimization ApproachSmoothness term
Correspondence term
Energy Minimization Approach
Depth term
Energy Minimization Approach
Front View Top View
Boundary term
Energy Minimization Approach
Energy Minimization ApproachMinimize energy equation
Segmentation and InitializationForeground / background segmentation
Segmentation and InitializationMesh Model Generator
Segmentation and InitializationReinitialization
Experimental Results
• We captured RGBD video sequences of shirts, pants, shorts, and posters to test our proposed method’s ability to handle different non-rigid objects in a variety of scenarios.
Four experiments were conducted:
1) Estimating pose of clothing
2) Estimating pose of posters
3) Reinitializing mesh after in-plane rotation
4) Quantitative Results
Experimental Results
Shirt partially occluded
Shirt moving in-plane and out-of-plane
Shirt changing scale
Shirt translating from side to side
Shirt moving in-plane
• Estimating pose of clothing•7 shirts•1 pair of pants•2 pairs of shorts
Experimental Results
• Estimating pose of posters
Poster with little texture moving out-of-plane
Poster with a lot of texture moving out-of-plane
Experimental Results
• Reinitializing mesh after in-plane rotation
Experimental Results
• Quantitative Results
Conclusion Clothing Classification
Extraction / Isolation A novel approach in which a pile of laundry is sifted by an
autonomous robot system in order to separate each item. Hanging position
Using interaction to provide multiple views of an object and capture more visual data
The results show that, on average, classification rates using robot interaction are 59% higher than those that do not use interaction.
Lying Position Multi-layer approach involving a mixture of global and local
features Characteristics and selection masks achieve, on average, an
improvement of 27.47% for three categories 17.90% for four categories 10.35% for seven categories
Conclusion Unfolding Clothing
An approach to interactive perception in which a piece of laundry is flattened out into a canonical position by pulling at various locations of the cloth.
The algorithm is shown to flatten a simulated cloth by 95.57% of its total area
Pose Estimation A new and novel algorithm that estimates the 3D configuration of a
deformable object through an RGBD video sequence An energy model is used to create a non-linear energy function and the
information is computed using a semi-implicit scheme Energy terms
Smoothness Feature point correspondence Depth Boundary
References
Copies of: Publications Code Datasets Videos
are located at
www.bryanwillimon.com