Detection, Segmentation, and Pose Recognition of
Hands in Images
byChristopher Schwarz
Thesis Chair: Dr. Niels da Vitoria Lobo
Outline• Introduction• Detection and Segmentation
– Line Finding– Curve Finding– Detection– Grouping– Results
• Pose Recognition– Preprocessing– Matching– Results
• Discussions and Conclusions
Introduction
• Hands present an exciting challenge for Computer Vision researchers.– Foils traditional object detection due to nonrigidity and
21 DoF
• Uses:– Surveillance applications:
• Gang signs, obscene gestures, drawing of a weapon
– Human-Computer Interaction• Alternative input devices, motion capture, augmented reality.
Terminology
• Detection: Find presence of target • Segmentation: Separate known target from
background• Pose Recognition: Determine what pose or posture
a hand is in.
Related Work
• Huang [2000]• Athitsos and Sclaroff [2003]• Kölsch and Turk [2004]• Baris Caglar [2005]
Detection and Segmentation Outline
• High-resolution images
• Monochromatic images
• Straight fingers
• Open fingers
Part 1: Detection and Segmentation
Input Image
Generate Line
Sketch
Find Curves
Find Candidate
Fingers
Group and Revisit
Line Sketch Image
• Use a Customized Line Finder– Modified Burns
– Replace line combination with iterative method
– Add a “cost of fit” measure per line
• Union results of running Line Finder over 5 varying inputs to obtain Line Sketch– 4 varying scale
– 1 “Double Canny” input• Large-gaussian Canny over output of small-gaussian canny to
divide textured regions from untextured regions
Part 1: Detection and Segmentation
Line Finder
• Iterative Joining of Lines– Find line segments
– Find nearby, almost-parallel line pairs
– If pair meets thresholds, combine them– Rejoins lines split from angle thresholds or gaps in the
edge input.
Part 1: Detection and Segmentation
Line Finder
• Cost of Fit Measure output with each line– Cost of fitting line model to underlying data
These lines will have a higher Cost of Fit
Part 1: Detection and Segmentation
Line Sketch
Input image
Unioned lines of length >= 15
Unioned Components:
Part 1: Detection and Segmentation
Blur 0 Blur 1 Blur 2 Half-Size Double Canny
Curve Finder• Second input to algorithm• Discovers curves that may represent
fingertips• See Jan Prokaj’s thesis:
Scale Space Based Grammar for Hand Detection• Model:
Part 1: Detection and Segmentation
FingerFinder Pseudocode
For each pair of lines
if pair meets criteria
for all curves nearby curvesif curve meets criteria
add fingerCandidate
Part 1: Detection and Segmentation
Finger Candidate Criteria
• “Finger Score” based on empirically found thresholds
• Criteria– Geometric
– Other
Part 1: Detection and Segmentation
Geometric Criteria
• 11 tests measuring how well a line pair and a curve approximates target configuration:
Part 1: Detection and Segmentation
Non-Geometric Criteria
• Line Inaccuracy: Measure of line curvature found during line finding
• Canny Density: Amount of edge pixels detected in area.
Variance in Canny Density: Sparse finger regions against cluttered background
Part 1: Detection and Segmentation
Grouping Candidate Fingers
1. Find finger groups possibly within the same hand using:
– Locations, using Euclidian distance
– Region intensities, comparing median values
2. Revisit weaker candidates to reinstate if supported by neighbors
Part 1: Detection and Segmentation
Results
First row: Input images
Second row: "Strong" candidates before grouping
Third row: Detected fingers, including those re-added during grouping
Part 1: Detection and Segmentation
Grouping Result Breakdown• Results show detections from all groups • Often, individual groups divide false from true
positives
Part 1: Detection and Segmentation
Pose Recognition Goals
Segmentation-based method using a database
and an input contour
Part 2: Pose Recognition
Assumes:
• High-resolution
• Open fingers
Preprocessing
1. Erode
2. Dilate
3. Compare with the original to find protrusions.
Input contour silhouette
Part 2: Pose Recognition
Preprocessing is identical for the test and every database image.
Preprocessing
4. Ignore tiny protrusions as palm
5. Remove palm
6. Use K-Means clustering to find center of palm from wrist-palm segment
7. Count “finger” segments and find average direction
Part 2: Pose Recognition
Preprocessing Examples
• Matching takes test and set of database images processed in this way
Part 2: Pose Recognition
Matching Phase Overview
• Chamfer Distance• Segment-Based Matching
Part 2: Pose Recognition
Matching via sum of two distance measures:
Chamfer Distance
• Numerical similarity between edge images• For each point in X, find nearest point in Y• The average is the chamfer distance
XxYy
yxX
YXc min1
),(
Part 2: Pose Recognition
Chamfer Distance Direction
c(X,Y) != c(Y,X)X Y
c(X,Y) < c(Y,X)
“Undirected” Chamfer = c(X,Y) + c(Y,X)
Part 2: Pose Recognition
Segment Based Matching:Overview
• Generate CODAA Vector for every pair of test segment and model segment.
• Vector contains five segment comparators• Rank comparator vectors• Rank database images with sum of comparator
rankings
Part 2: Pose Recognition
Segment Based Matching:CODAA Vectors
C Chamfer distance between contours
O Difference between orientations
D Distance between centers
A Difference in size
A Difference in angle relative to palm center
Part 2: Pose Recognition
Segment Based Matching
1. Score each CODAA vector via progressive thresholds of the five values.
2. Rank vectors according to scores
3. For each model image segment, find match in test image with highest score
4. For each segment in test image, find match in model image with highest score
5. Sum “forward” and “reverse” measures
6. Divide by number of fingers
7. Rank model images by score
Part 2: Pose Recognition
Combination
• Combine results of Chamfer Distance and SBM by summing the Log (base 2) of a model’s rank in each measure.
• Rank models by this combined score• Filter known-incorrect models:
– Incorrect finger count
– Incorrect average finger angle
Part 2: Pose Recognition
Video Test ResultsUse video frames as a "database," to find
ones matching an input pose
Part 2: Pose Recognition
Publications
• Segment-Based Hand Pose Estimation. In IEEE CRV 2005.
• Hand Detection and Segmentation for Pose Recognition in Monochromatic Images. In progress.
• Line Sketch. To be written.
Future Work
• Develop and test bridge between segmentation and recognition algorithms
• Feasible to convert finger candidate regions into framework of SBM
• Results improved if palm center can be reliably located