a new writing experience : finger writing in the air using a kinect sensor xin zhang, zhichao ye,...
TRANSCRIPT
A New Writing Experience :Finger Writing in the Air Using a Kinect SensorXin Zhang, Zhichao Ye, Lianwen Jin,
Ziyong Feng, and Shaojie Xu
MultiMedia, IEEE, 2013
FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR
Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu
IEEE International Conference onMultimedia and Expo Workshops (ICMEW), 2013
4
Introduction• So far most of writing systems still rely on:
• Keyboard
• Touch screen
• …(Extra devices)
• Essential goal of HCI:
• Making interaction between user and computer more natural
5
Introduction• In this paper:
• Propose a finger-writing-in-the-air system (based on Kinect):
• Using depth, color and motion information
• Real-time
• User-friendly and unconstrained
Related work• Hand Segmentation
• Skin color:• Gaussian (mixture) model[2]
• Illumination and hand-face overlapping
• Depth:• noise
• Motion:• Motion Cue[3]
• The hand should be the most distinct moving object.
X
X
X
7
Related work• Fingertip Detection
• Curvature[6]
• Template matching[1]
• Geodesic distance
[1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007.
[2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005.
[3] Jonathan Alon, Vassilis Athitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009.
[6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011.
8
Related work• Writing-in-the-air system [10]:
Hand Segmentation
Data Conversion
Region Clustering
Fingertip Identification
Arm point
Fingertip
K-means
[10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012.
9
13
• Depth Model
• Solve the issues:• lighting • hand-face overlapping• moving background
• Hand D:
Hand Segmentation
R(n) : hand region at frame nω : : growth factor
depth
↑
↑
15
• Skin Model
• YCbCr color space
• Quantify Y Component into three regions:
• Bright
• Normal
• Dark
• Gaussian classifier[2]:
Hand Segmentation
Reduce the storage size
skin
Non-skin
: mean vector of the i-th skin class covariance of the i-th skin class mean vector of the i-th non-skin covariance of the i-th non-skin class
(Squared Mahalanobis distance)
17
• Background Model• Codebook background model[8]
•
Hand Segmentation[8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005.
18
• Background Model• Codebook background model[8]
•
Hand Segmentation
Color image A Color image BForeground result A Foreground result B
20
• DSB-MM segmentation algorithm
• Each model should have different reliabilities.
• Adaptive voting system
• A pixel is kept as hand pixel by
Hand Segmentation
21
• Artificial Neural Network (ANN) • (1) All the models contribute to the final result.• (2) None of them is absolutely reliable.
Hand Segmentation
“1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3
Training:resilient back propagation algorithm (RPROP)
25
• Side-mode
• Fingertip : the farthest point from the arm point
• Palm point: • Ellipse fitting technique (center point)
• Arm point: • The center of the increased region
Fingertip Detection
29
Experimental Results• Intel Core i5-2400 CPU
• 3.10 GHz and 4 Gbytes of RAM
• 20 frames per second(fps)
• 375 videos(44522 frames)
• Recognition of the classifier:
• 6763 frequently used Chinese character
• 26 English letters (upper case & lower case)
• 10 digits
30
Experimental Results• Finger-writing character recognition
• Linking all detected fingertip positions + mean filter
• Modified quadratic discriminant function (MQDF) character classifier[9]
[9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9):2916-2926, 2008.