![Page 1: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/1.jpg)
ASL2TXT
Converting sign language gestures from digital images to text
George Corser
![Page 2: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/2.jpg)
Presentation Overview
• Concept• Foundation: Barkoky & Charkari (2011)– Segmentation – Thinning
• My Contribution: Corser (2012)– Segmentation (similar to Barkoky)– CED: Canny Edge Dilation (Minus Errors)– Assumption: User trains his own phone
![Page 3: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/3.jpg)
Concept
• Deaf and hearing people talking on the phone, each using their natural language
• Sign-activated commands like voice-activated
![Page 4: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/4.jpg)
Situation: Drive Thru Window
1. Deaf person signs order2. Phone speaks order3. Confirmation on screen
Think:Stephen Hawking
![Page 5: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/5.jpg)
Process Flow
• Requires several conversion processes• Many have been accomplished• Remaining: ASL2TXT
![Page 6: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/6.jpg)
Goal: Find an Algorithm
• Find an image processing algorithm that recognizes ASL alphabet
= AWeb site
![Page 7: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/7.jpg)
Barkoky: Segmentation & Thinning
Barkoky countsendpoints to
determine sign(doesn’t work for ASL)
![Page 8: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/8.jpg)
Barkoky ProcessSegmentation1. Capture RGB image2. Rescale3. Extract using colors4. Reduce noise5. Crop at wrist6. Result: hand segment
Thinning7. Input: hand segment8. Apply thinning9. Find endpoints, joints10. Calculate lengths11. Clean short lengths12. Identify gesture by
counting endpoints
![Page 9: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/9.jpg)
1. Capture RGB Image2. Rescale
% ---------- 1. Capture RGB imagea = imread('DSC04926.JPG');figure('Name','RGB image'),imshow(a);
% ---------- 2. Rescale image to 205x154a10 = imresize(a, 0.1);figure('Name','Rescaled image'),imshow(a10);
![Page 10: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/10.jpg)
3. Extract Hand Using Colors
% ---------- 3. Extract hand using colorabw10 = zeros(205,154,1);for i=1:205, for j=1:154, if a10(i,j,2)<140 && a10(i,j,3)<100,
abw10(i,j,1)=255; end; end; end;figure('Name','Extracted'),imshow(abw10);
Note: Color threshold codediffers from Barkoky
![Page 11: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/11.jpg)
Colors: Training Set Histograms
![Page 12: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/12.jpg)
Colors: Training Set (2)
Excel
Red Green Blue
![Page 13: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/13.jpg)
Colors: Test Set Histograms
![Page 14: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/14.jpg)
4. Reduce Noise
% ---------- 4. Reduce noisefor i=2:204, for j=1:154, if abw10(i-1,j,1)==0
if abw10(i+1,j,1)==0, abw10(i,j,1)=0; end; end;
if abw10(i-1,j,1)==255 if abw10(i+1,j,1)==255,
abw10(i,j,1)=0; end; end;end; end;abw10 = imfill(abw10,'holes');
![Page 15: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/15.jpg)
5. Identify Wrist Position
% ---------- 5. Identify wrist positionfor i=204:-1:1, for j=1:154,
if abw10(i,j,1)==255, break; end; end; if j ~= 154 && abw10(i+1,j,1)~=255, wristi=i+1; wristj=j+1; break; end; end;
![Page 16: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/16.jpg)
Wrist Detection
• Algorithm searches bottom-to-top of image • Finds a leftmost white pixel above black pixel• Sets wrist position SE of found white pixel
![Page 17: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/17.jpg)
Corser: Segmentation & CED
• Segmentation (similar to Barkoky)– Color threshold technique slightly different– American Sign Language (ASL) alphabet, not
Persian Sign Language (PSL) numbers• Image Comparison: Tried Several Methods– Full Threshold (Minus Errors)– Diced Segments (Minus Errors)– Endpoint Count Difference– CED: Canny Edge Dilation
![Page 18: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/18.jpg)
ASL Training Set
Hit-or-miss: 23% Barkoky: 8%
![Page 19: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/19.jpg)
ASL Test Set
MATLAB
![Page 20: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/20.jpg)
A
![Page 21: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/21.jpg)
A
![Page 22: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/22.jpg)
B
![Page 23: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/23.jpg)
B
![Page 24: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/24.jpg)
C
![Page 25: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/25.jpg)
C
![Page 26: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/26.jpg)
D
![Page 27: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/27.jpg)
D
![Page 28: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/28.jpg)
E
![Page 29: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/29.jpg)
E
![Page 30: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/30.jpg)
F
![Page 31: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/31.jpg)
F
![Page 32: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/32.jpg)
G
![Page 33: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/33.jpg)
G
![Page 34: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/34.jpg)
H
![Page 35: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/35.jpg)
H
![Page 36: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/36.jpg)
I
![Page 37: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/37.jpg)
I
![Page 38: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/38.jpg)
J
![Page 39: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/39.jpg)
J
![Page 40: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/40.jpg)
K
![Page 41: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/41.jpg)
K
![Page 42: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/42.jpg)
L
![Page 43: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/43.jpg)
L
![Page 44: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/44.jpg)
M
![Page 45: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/45.jpg)
M
![Page 46: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/46.jpg)
N
![Page 47: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/47.jpg)
N
![Page 48: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/48.jpg)
O
![Page 49: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/49.jpg)
O
![Page 50: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/50.jpg)
P
![Page 51: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/51.jpg)
P
![Page 52: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/52.jpg)
Q
![Page 53: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/53.jpg)
Q
![Page 54: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/54.jpg)
R
![Page 55: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/55.jpg)
R
![Page 56: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/56.jpg)
S
![Page 57: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/57.jpg)
S
![Page 58: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/58.jpg)
T
![Page 59: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/59.jpg)
T
![Page 60: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/60.jpg)
U
![Page 61: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/61.jpg)
U
![Page 62: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/62.jpg)
V
![Page 63: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/63.jpg)
V
![Page 64: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/64.jpg)
W
![Page 65: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/65.jpg)
W
![Page 66: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/66.jpg)
X
![Page 67: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/67.jpg)
X
![Page 68: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/68.jpg)
Y
![Page 69: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/69.jpg)
Y
![Page 70: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/70.jpg)
Z
![Page 71: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/71.jpg)
Z
![Page 72: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/72.jpg)
Z
![Page 73: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/73.jpg)
Hybrid Algorithm Example
% ---------- MATLAB Code -------------------matchtotal = 0;if abs(x10range - x20range) < 20, matchtotal = matchtotal + 10;end;if abs(y10range - y20range) < 20, matchtotal = matchtotal + 11;end;matchtotal = matchtotal - abs(h10 - h20);% ----- h10, h20 are vector magnitudes -----
![Page 74: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/74.jpg)
Erosion Subtraction
![Page 75: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/75.jpg)
Canny Edge
![Page 76: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/76.jpg)
Canny Edge Dilation Code
% ---------- MATLAB Code -------------------
se = strel('disk',5);a10 = edge(a10,'canny');a20 = edge(a20,'canny');a10 = imdilate(a10,se);a20 = imdilate(a20,se);
% ----- Then calculate matches minus errors
![Page 77: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/77.jpg)
Experimental Results
Technique CorrectFull Threshold (Minus Errors) 19% (27%)Diced Segments (Minus Errors) 23% (27%)Barkoky Endpoint Count Diff. 8%Hybrid - Height/Width/Endpoints 19%Erosion Subtraction 15%Canny Edge Dilation (Minus Errors) 12% (35%)
![Page 78: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/78.jpg)
Disadvantages
• Dependent on lighting conditions• Fails with flesh-tone backgrounds• Requires calibration to a specific user• Limited applications: text messaging,
activation (“sign” similar to voice activation)• ASL numbers (A=10, D=1, O=0, V=2, W=6)• Alphabet is tiny portion of full translation:
complete translation maybe many years away
![Page 79: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/79.jpg)
Future Work
• Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme)
• Could write letter-by-letter algorithm• Could use range camera to compute distance
of finger instead of shape of hand• Motion analysis or edge count• Many possibilities… we’ve only just begun!
Cue: music http://www.youtube.com/watch?v=__VQX2Xn7tI
![Page 80: ASL2TXT Converting sign language gestures from digital images to text George Corser](https://reader038.vdocuments.us/reader038/viewer/2022102818/56649cc55503460f9498ee31/html5/thumbnails/80.jpg)
The End