byron hood | version 0.4 computer systems lab project2007-2008 sign language recognition using...

Byron Hood | version 0.4Computer Systems Lab Project2007-2008

Sign Language Sign Language Recognition using Recognition using

WebcamsWebcams


OverviewOverview Average person’s typing speed

Composing: ~19 words per minute Transcribing: ~33 words per minute

Sign speaker Full sign language: ~200 words per minute Spelling out: estimate: 50 words per minute Up to 3x faster


Purpose and ScopePurpose and Scope Native signers can input faster Benefits:

Hearing & speaking disabled Sign interpreters

Just letters & numbers for now Additional complexity too much to handle Would require smaller distinctions


ResearchResearch Related projects

Using mechanical gloves, colored gloves Tracking body parts Neural network-based application

Still images: 92% accuracy Motion: less than 50% accuracy

Feature vector-based application Also about 90% accuracy on stills No motion tests


More ResearchMore Research Image techniques:

Edge detection (Robert’s Cross) Line detection (Hough transform) Line interpretation methods

Chaining groups of lines Macro-scale templates

Residual math Memory management


Testing ModelTesting Model Human interaction necessary General testing model:

bhood@testing: ~/syslab-tech $ \> ./main images/hand.png

[DEBUG] Edge detect time: 29 ms

Errors: 0 Warnings: 0


Program ArchitectureProgram Architecture

Webcam capture

Edge detection

Line detection

Interpretation

Attribute matching

IMAGEIMAGE

FEATURE FEATURE OUTLINEOUTLINE

LINE LISTLINE LIST

FINGER FINGER POSITONSPOSITONS

SERVER PROCESSSERVER PROCESS


Edge Detection ResultsEdge Detection Results Results:

Outlines the important edges and not much besides

Robert’s Cross balances detection of major and minor lines

Original image(800 x 703)

Final image(800 x 703)


Cropping ResultsCropping Results

Original(800 x 703)

Result(633 x 645)

Remove useless rows & columns with no features

Better contrast Very large

optimization Memory savings Area difference

means order n2


Line detectionLine detection Finished!

Recently finished tweaking sensitivities

Still a few potential memory issues


Line GroupingLine Grouping Part of line

detection Large

optimization Iterate over an

order of magnitude fewer items

Easier to handle, more pronounced trends

Examples of line Examples of line groups, called “chains”groups, called “chains”


Line InterpretationLine Interpretation Chaining groups of lines Templates

Generation Template-based comparison

Line residuals Use point coordinate averages Calculate average offset from average Easy to find height of finger


Sample OutputSample OutputAfter a typical run:10 days, 6:12:19 until graduation!!bhood@testing ~/syslab-tech/src $ ./main hand.pngEdge detection took 0.04 secImage cropping took 0.00 secLine detection took 0.17 sec (detected 1424 lines)Line chaining took 0.25 sec (detected 130 chains)Getting orientation took -0.08 src (1 => ORIENTATION_FORWARD)Getting pinky pos. took 0.00 sec (2 => FINGER_BENT)Getting ring pos. took 0.00 sec (2 => FINGER_BENT)Getting middle pos. took 0.01 sec (2 => FINGER_BENT)Getting index pos. took -0.00 sec (4 => FINGER_TUCKED)Overall process took 0.47 sec[TOTALCOUNT] allocated: 10718901, freed: 10364880; leaked: 354021.


TimingTiming Timing data from runs:

To nearest hundredth of a second(0)Edge detection: 0.04 sec(1)Image cropping: 0.00 sec(2)Line detection: 0.17 sec(3)Line chaining: 0.25 sec(4)orientation: 0.08 sec(5)Pinky finger: 0.00 sec(6)Ring finger: 0.00 sec(7)Middle finger: 0.01 sec(8)Index finger: 0.00 sec(9)Overall process: 0.47 sec

A little slow considering goal of real-time


The Mysterious FutureThe Mysterious Future Perfect line interpretation

Work on memory management Am leaking large quantities (~50K) of memory Aggressive profiling needed

Finish camera-computer interaction Device control must be precise, picky


The End!The End! Code will be available to future years

Contact me for a copy:[email protected]

byron hood | version 0.4 computer systems lab project2007-2008 sign language recognition using...

Documents