h a n d t a l k

H A N D T A L K

An Interactive Web-Based Sign Language Modeler

The Proponents . . .

Natalie Rubnan Que

Maria Melissa Mejia

Marjorie Joy Pantig

Jackson Velasco

Overview of Research

HandTalk – translation of sign language coupled with an instant messengerMotion Capture – realistic hand

movementASL (American Sign Language) – widely

used language Limited to fingerspelling and goes back to

open palm position


Research . . . 3D Graphics on

the WebAnimation

TechniquesWeb Application

Implementation

Quick Demo Here?


Research Objectives Met Study basic sign language Review sign language related papers , software, theses Study formulas, algorithms about hand modeling Study joints of the hand Study use of motion capture data Investigate animation techniques Study web application issues Apply HCI concepts in system design


System Objectives Met Display a 3D hand model on a web page in WWW. Provide function to choose which hand Provide functions for modeling transformations on the model Provide a function that would accept user text input Provide a means for the user to choose whom the receiver of

the text input is Provide a function that would parse input and divide into

token stream Provide a function that would accept token stream and

retrieve motion data


System Objectives Met Provide a function that would enable 3D hand model to

perform fingerspelling motion of user input Provide an option to set speed of translation Provide a means for the user to identify himself in system Provide a means for the user to keep track of his/her contacts

list Provide a means for the user to keep track of the messages

he/she received


General Objective:

“To research on the design and implementation of an interactive web-based sign language modeler.”

Overview of Proposed System

Three Major Modules:Hand ModelerSign ModelerSign Language Messenger

Overview of Proposed System

Overall Structure of the System

Maybe we could make a Flash movie here that shows how system works during operation

Proposed System…Hand Modeler

Hand Modeler – hand model generationProblem: Which 3D graphics tool?Choices:

VRMLMicrosoft .NET TechnologyMacromedia Director Shockwave


Chosen Solution . . . Macromedia Director Shockwave

Wide range of multimedia applications VRML slower and limited Revolution 3D, not for the Web


Shockwave Control Sprites Scripts State flags


Major Functions:Select Hand ModelRotationCamera ViewAdjust LightingReset All

Hand Modeler Functions

Select Hand Model Right, Left or Both

Hands Right Hand values

negated for Left Hand

Hierarchy of Joints


Select Hand ModelProblem # 1: Speed IssuesSolution: Trade between realism and speed


Select Hand Model Problem # 2:

Default Orientation of Hand

Solution: Adjustment of Camera


RotationRotate about X, Y, Z axisRoot Joint Only10 degrees Rotation of Model Itself, not Camera


RotationProblem # 1: Inconsistent hand orientations

due to a combination of rotationsSolution: A list instead of state flags


Camera ViewTranslation and ScalingState flagsCamera instead of model


Adjust Lighting Brightness and

Contrast Contrast disabled


Adjust Lighting Problem: Default Director light Solution: Lighting from scripts


Adjust DelayDelay between framesState flag

Hand Modeler Animation

Frame-by-frame Data for specific frame applied Adjust from user controls

Proposed System…Sign Language Modeler

Sign Language Modeler – retrieve data and pass to Hand ModelerThree languages:

JavaJavaScriptShockwave Lingo

Data on client side – minimum network delay

Proposed System…Sign Language Modeler

Sign Language Modeler Problem:

Slow retrieval of data

Solution:

Streaming of data

Sign Language Modeler Functions

Process Text Read user inputJavaScript for speed and EvalscriptValidation in JavaScript, not Java


Process Motion DataToken = parsing of motion data in BVH fileDigitally signed Java appletDependent on structure of BVH fileString tokenizer: only offset values

(OFFSET part), number of frames, motion data (FRAMES part)

Comma delimiter (,)


Process Motion Datavariable data: faster tokenizing, shorter

strings to appendData sent back as string of form:

<character translated> <offsets> <number of frames> <motion data>


Process Motion DataProblem # 1: Different natures of security

certificates in Netscape and IESolution: Two implementations

Problem # 1: No permissionSolution: Netscape asks again, IE need to

restart


Map Joint Angle DataShockwave LingoOffset and motion data to jointsData from Process Motion Data (Java)

through JavaScript function EvalscriptMaps specific data in every time frame


Map Joint Angle Datavariable MocapData: data from EvalscriptValues retrieved from MocapData based on

form of stringvariable PassData: current motion data to

be mapped


Map Joint Angle DataProblem: Shared data conflict due to

concurrencyChoices:

Binary SemaphoresSemaphoresWait-Signal Mechanisms

Solution: Binary Semaphores – simple, yet adequate and effective

Proposed System…Sign Language Messenger

Instant messagingSeparation of major functions instead of

centralization – no bottlenecks

Sign Language Messenger…The Server Program

All the users, all the messagesinstant_messenging class:

client >> web server >> server program


Keeping Track of Users…client_thread group: dynamic storage, no

waste of time in file accessDifferent functions for interactivity (threads)

client_threadgroup.write_message_to(x, sender + "¿" + recipient + "¿" + msg);


Keeping Track of Users…Problem: Information transfer between

threadsChoices:

ArraysVector Lists

Solution: Vectors: for distributed computing


Enabling User Connectionserver_temp_thread: thread-embedded

socket to a clientServer alert from clientNecessary functions firstint x = client_threadgroup.search_for_username(s);

CLIENT: socket = new Socket("server ip", port);

SERVER: server_temp_thread t = new server_temp_thread(new_socket);

t.start();


Enabling User ConnectionProblem: Checking if logged before

acceptance as userSolution: client_thread_group static, so

accessible

Sign Language Messenger Functions

Create New Accountadd_db: add to dbcheck_existingPrepared statements instead of one-by one

insertion: faster


LoginDatabase instead of files, fasterProblem: Status of loginSolution:

lok: login ok lusok: password incorrect lnotok: login not ok


Details Change or retrieve

password Database instead of

files, faster


Contacts Listcontacts classFile instead of database, too much space


Message: Read MessagesThreadFormat of message:

<sender> <recipient> <message> “¿”: delimiter


Message: InTray Files instead of database: avoid redundancy


Message: Send MessagesUser message restructured into defined

format


Log-outClose session

Sign Language Messenger Misc. Design Issues

Java Server Pages vs. Java ServletsJSP over ASPJSP same functionality as servletPresentation from logicMixed HTML and code? Java Beans.


Sessions vs. Cookies Cookies: network traffic, corrupted, disabled Sessions: 30 minute lifetime


JavaScript vs. VBScript VBScript only for IE JavaScript for all


Resin vs. Other Servlet Engines Resin: fastest, Java support, auto-compilation

HCI Issues

TextText size 2Readable colorsArial & ImpactFamiliar words

HCI Issues

GraphicsThree-color schemeMinimal amount of images

HCI Issues

Layout and Structure Simple, visible,

orthodox

HCI Issues

Speed and Accessibility Tried-and-tested technologies Graphics minimized Data on client machine

System Features

Testing Results & Observations

Black Box TestingWhite Box TestingGraphics TestingPerformance TestingSurvey

Results based from speed of translation and loading? Don’t know what else could be put here. Doc Bong said not to put results of all functions, so I guess the results of those functions that a user may deem important? (Speed and loading of page?)

Testing Results & Observations

Survey Results:

Conclusion and Recommendations

Additional functionalities/featuresAdjust LightingCurrent Letter TrackingStreaming and ConcurrencyUpdate BVH files

Conclusion and Recommendations

RecommendationsAdditional FeaturesOther Applications for Sign Language

Modeler and Hand ModelerEffective Tool for Communication

H A N D T A L K

No animals were harmed during the making of this presentation.

h a n d t a l k

Documents