h a n d t a l k

H A N D T A L K

An Interactive Web-Based Sign Language Modeler

The Proponents . . .

Natalie Rubnan Que

Maria Melissa Mejia

Marjorie Joy Pantig

Jackson Velasco

Overview of Research

HandTalk – translation of sign language coupled with an instant messengerMotion Capture – realistic hand

movementASL (American Sign Language) – widely

used language Limited to fingerspelling and goes back to

open palm position

Overview of Research

Research . . . 3D Graphics on

the WebAnimation

TechniquesWeb Application

Implementation

Quick Demo Here?

Overview of Research

Research Objectives Met Study basic sign language Review sign language related papers , software, theses Study formulas, algorithms about hand modeling Study joints of the hand Study use of motion capture data Investigate animation techniques Study web application issues Apply HCI concepts in system design

Overview of Research

System Objectives Met Display a 3D hand model on a web page in WWW. Provide function to choose which hand Provide functions for modeling transformations on the model Provide a function that would accept user text input Provide a means for the user to choose whom the receiver of

the text input is Provide a function that would parse input and divide into

token stream Provide a function that would accept token stream and

retrieve motion data

Overview of Research

System Objectives Met Provide a function that would enable 3D hand model to

perform fingerspelling motion of user input Provide an option to set speed of translation Provide a means for the user to identify himself in system Provide a means for the user to keep track of his/her contacts

list Provide a means for the user to keep track of the messages

he/she received

Overview of Research

General Objective:

“To research on the design and implementation of an interactive web-based sign language modeler.”

Overview of Proposed System

Three Major Modules:Hand ModelerSign ModelerSign Language Messenger

Overview of Proposed System

Overall Structure of the System

Maybe we could make a Flash movie here that shows how system works during operation

Proposed System…Hand Modeler

Hand Modeler – hand model generationProblem: Which 3D graphics tool?Choices:

VRMLMicrosoft .NET TechnologyMacromedia Director Shockwave

Proposed System…Hand Modeler

Chosen Solution . . . Macromedia Director Shockwave

Wide range of multimedia applications VRML slower and limited Revolution 3D, not for the Web

Proposed System…Hand Modeler

Shockwave Control Sprites Scripts State flags

Proposed System…Hand Modeler

Major Functions:Select Hand ModelRotationCamera ViewAdjust LightingReset All

Hand Modeler Functions

Select Hand Model Right, Left or Both

Hands Right Hand values

negated for Left Hand

Hierarchy of Joints

Hand Modeler Functions

Select Hand ModelProblem # 1: Speed IssuesSolution: Trade between realism and speed

Hand Modeler Functions

Select Hand Model Problem # 2:

Default Orientation of Hand

Solution: Adjustment of Camera

Hand Modeler Functions

RotationRotate about X, Y, Z axisRoot Joint Only10 degrees Rotation of Model Itself, not Camera

Hand Modeler Functions

RotationProblem # 1: Inconsistent hand orientations

due to a combination of rotationsSolution: A list instead of state flags

Hand Modeler Functions

Camera ViewTranslation and ScalingState flagsCamera instead of model

Hand Modeler Functions

Adjust Lighting Brightness and

Contrast Contrast disabled

Hand Modeler Functions

Adjust Lighting Problem: Default Director light Solution: Lighting from scripts

Hand Modeler Functions

Adjust DelayDelay between framesState flag

Hand Modeler Animation

Frame-by-frame Data for specific frame applied Adjust from user controls

Proposed System…Sign Language Modeler

Sign Language Modeler – retrieve data and pass to Hand ModelerThree languages:

JavaJavaScriptShockwave Lingo

Data on client side – minimum network delay

Proposed System…Sign Language Modeler

Sign Language Modeler Problem:

Slow retrieval of data

Solution:

Streaming of data

Sign Language Modeler Functions

Process Text Read user inputJavaScript for speed and EvalscriptValidation in JavaScript, not Java

Sign Language Modeler Functions

Process Motion DataToken = parsing of motion data in BVH fileDigitally signed Java appletDependent on structure of BVH fileString tokenizer: only offset values

(OFFSET part), number of frames, motion data (FRAMES part)

Comma delimiter (,)

Sign Language Modeler Functions

Process Motion Datavariable data: faster tokenizing, shorter

strings to appendData sent back as string of form:

<character translated> <offsets> <number of frames> <motion data>

Sign Language Modeler Functions

Process Motion DataProblem # 1: Different natures of security

certificates in Netscape and IESolution: Two implementations

Problem # 1: No permissionSolution: Netscape asks again, IE need to

restart

Sign Language Modeler Functions

Map Joint Angle DataShockwave LingoOffset and motion data to jointsData from Process Motion Data (Java)

through JavaScript function EvalscriptMaps specific data in every time frame

Sign Language Modeler Functions

Map Joint Angle Datavariable MocapData: data from EvalscriptValues retrieved from MocapData based on

form of stringvariable PassData: current motion data to

be mapped

Sign Language Modeler Functions

Map Joint Angle DataProblem: Shared data conflict due to

concurrencyChoices:

Binary SemaphoresSemaphoresWait-Signal Mechanisms

Solution: Binary Semaphores – simple, yet adequate and effective

Proposed System…Sign Language Messenger

Instant messagingSeparation of major functions instead of

centralization – no bottlenecks

Sign Language Messenger…The Server Program

All the users, all the messagesinstant_messenging class:

client >> web server >> server program

Sign Language Messenger…The Server Program

Keeping Track of Users…client_thread group: dynamic storage, no

waste of time in file accessDifferent functions for interactivity (threads)

client_threadgroup.write_message_to(x, sender + "¿" + recipient + "¿" + msg);

Sign Language Messenger…The Server Program

Keeping Track of Users…Problem: Information transfer between

threadsChoices:

ArraysVector Lists

Solution: Vectors: for distributed computing

Sign Language Messenger…The Server Program

Enabling User Connectionserver_temp_thread: thread-embedded

socket to a clientServer alert from clientNecessary functions firstint x = client_threadgroup.search_for_username(s);

CLIENT: socket = new Socket("server ip", port);

SERVER: server_temp_thread t = new server_temp_thread(new_socket);

t.start();

Sign Language Messenger…The Server Program

Enabling User ConnectionProblem: Checking if logged before

acceptance as userSolution: client_thread_group static, so

accessible

Sign Language Messenger Functions

Create New Accountadd_db: add to dbcheck_existingPrepared statements instead of one-by one

insertion: faster

Sign Language Messenger Functions

LoginDatabase instead of files, fasterProblem: Status of loginSolution:

lok: login ok lusok: password incorrect lnotok: login not ok

Sign Language Messenger Functions

Details Change or retrieve

password Database instead of

files, faster

Sign Language Messenger Functions

Contacts Listcontacts classFile instead of database, too much space

Sign Language Messenger Functions

Message: Read MessagesThreadFormat of message:

<sender> <recipient> <message> “¿”: delimiter

Sign Language Messenger Functions

Message: InTray Files instead of database: avoid redundancy

Sign Language Messenger Functions

Message: Send MessagesUser message restructured into defined

format

Sign Language Messenger Functions

Log-outClose session

Sign Language Messenger Misc. Design Issues

Java Server Pages vs. Java ServletsJSP over ASPJSP same functionality as servletPresentation from logicMixed HTML and code? Java Beans.

Sign Language Messenger Misc. Design Issues

Sessions vs. Cookies Cookies: network traffic, corrupted, disabled Sessions: 30 minute lifetime

Sign Language Messenger Misc. Design Issues

JavaScript vs. VBScript VBScript only for IE JavaScript for all

Sign Language Messenger Misc. Design Issues

Resin vs. Other Servlet Engines Resin: fastest, Java support, auto-compilation

HCI Issues

TextText size 2Readable colorsArial & ImpactFamiliar words

HCI Issues

GraphicsThree-color schemeMinimal amount of images

HCI Issues

Layout and Structure Simple, visible,

orthodox

HCI Issues

Speed and Accessibility Tried-and-tested technologies Graphics minimized Data on client machine

System Features

Testing Results & Observations

Black Box TestingWhite Box TestingGraphics TestingPerformance TestingSurvey

Results based from speed of translation and loading? Don’t know what else could be put here. Doc Bong said not to put results of all functions, so I guess the results of those functions that a user may deem important? (Speed and loading of page?)

Testing Results & Observations

Survey Results:

Testing Results & Observations

Survey Results:

Conclusion and Recommendations

Additional functionalities/featuresAdjust LightingCurrent Letter TrackingStreaming and ConcurrencyUpdate BVH files

Conclusion and Recommendations

RecommendationsAdditional FeaturesOther Applications for Sign Language

Modeler and Hand ModelerEffective Tool for Communication

H A N D T A L K

No animals were harmed during the making of this presentation.