visicast 2002 technical audit 4 october 2002, brussels michele wakefield - project manager, itc

ViSiCAST 2002 Technical AuditViSiCAST 2002 Technical Audit

4 October 2002, Brussels4 October 2002, BrusselsMichele Wakefield - Project Manager, ITCMichele Wakefield - Project Manager, ITC

The ViSiCAST ProjectThe ViSiCAST Project

ViVirtual rtual SiSigning gning CCapture apture AAnimation nimation SStorage and torage and TTransmissionransmission

Aims of ViSiCAST ProjectAims of ViSiCAST Project

“…“…support improved access by deaf citizens to support improved access by deaf citizens to information and services in sign language”information and services in sign language”

by successfully developing signing systems for by successfully developing signing systems for

broadcast, WWW & ‘over the counter’ type applicationsbroadcast, WWW & ‘over the counter’ type applications

user friendly methods to capture & generate signsuser friendly methods to capture & generate signs

machine readable system to describe gestures machine readable system to describe gestures

... preferred medium is sign language... preferred medium is sign language

Independent Television

CommissionTelevirtual

University of East Anglia

The Post Office

Royal National Institute for Deaf People

Instituutvoor Doven

Hamburg University

Institut für Rundfunktechnik

Institut National des Télécommunications

ViSiCAST ConsortiumViSiCAST Consortium

Project DimensionsProject Dimensions

DurationDuration StartStart : January 2000: January 2000 FinishFinish : December 2002: December 2002 36 months36 months

Total Costs Total Costs 3770kECU total 3770kECU total 2876kECU funding from EC2876kECU funding from EC

ViSiCAST Project HighlightsViSiCAST Project Highlights

Signing transmissions demonstrated at IBC 2002Signing transmissions demonstrated at IBC 2002 MPEG-4 compliant INT-IRT demonstrator to deliver an open signing MPEG-4 compliant INT-IRT demonstrator to deliver an open signing

service for broadcast DTVservice for broadcast DTV

BBC demonstrator to deliver closed DTV signing service BBC demonstrator to deliver closed DTV signing service Translate simple sentences in real time to sign animationTranslate simple sentences in real time to sign animation WWW Weather-forecaster launched in the NetherlandsWWW Weather-forecaster launched in the Netherlands Interactive sign language learning tool Interactive sign language learning tool 2nd trial of TESSA system now nationwide and RNID re-2nd trial of TESSA system now nationwide and RNID re-

promoting ViSiCAST promoting ViSiCAST after success of pilot at Science Museum, Londonafter success of pilot at Science Museum, London encouraging national media coverageencouraging national media coverage

Internet Community Broadcast

Evaluation

Exploitation

Animation Linguistics

ViSiCAST Project StructureViSiCAST Project Structure

Technology

User Application

Exploitation &Dissemination

Presentations by Core StreamsPresentations by Core Streams

Technology: Animation & LinguisticsTechnology: Animation & Linguistics WP4 AnimationWP4 Animation WP5 LinguisticsWP5 Linguistics

User: Applications User: Applications WP2 Sign TutorWP2 Sign Tutor

WP1 BroadcastWP1 Broadcast

WP2 WWWWP2 WWW

WP3 Face to FaceWP3 Face to Face

WP6 UsabilityWP6 Usability

Exploitation & DisseminationExploitation & Dissemination

Presentation by Streams - Presentation by Streams - AnimationAnimation

WP 4 AnimationWP 4 Animation Increased realism in sign generationIncreased realism in sign generation Enhanced signing experienceEnhanced signing experience

WP5 Sign Language LinguisticsWP5 Sign Language Linguistics Use of natural sign languageUse of natural sign language Synthesis of sign language gesturesSynthesis of sign language gestures

Animation Work: ObjectivesAnimation Work: Objectives

WP4:WP4: Develop Hi-Resolution Avatars + related capture, Develop Hi-Resolution Avatars + related capture,

and animationand animation To enable and support application development To enable and support application development

in WPs 1-2-3 using WP4 (& WP5) Productin WPs 1-2-3 using WP4 (& WP5) Product To further develop, compare and integrate both To further develop, compare and integrate both

proprietary and standard solutions, where proprietary and standard solutions, where appropriate, in networked environmentsappropriate, in networked environments

Technology: WP4, AnimationTechnology: WP4, Animation

At start of Year:At start of Year:

Visia 2Visia 2

Running in Mask 1Running in Mask 1 Using Motion Capture Using Motion Capture

Data onlyData only Reasonable Reasonable

animation, expression animation, expression etc.etc.


Visia 2 in MPEG-4Visia 2 in MPEG-4 Mesh partitioned intoMesh partitioned into

anatomical segmentsanatomical segments MPEG-4 compliant MPEG-4 compliant

authoring toolauthoring tool Animation editing tool Animation editing tool Server-client tool for TX of Server-client tool for TX of

animation parametersanimation parameters MPEG-4 SNHC playerMPEG-4 SNHC player

<25fps<25fps Embedded within an Embedded within an

MPEG-4 set-top boxMPEG-4 set-top box


Visia 3Visia 3 Updated Virtual HumanUpdated Virtual Human Higher resolution & Higher resolution &

polygon count, more polygon count, more realistic photographic realistic photographic textures textures

Improved articulationImproved articulation Mesh distortion applied to Mesh distortion applied to

garmentsgarments Facial expression via Facial expression via

skeleton manipulation & skeleton manipulation & morphsmorphs

Speech EnabledSpeech Enabled


Visia 3Visia 3

New host software - Mask New host software - Mask TNGTNG

Writing new Active X Writing new Active X ControlsControls

Superior functionality, Superior functionality, lighting and Camera FX, lighting and Camera FX, image quality, frame rate, image quality, frame rate, flexibility etc.flexibility etc.

>75 FPS>75 FPS


Visia 3Visia 3

Running in Mask Running in Mask TNG GraphTNG Graph


Facial MorphsFacial Morphs

Created in Maya, exported to Created in Maya, exported to Mask TNGMask TNG

Based on Sign Language Based on Sign Language expressions (BSL Dictionary)expressions (BSL Dictionary)

Inter-operable Inter-operable Variable weighting (<100%+)Variable weighting (<100%+) May be used with Mo-Cap May be used with Mo-Cap

data or for synthetic signdata or for synthetic sign


Facial Animation - Facial Animation - Experimental WorkExperimental Work

Tracking of Active Tracking of Active Shape ModelsShape Models

Tracking of Active Tracking of Active Appearance ModelsAppearance Models


Facial Animation - Facial Animation - Experimental WorkExperimental Work

Vision-based motion Vision-based motion capture of facial capture of facial

expressions using expressions using MPEG-4 compliant MPEG-4 compliant

templatestemplates..

WP4: Synthetic Animation - WP4: Synthetic Animation - IntroductionIntroduction

Task:Task: Make avatar do signing syntheticallyMake avatar do signing synthetically as specified by ViSiCAST’s Signing Gesture Markup as specified by ViSiCAST’s Signing Gesture Markup

Language - SiGMLLanguage - SiGML

Motive:Motive: Synthetic animation is more flexible than animation via Synthetic animation is more flexible than animation via

motion-capture - “just write some more SiGML”motion-capture - “just write some more SiGML”

Support Natural-Language-to-Animation strategy of WP4-5Support Natural-Language-to-Animation strategy of WP4-5

In broadcasting applications: put synthetic player on In broadcasting applications: put synthetic player on

receiver and transmit SiGML - receiver and transmit SiGML - veryvery low bandwidth low bandwidth

WP4: Synthetic Animation - WP4: Synthetic Animation - ContextContext

Televirtual Avatar is a deformable textured Televirtual Avatar is a deformable textured MeshMesh

Mesh shape and position are determined by Mesh shape and position are determined by configuration of underlying configuration of underlying SkeletonSkeleton skeleton configuration: a.k.a. “Bone-Set”skeleton configuration: a.k.a. “Bone-Set”

To animate avatar: need to generate stream of Bone-To animate avatar: need to generate stream of Bone-Sets - one per frame of animationSets - one per frame of animation i.e. BAF data streami.e. BAF data stream - BAF = “Bones Animation Format”- BAF = “Bones Animation Format” Data intensive: 4Kb per bone-setData intensive: 4Kb per bone-set

WP4: Synthetic Animation - WP4: Synthetic Animation - Technical ApproachTechnical Approach

SiGML specifies gestures through:SiGML specifies gestures through: Postures:Postures:

hand shapehand shape hand orientation - palm and extended finger directionhand orientation - palm and extended finger direction position of hand(s) in signing spaceposition of hand(s) in signing space

Motions - straight-line, circular, zig-zag etc.Motions - straight-line, circular, zig-zag etc.

Synthetic Animation Engine:Synthetic Animation Engine: specifies hand bone configuration for given posturespecifies hand bone configuration for given posture configures arm/shoulder bones using Inverse Kinematicsconfigures arm/shoulder bones using Inverse Kinematics implements transition from one posture to next using non-implements transition from one posture to next using non-

linear interpolation - often via linear interpolation - often via control systemcontrol system modelling modelling

WP4: Synthetic Animation - WP4: Synthetic Animation - Progress (i)Progress (i)

Initial Prototype (D4-2) delivered 2001-12Initial Prototype (D4-2) delivered 2001-12 Supported most of manual SiGMLSupported most of manual SiGML Implemented in Perl (interpreted scripting language)Implemented in Perl (interpreted scripting language) BAF/VRML output to file - and then to avatarBAF/VRML output to file - and then to avatar Relatively slow - often < 15 fpsRelatively slow - often < 15 fps Perl module packaged as ActiveX controlPerl module packaged as ActiveX control

relatively unwieldy architecturerelatively unwieldy architecture

Enhancements for 2002-02 (M5-11)Enhancements for 2002-02 (M5-11) BAF data stream cached in memory-fed directly to avatarBAF data stream cached in memory-fed directly to avatar Front-end(for WP5): HamNoSys input server, with built-in Front-end(for WP5): HamNoSys input server, with built-in

HamNoSys-to-SiGML translationHamNoSys-to-SiGML translation

WP4: Synthetic Animation - WP4: Synthetic Animation - Progress (ii)Progress (ii)

HamNoSys-to-Signing (Fast) 2002-06HamNoSys-to-Signing (Fast) 2002-06 Synthetic Animation Engine re-implemented in C++Synthetic Animation Engine re-implemented in C++

50 times faster - generates approx. 1000 fps, supporting real-50 times faster - generates approx. 1000 fps, supporting real-time streamed input (e.g. Broadcast, WWW)time streamed input (e.g. Broadcast, WWW)

More flexible framework - basis for improved authenticityMore flexible framework - basis for improved authenticity Modular system architecture - supports flexible application Modular system architecture - supports flexible application

development, scripting in WWW pages, etc.development, scripting in WWW pages, etc.

Upgrade to Mask2 2002-09Upgrade to Mask2 2002-09 Interface to new primitive Mask2 ActiveX controlInterface to new primitive Mask2 ActiveX control

allows better control of animation frame schedulingallows better control of animation frame scheduling BAF replaced by VBM (ViSiCAST Bones and Morphs) - BAF replaced by VBM (ViSiCAST Bones and Morphs) -

provides framework for support of non-manual SiGMLprovides framework for support of non-manual SiGML

Presentation by Streams - Presentation by Streams - LinguisticsLinguistics

WP 4 AnimationWP 4 Animation Increased realism in sign generationIncreased realism in sign generation

Enhanced signing experienceEnhanced signing experience

WP5 Sign Language LinguisticsWP5 Sign Language Linguistics Use of natural sign languageUse of natural sign language Synthesis of sign language gesturesSynthesis of sign language gestures

WP 5: Language TechnologyWP 5: Language Technology

Goal within the project: Goal within the project: To provide semi-automatic translation from To provide semi-automatic translation from

English into BSL, DGS, NGTEnglish into BSL, DGS, NGT

Can also be used to assist the user in Can also be used to assist the user in monolingual language inputmonolingual language input No writing system for sign languages establishedNo writing system for sign languages established

Presentation by StreamsPresentation by Streams

Animation and LinguisticsAnimation and Linguistics

User ApplicationsUser Applications

Exploitation and DisseminationExploitation and Dissemination

WP2 Sign TutorWP2 Sign Tutor WP1 Television WP1 Television

Closed signing for Broadcast DTTClosed signing for Broadcast DTT

WP2 Internet WP2 Internet Information and Education for Sign Language Learners Information and Education for Sign Language Learners

WP3 Face to FaceWP3 Face to Face High Street Post Office Counter ServicesHigh Street Post Office Counter Services

WP6 Comparison of virtual signingWP6 Comparison of virtual signing with video-recorded Human Signingwith video-recorded Human Signing

Presentation by Streams - Presentation by Streams - Sign TutorSign Tutor

Presentation by Streams - Presentation by Streams - TelevisionTelevision


Closed signing for Broadcast DTTClosed signing for Broadcast DTT Enhanced signing experienceEnhanced signing experience Regulation and StandardsRegulation and Standards

WP2 Internet WP2 Internet Information and Education for Sign Language Learners Information and Education for Sign Language Learners

WP3 Face to FaceWP3 Face to Face WP6 Comparison of virtual signingWP6 Comparison of virtual signing

Low transmission rate < 25 kbit/sLow transmission rate < 25 kbit/s Compatibility with signing on other media and sign Compatibility with signing on other media and sign languageslanguages Precise, sharp representation of signerPrecise, sharp representation of signer Open display optionsOpen display options Compliance with international standards: MPEG, DVBCompliance with international standards: MPEG, DVB Future-proof:Future-proof:

cost savingcost saving allows vast no. of signed programmesallows vast no. of signed programmes unified framework from video-based to VH signingunified framework from video-based to VH signing

Virtual Humans on TVVirtual Humans on TV: The Advantages: The Advantages

Integrated TX system for broadcast to STBsIntegrated TX system for broadcast to STBs

Implementing virtual human s/w in STBImplementing virtual human s/w in STB

MPEG-2 delivery layer for maximum compliance:MPEG-2 delivery layer for maximum compliance: with existing hardwarewith existing hardware with MPEG & DVB standards with MPEG & DVB standards with proprietary formatswith proprietary formats

MPEG-4 Audio-Video codec and player MPEG-4 Audio-Video codec and player MPEG-4 compliant virtual humanMPEG-4 compliant virtual human MPEG-4 SNHC virtual human codec and player MPEG-4 SNHC virtual human codec and player MPEG-4 based closed signing service demonstrated MPEG-4 based closed signing service demonstrated

at IBC 2002at IBC 2002

Broadcast VH Signing:Broadcast VH Signing:AchievementsAchievements

MPEG-4MPEG-4SNHCSNHC

encoderencoder

MPEG-4MPEG-4videovideo

encoderencoder

MPEG-2 MPEG-2 AV AV

encoderencoder

MPEG-2 MPEG-2 AV AV

decoderdecoder

CompositorCompositorMUXMUX

PacketPacket

BAFBAFencoderencoder

MPEG-4MPEG-4multimediamultimedia

playerplayer dedePacketPacket

dedeMUXMUX

Encoder DecoderSystem System

Compositor

normativenormative

proprietaryproprietary

MPEG-4MPEG-4SNHCSNHC

decoderdecoder

BAFBAFdecoderdecoder

ProprietaryProprietaryMultimediaMultimedia

playerplayer

Broadcast VH Signing:Broadcast VH Signing:Functional architectureFunctional architecture

MPEG-2TS

Delivery

MPEG-4MPEG-4videovideo

decoderdecoder

Broadcast VH Signing:Broadcast VH Signing:System layer implementationSystem layer implementation

UDP/TCPUDP/TCPpacketiserpacketiser

IRT-DSPIRT-DSPMPEGMPEG

encoderencoder

RFRFmodulatormodulator

DVBDVBreceiver receiver

cardcard

IPIPfilterfilter

SystemSystem SystemSystemDeliveryDelivery

EncoderEncoder DecoderDecoderCompositorCompositor

MPEG-2MPEG-2TSTS

Broadcast VH Signing:Broadcast VH Signing:PerspectivesPerspectives

Advanced TX system for broadcast to Advanced TX system for broadcast to

MHP compliant STBsMHP compliant STBsOpen, MPEG & DVB compliant architectureOpen, MPEG & DVB compliant architecture

Improved synchronisation layer Improved synchronisation layer

Integrating a compositing layerIntegrating a compositing layer

Implementing an enriched MPEG-4 multimediaImplementing an enriched MPEG-4 multimedia

authoring tool authoring tool

Integrating SiGML streamIntegrating SiGML stream

DemonstrationDemonstration

Presentation by Streams -Presentation by Streams -WWW - WWW - Web pages with signing Field trialsWeb pages with signing Field trials



WP2 InternetWP2 Internet Information and Education for sign language learners Information and Education for sign language learners Web-pages with signingWeb-pages with signing


WP6 Comparison of virtual signingWP6 Comparison of virtual signing

weather signs

avatar

content provider

forecast creation tool

user‘play list’

Internet

web-browser + plug-in

1rst DEMO 2nd DEMO

Weather Forecast AWeather Forecast Applicationpplication

DemoDemo

Hosting at site of Dutch Deaf organisation Hosting at site of Dutch Deaf organisation Dovenschap: www.dovenschap.orgDovenschap: www.dovenschap.org

Running from end-June until end-OctoberRunning from end-June until end-October

Deaf users can join the field trial by filling in a form Deaf users can join the field trial by filling in a form on the websiteon the website

CD-rom with necessary software sent to usersCD-rom with necessary software sent to users

The field trials with Deaf usersThe field trials with Deaf users

Field Trial PromotedField Trial Promoted70 e-mails to webmasters of Deaf clubs, Deaf 70 e-mails to webmasters of Deaf clubs, Deaf

schools, Deaf organisations and private sites of schools, Deaf organisations and private sites of Deaf personsDeaf persons

promotion on Teletext (T.V.)promotion on Teletext (T.V.)on informative websites for Deaf peopleon informative websites for Deaf peoplevisit at meeting of national Deaf organisation visit at meeting of national Deaf organisation

with 12 member organisationswith 12 member organisationsarticle in magazine for sign language interpretersarticle in magazine for sign language interpreters30 CD-roms sent to Deaf clubs and schools30 CD-roms sent to Deaf clubs and schools

Trial FeedbackTrial Feedback

Helpdesk, contacted by e-mailHelpdesk, contacted by e-mail

Discussion page on websiteDiscussion page on website

Evaluation form: software and installation, Evaluation form: software and installation, included with receiving softwareincluded with receiving software

Evaluation form: avatar and sign language, will be Evaluation form: avatar and sign language, will be sent end of October 2002sent end of October 2002

Present SituationPresent Situation

Field trial still runningField trial still running

News slowly spreadingNews slowly spreading

Positive reactionsPositive reactions

Results at the end of NovemberResults at the end of November

Presentation by Streams – Presentation by Streams – Face to FaceFace to Face



WP2 Internet WP2 Internet WP3 Face to FaceWP3 Face to Face

High Street Post Office Counter ServicesHigh Street Post Office Counter Services Close involvement with RNIDClose involvement with RNID


WP3 OverviewWP3 Overview

Evaluation – October 2001Evaluation – October 2001 New TESSA system – Mar 2002New TESSA system – Mar 2002 Post Office Trial – May 2002 – PresentPost Office Trial – May 2002 – Present Sign Recognition – April 2002 – Sign Recognition – April 2002 –

PresentPresent

Evaluation – October 2001Evaluation – October 2001

Evaluation conducted at PO concept Evaluation conducted at PO concept store using TESSA V3. store using TESSA V3.

10 Deaf People and 5 Counter Clerks 10 Deaf People and 5 Counter Clerks participated over 10 days.participated over 10 days.

Mirror of previous evaluation + Some Mirror of previous evaluation + Some comparative tests of virtual signing comparative tests of virtual signing with a video recorded human signer with a video recorded human signer (full details in WP6 presentation)(full details in WP6 presentation)

Evaluation – ObservationsEvaluation – Observations

Clerks complained about the speed of Clerks complained about the speed of transactionstransactions

Caused by :Caused by : Toggle switch for recogniserToggle switch for recogniser Mis-recognitions caused by large vocabularyMis-recognitions caused by large vocabulary Poor mapping from recognised speech to Poor mapping from recognised speech to

phrasesphrases Cumbersome graphical interfaceCumbersome graphical interface

Tessa V4 – Recognition Tessa V4 – Recognition SystemSystem

‘‘Bag of words’ language model.Bag of words’ language model.

– Only words relevant to post office phrases recognised– Many fewer insertion errors– More resilient to external noise

Hello

Where

Goodbye

Going

First

Second

Class

…

TESSA V4 – Phrase MappingTESSA V4 – Phrase Mapping

Phrase mapping Phrase mapping system derived from system derived from work on Automatic work on Automatic Call RoutingCall Routing

Represent each of Represent each of the signed phrases the signed phrases and the test phrase and the test phrase as vectors in a co-as vectors in a co-occurrence matrixoccurrence matrix

AA 00 22 00 .. .. .. 11

AboutAbout 00 00 00 .. .. .. 00

AccessAccess 11 00 00 .. .. .. 00

AccountAccount 00 11 11 .. .. .. 00

.. .. .. .. .. .. .. ..

.. .. .. .. .. .. .. ..

.. .. .. .. .. .. .. ..

YouYou 00 00 00 .. .. .. 00

you’veyou’ve 0 0 00 00 .. .. .. 11

YourYour 1 1 00 00 .. .. .. 00

Phr

ase

1P

hras

e 1

Phr

ase

2P

hras

e 2

Phr

ase

3P

hras

e 3

Phr

ase

NP

hras

e N

TESSA V4 – Phrase MappingTESSA V4 – Phrase Mapping

Weight the entry W(i,j) such that :Weight the entry W(i,j) such that :

)|Pr(log()|1Pr(

)log(

1*)),(log(1(),( iwjpiw

jN

jjp

jNjiWjiW

More details in

S. Cox. “Speech and Language Processing for a Constrained Speech Translation System”. In Proc. Int. Conf. On Spoken Language Processing. October 2002

M.Lincoln and S.Cox. “A Comparison of Language Processing Techniques for a Constrained Speech Translation System” (Submitted ICASSP 2003)

• Calculate distance between vectors Calculate distance between vectors representing each canonical phrase representing each canonical phrase and input phrase.and input phrase.

TESSA V4 - Mapping TESSA V4 - Mapping EvaluationEvaluation

Subset of 155 phrases.Subset of 155 phrases. 5 Talkers, each asked to5 Talkers, each asked to

write down another way of expressing the write down another way of expressing the phrasephrase

record speaker saying this phraserecord speaker saying this phrase Recognise speech (NB No Adaptation)Recognise speech (NB No Adaptation)

75.1% Correct ; 49.8% Accurate75.1% Correct ; 49.8% Accurate

Test phrase mapping on both text and Test phrase mapping on both text and recognised speechrecognised speech

TESSA V4 – Mapping EvaluationTESSA V4 – Mapping Evaluation

TESSA V4 – User InterfaceTESSA V4 – User Interface

• Push to talk (automatic end of speech detection)

• Larger Buttons

• Common Phrases which don’t need to be spoken

• Continually updated list of top 5 most used signs

Post Office Trial - Set-upPost Office Trial - Set-up

Tessa V4 usedTessa V4 used5 Post Offices 5 Post Offices

London, Bristol, Derby, Liverpool, WolverhamptonLondon, Bristol, Derby, Liverpool, Wolverhampton

Known Deaf Communities In Each Area Known Deaf Communities In Each Area 3 Months Duration3 Months DurationEquipment Given Health Safety Equipment Given Health Safety

ApprovalApprovalTrained 19 Counter ClerksTrained 19 Counter ClerksProvided Help Desk Support Provided Help Desk Support

Post Office Trial - SurveyPost Office Trial - Survey

Independent Survey Customers by Independent Survey Customers by RNIDRNID

Independent Survey of Counter ClerksIndependent Survey of Counter Clerks All Users Given RNID QuestionnaireAll Users Given RNID Questionnaire All Counters Clerks InterviewedAll Counters Clerks Interviewed

Post Office Trial - PublicityPost Office Trial - Publicity

BBC See Hear – early OctBBC See Hear – early Oct Channel 4 – Documentary on BSLChannel 4 – Documentary on BSL Disability Times – 1 October 2002Disability Times – 1 October 2002 BBC Worldwide – 24 August 2002BBC Worldwide – 24 August 2002 ITV London Tonight – 21 August 2002ITV London Tonight – 21 August 2002 Liverpool Echo – 1 August 2002Liverpool Echo – 1 August 2002 Camden Chronicle – 1 August 2002Camden Chronicle – 1 August 2002 Wolverhampton Chronicle – 25 July 2002Wolverhampton Chronicle – 25 July 2002

Post Office Trial - PublicityPost Office Trial - Publicity

Bristol Evening Post – 22 July Bristol Evening Post – 22 July Liverpool Echo – 19 JulyLiverpool Echo – 19 July Derby Evening Telegraph – 18 JulyDerby Evening Telegraph – 18 July Wolverhampton Express and Star – 17 Wolverhampton Express and Star – 17

July July

Sign Language recognitionSign Language recognition

Preliminary investigationPreliminary investigation 6 Gestures, 10 training and 5 testing examples6 Gestures, 10 training and 5 testing examples Single userSingle user Motion captured dataMotion captured data HMM recognition systemHMM recognition system

Initial results – 95% accuracyInitial results – 95% accuracy

Sign Language RecognitionSign Language Recognition

Comparison of recognition using motion Comparison of recognition using motion captured data and video.captured data and video.

Collaboration with EU ‘WISDOM’ project. Collaboration with EU ‘WISDOM’ project. Currently Recording and editing multiuser Currently Recording and editing multiuser

database.database. 10 signs, 10 training and 5 testing examples10 signs, 10 training and 5 testing examples 5 users5 users Motion captured and videoMotion captured and video

RNID to make independent evaluation of RNID to make independent evaluation of recognition accuracy.recognition accuracy.

Presentation by Streams – Usability of Presentation by Streams – Usability of Virtual SigningVirtual Signing



WP2 Internet WP2 Internet Information and Education for Deaf PeopleInformation and Education for Deaf People



60 phrases from the PO TESSA system 60 phrases from the PO TESSA system signed by human interpreter on videosigned by human interpreter on video

120 phrases signed by the virtual human120 phrases signed by the virtual human 10 profoundly deaf people whose first 10 profoundly deaf people whose first

language is BSLlanguage is BSL Outcome measures:Outcome measures:

Accuracy of identificationAccuracy of identification Subjective ratings for each phraseSubjective ratings for each phrase Overall subjective ratingsOverall subjective ratings

MethodsMethods

Accuracy of identificationAccuracy of identification

0

20

40

60

80

100

Whole phrases Sign units

Acc

ura

cy (

%)

Virtual

Human

Subjective RatingsSubjective Ratings

Ease of identification

0

20

40

60

80

100

1 2 3 4 5

Rat

ing

s (%

)

Virtual

Human

Acceptability

0

20

40

60

80

100

1 2 3R

ati

ng

s (

%) Virtual

Human

Low HighVery easyVery difficult

Visual Analogue ScalesVisual Analogue Scales

0

20

40

60

80

100

Clarity Acceptability Phrases Sign units

Rat

ing

/ A

ccu

racy

(%

)

Virtual Human

Usability ConclusionsUsability Conclusions Higher accuracy of identification for human than virtual Higher accuracy of identification for human than virtual

signed phrases (signed phrases (20%)20%) Some improvements in intelligibility of virtual signing Some improvements in intelligibility of virtual signing

requiredrequired Non-ceiling benchmark of accuracy determinedNon-ceiling benchmark of accuracy determined 60% virtual signed phrases judged as good as human 60% virtual signed phrases judged as good as human

signed phrasessigned phrases Greater scope for improvements in terms of subjective Greater scope for improvements in terms of subjective

views of virtual signingviews of virtual signing Impressive results for virtual signingImpressive results for virtual signing

Exploitation and Exploitation and Dissemination HighlightsDissemination Highlights

TESSA IT Awards & success in the communityTESSA IT Awards & success in the community WWW Weather Forecaster launched in 2 European WWW Weather Forecaster launched in 2 European

Sign Languages & encouraging feedbackSign Languages & encouraging feedback IvD & RNID host in UK and the Netherlands IvD & RNID host in UK and the Netherlands

Close Involvement of Deaf PeopleClose Involvement of Deaf People RNID promoting ViSiCAST nationallyRNID promoting ViSiCAST nationally

BBC Collaboration for BBC Collaboration for closed signing solution for closed signing solution for broadcasting DTV for bandwidth efficiencybroadcasting DTV for bandwidth efficiency

Increasing amount of in-vision signing disliked by hearing peopleIncreasing amount of in-vision signing disliked by hearing people

Impacts on DTT multiplexes where bit-rate is already at a premiumImpacts on DTT multiplexes where bit-rate is already at a premium

Exploitation & DisseminationExploitation & Dissemination

UK Government 10 year target - 5%programmes UK Government 10 year target - 5%programmes on DTT services to be signedon DTT services to be signed

Today, services use ‘open signing’Today, services use ‘open signing’ Hearing viewers can find distractingHearing viewers can find distracting Seldom transmitted at peak viewing times Seldom transmitted at peak viewing times

Closed signing offers freedom Closed signing offers freedom for viewers - to turn on and offfor viewers - to turn on and off scheduling freedom for broadcastersscheduling freedom for broadcasters but needs extra transmission feedbut needs extra transmission feed

ViSiCAST uses ‘virtual human’ ViSiCAST uses ‘virtual human’ reducing bandwidth needs by factor of ten compared to video reducing bandwidth needs by factor of ten compared to video

Closed Signing – Why an avatar-based Closed Signing – Why an avatar-based solution ?solution ?

MPEG2 coding (0.5-1Mbit/s)MPEG2 coding (0.5-1Mbit/s) only 1 service signed per multiplex if at allonly 1 service signed per multiplex if at all

MPEG4 coding (<350Kbit/s)MPEG4 coding (<350Kbit/s) no more that 2 services signed per multiplexno more that 2 services signed per multiplex more efficient compression, and ability to code non-more efficient compression, and ability to code non-

rectangular objectsrectangular objects

Animated Avatars (<100Kbit/s)Animated Avatars (<100Kbit/s) may be possible to sign all services in a multiplexmay be possible to sign all services in a multiplex need new techniques to capture motion of real signersneed new techniques to capture motion of real signers

Closed Signing Requirements for the Closed Signing Requirements for the BroadcasterBroadcaster

Be compatible with existing studio, Be compatible with existing studio, distribution & monitoring infrastructuresdistribution & monitoring infrastructures

maintain freedom to schedule as neededmaintain freedom to schedule as needed accommodate live signing and reactive accommodate live signing and reactive

schedulingscheduling allow for regional content insertion and allow for regional content insertion and

time-shifting &time-shifting & cope with the variety of picture display cope with the variety of picture display

formatsformats

Avatar Signing developments Avatar Signing developments for broadcastingfor broadcasting

Motion capture needs to be Motion capture needs to be efficient and signer-independentefficient and signer-independent enabling signing of live and reactive enabling signing of live and reactive

broadcast materialbroadcast material best suited for offline broadcasting todaybest suited for offline broadcasting today

Facial motion capture needs Facial motion capture needs refinements refinements Increasing realism make avatars more Increasing realism make avatars more

acceptableacceptable

Signing CaptureSigning Capture - - Studio Implementation Studio Implementation

Original Programme

Monitor

Camera

SDI

SDI Coding /Compression

Signing Data SDI inserter

Ethernet

SDI with embedded Signing Data

Tape

Video Server

Signer

Motioncapture

Studio and distribution issuesStudio and distribution issues

Provision of television programme material with Provision of television programme material with associated signingassociated signing

Development of equipment for conveying signing data Development of equipment for conveying signing data within studio infrastructurewithin studio infrastructure We have developed hardware to add signing or motion capture We have developed hardware to add signing or motion capture

data to a SDI video stream.data to a SDI video stream. The main program video/audio, and the corresponding data can The main program video/audio, and the corresponding data can

then be routed via standard studio infrastructure.then be routed via standard studio infrastructure. The combined A/V and signing data can also be stored on server The combined A/V and signing data can also be stored on server

or video tapeor video tape Development of DVB inserter agnostic of signing signal coding Development of DVB inserter agnostic of signing signal coding

methodmethod Development of end-to-end DT demonstratorDevelopment of end-to-end DT demonstrator

visicast 2002 technical audit 4 october 2002, brussels michele wakefield - project manager, itc

Documents

animation visia

language slide

animation work

reasonable animation

itc slide

ec slide

transmission slide

fps slide