visicast 2001 technical audit 8 october 2001, brussels michele wakefield - project manager, itc

62
ViSiCAST 2001 Technical Audit ViSiCAST 2001 Technical Audit 8 October 2001, Brussels 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC Michele Wakefield - Project Manager, ITC

Upload: elian-stammer

Post on 01-Apr-2015

215 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

ViSiCAST 2001 Technical AuditViSiCAST 2001 Technical Audit

8 October 2001, Brussels8 October 2001, BrusselsMichele Wakefield - Project Manager, ITCMichele Wakefield - Project Manager, ITC

Page 2: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

The ViSiCAST ProjectThe ViSiCAST Project

VVirtual irtual SSigning igning CCapture apture AAnimation nimation SStorage and torage and TTransmissionransmission

Page 3: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Aims of ViSiCAST ProjectAims of ViSiCAST Project

“…“…support improved access by deaf support improved access by deaf citizens to information and services citizens to information and services in sign language”in sign language”

user friendly methods to capture & generate signsuser friendly methods to capture & generate signs

machine readable system to describe gestures machine readable system to describe gestures

... preferred medium is sign language... preferred medium is sign language

Page 4: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Independent Television

CommissionTelevirtual

University of East Anglia

The Post Office

Royal Institute for Deaf People

Instituutvoor Doven

Hamburg University

Institut für Rundfunktechnik

Institut National des Télécommunications

ViSiCAST ConsortiumViSiCAST Consortium

Page 5: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Project DimensionsProject Dimensions

DurationDuration StartStart : January 2000: January 2000 FinishFinish : December 2002: December 2002 36 months36 months

Total Costs Total Costs 3770kECU total 3770kECU total 2876kECU funding from EC2876kECU funding from EC

Page 6: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

ViSiCAST Project HighlightsViSiCAST Project Highlights

Prototype enabling text translation and direct Prototype enabling text translation and direct synthesis of sign language gesturessynthesis of sign language gestures

Quality assessment support to other EU projectQuality assessment support to other EU project New TESSA system trial at Science Museum, LondonNew TESSA system trial at Science Museum, London

Achieved BCS IT Award and Gold MedalAchieved BCS IT Award and Gold Medal

Innovative transmission assessment for broadcast TVInnovative transmission assessment for broadcast TV BBC seek to deliver a closed signing service for broadcast DTVBBC seek to deliver a closed signing service for broadcast DTV

WWW Weather-forecaster with Virtual Signer WWW Weather-forecaster with Virtual Signer available in 3 Sign Languagesavailable in 3 Sign Languages

Page 7: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

WWW High Street Broadcast

Evaluation

Exploitation

Animation Linguistics

ViSiCAST Project StructureViSiCAST Project Structure

Technology

User Application

Exploitation &Dissemination

Page 8: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Technology Focus ObjectivesTechnology Focus Objectives

WP 4 AnimationWP 4 Animation Increased realism in sign generationIncreased realism in sign generation

Enhanced signing experienceEnhanced signing experience

WP5 Sign Language LinguisticsWP5 Sign Language Linguistics Use of natural sign languageUse of natural sign language Synthesis of sign language gesturesSynthesis of sign language gestures

Page 9: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Animation Work: ObjectivesAnimation Work: Objectives

WP4:WP4: Develop Hi-Resolution Avatars + related capture, Develop Hi-Resolution Avatars + related capture,

animation and transmission formats inc. animation and transmission formats inc. compressioncompression

To enable and support application development To enable and support application development in WPs 1-2-3 using WP4 (& WP5) Product.in WPs 1-2-3 using WP4 (& WP5) Product.

To further develop, compare and integrate both To further develop, compare and integrate both proprietary and standard solutions, where proprietary and standard solutions, where appropriateappropriate

Page 10: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC
Page 11: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Animation: Current WorkAnimation: Current Work

Through Year TwoThrough Year Two Continued to support Application developmentContinued to support Application development Continuous upgrade to VISIA / TESSA player Continuous upgrade to VISIA / TESSA player

(Open GL renderer under Active X control)(Open GL renderer under Active X control) Bug fixing / Motion capture supportBug fixing / Motion capture support .baf format and compression layer with WP1 to .baf format and compression layer with WP1 to

create Broadcast Demonstrator using Vsicast create Broadcast Demonstrator using Vsicast systemsystem

MPEG compatability / parallel development in MPEG compatability / parallel development in WP4 and applicationsWP4 and applications

Page 12: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC
Page 13: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Animation: Continuing & future Animation: Continuing & future WorkWork

Working on ways to improve facial Working on ways to improve facial animation / realism (forehead / eyes)animation / realism (forehead / eyes)

Exploring Statistical Methods to define Exploring Statistical Methods to define and generate facial Animation and generate facial Animation

Working on ways to facilitate Avatar Working on ways to facilitate Avatar creation (Photographic acquisition)creation (Photographic acquisition)

Mask 2 + Improved Mo CapMask 2 + Improved Mo Cap

Page 14: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC
Page 15: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC
Page 16: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

MPEG-4 SNHC for interoperable animationMPEG-4 SNHC for interoperable animation MPEG-4 SNHC MPEG-4 SNHC playerplayer and and

serverserver delivered in June 2001 delivered in June 2001

MPEG-4 compliant AnimationMPEG-4 compliant AnimationAchievementsAchievements

5 to 25 kbit/s5 to 25 kbit/s

7 to 14 bit/vertex7 to 14 bit/vertex

Making use of a MPEG-4 compliant Making use of a MPEG-4 compliant VisiaVisia model model Compliance with VRML standard (H-Anim Compliance with VRML standard (H-Anim

specifications)specifications)

Incorporating a full compression layerIncorporating a full compression layer 3D mesh & texture encoding3D mesh & texture encoding Motion parameters (BAP/FAP) encodingMotion parameters (BAP/FAP) encoding

Implementing importation and editing toolsImplementing importation and editing tools

Open delivery interface: MPEG-2, IP, ATM ... Open delivery interface: MPEG-2, IP, ATM ...

Page 17: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Advanced interoperable distributed animation systemAdvanced interoperable distributed animation system

Improved facial animationImproved facial animation

MPEG-4 System layer implementationMPEG-4 System layer implementation Multimedia (audio, video, text…) synchronisationMultimedia (audio, video, text…) synchronisation Error resilience Error resilience Management of scene descriptionManagement of scene description

MPEG-compliant SiGML-driven animationMPEG-compliant SiGML-driven animation

Open input/output interfaceOpen input/output interface

MPEG-4 compliant AnimationMPEG-4 compliant AnimationPerspectivesPerspectives

Page 18: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Presentation by Streams - Presentation by Streams - LinguisticsLinguistics

WP 4 AnimationWP 4 Animation Increased realism in sign generationIncreased realism in sign generation

Enhanced signing experienceEnhanced signing experience

WP5 Sign Language LinguisticsWP5 Sign Language Linguistics Use of natural sign languageUse of natural sign language Synthesis of sign language gesturesSynthesis of sign language gestures

Page 19: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

WP 5: Language TechnologyWP 5: Language Technology

Goal within the project: Goal within the project: To provide semi-automatic translation from To provide semi-automatic translation from

English into BSL, DGS, NGTEnglish into BSL, DGS, NGT

Can also be used to assist the user in Can also be used to assist the user in monolingual language inputmonolingual language input No writing system for sign languages establishedNo writing system for sign languages established

Page 20: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

The last year: 3 deliverablesThe last year: 3 deliverables

D5-1: Defining the interfacesD5-1: Defining the interfaces

D5-2: Transfer to XML: D5-2: Transfer to XML: SiGML definitionSiGML definition

D5-3 Prototype translation system: D5-3 Prototype translation system: English to notationEnglish to notation

Page 21: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

D5-1: Defining the interfacesD5-1: Defining the interfaces

Adaptation of Discourse Representation StructureAdaptation of Discourse Representation Structure

Extension of HamNoSys, a phonetic transcription Extension of HamNoSys, a phonetic transcription system for sign languagesystem for sign language

Notation conventions for all non-manual aspects Notation conventions for all non-manual aspects relevant for (European) sign languagesrelevant for (European) sign languages Body movementBody movement Head movementHead movement Facial expressionsFacial expressions Mouthing and Mouth gesturesMouthing and Mouth gestures Eye movementEye movement Synchronicity with manual elementsSynchronicity with manual elements

Page 22: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

D5-2: SiGMLD5-2: SiGML

Defines XML domain based on D5-1 Defines XML domain based on D5-1 manual and non-manual notationmanual and non-manual notation

Simple timing modelSimple timing model Probably to be revised to ease Probably to be revised to ease

integration with upcoming integration with upcoming synchronisation models as required for synchronisation models as required for broadcasting etc.broadcasting etc. SMIL, XMT (MPEG4) etc.SMIL, XMT (MPEG4) etc.

Page 23: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

D5-3: Proto text-to-sign notationD5-3: Proto text-to-sign notation

English to semantics (DRS)English to semantics (DRS) CMU ParserCMU Parser DRS constructionDRS construction

Semantics to sign language notationSemantics to sign language notation DRS to HPSG semantics (ALE/MRS)DRS to HPSG semantics (ALE/MRS) HPSG generation (ALE/LinGo)HPSG generation (ALE/LinGo) HPSG PHON (HamNoSys) to SiGMLHPSG PHON (HamNoSys) to SiGML

Page 24: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

HPSG modelling of sign languagesHPSG modelling of sign languages

Aiming at proper sign language, not Aiming at proper sign language, not anything like SEEanything like SEE

No detailed grammars published, no No detailed grammars published, no usable dictionariesusable dictionaries

Most importantly: Data-drivenMost importantly: Data-driven Lexicon and every aspect of our grammar Lexicon and every aspect of our grammar

fragmentfragment

Page 25: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Example: Verifying detailsExample: Verifying details

Zur Anzeige wird der QuickTime™ Dekompressor “Cinepak”

benötigt.

Zur Anzeige wird der QuickTime™ Dekompressor “Sorenson Video”

benötigt.

Page 26: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Demo: D5-3 plus D4-2Demo: D5-3 plus D4-2

Due month 26 (Feb 02), i.e. work in Due month 26 (Feb 02), i.e. work in progressprogress

Complete route from English to sign Complete route from English to sign language animationlanguage animation

Page 27: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Convert avatar-independent SiGML to Convert avatar-independent SiGML to avatar-specific description:avatar-specific description:

Define all SiGML locations (shoulder, eyes, fingertip, Define all SiGML locations (shoulder, eyes, fingertip, etc.) in terms of the avatar's geometryetc.) in terms of the avatar's geometry Define hand shapes in terms of rotations of the hand Define hand shapes in terms of rotations of the hand

jointsjoints Determine arm joint rotations from hand positions by Determine arm joint rotations from hand positions by

inverse kinematicsinverse kinematics Convert SiGML movements into numerically defined Convert SiGML movements into numerically defined

trajectoriestrajectories Output in BAF format or VRMLOutput in BAF format or VRML

Synthetic Animation of SiGMLSynthetic Animation of SiGML

Page 28: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Model each joint by a second-order Model each joint by a second-order control systemcontrol system

a muscle applies a torque to the joint, resisted by a a muscle applies a torque to the joint, resisted by a moment of inertia and dampingmoment of inertia and damping

Generate different types of motion (fast, Generate different types of motion (fast, slow, etc.) by varying the model slow, etc.) by varying the model parametersparameters

Biocontrol modelBiocontrol model

Page 29: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

If only hands, arms, and face are If only hands, arms, and face are animated, the result is stiff and lifeless.animated, the result is stiff and lifeless.

Animate the spine and head by mixing Animate the spine and head by mixing “ambient motion” from motion capture files “ambient motion” from motion capture files with synthetic animation.with synthetic animation.

Ambient motionAmbient motion

Page 30: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Closing the feedback loopClosing the feedback loop

So far, only the native signers involved in the So far, only the native signers involved in the project can judge the output of our HPSG project can judge the output of our HPSG generation systemgeneration system Requires intimate knowledge of HamNoSys at leastRequires intimate knowledge of HamNoSys at least

With the animation output, we have access to the With the animation output, we have access to the native signers’ intuition of much more people than native signers’ intuition of much more people than todaytoday

Opens the way to more formal evaluation of the Opens the way to more formal evaluation of the generation system than is available to dategeneration system than is available to date

Page 31: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Summary: Language TechnologySummary: Language Technology

First successful steps in HPSG language modelling First successful steps in HPSG language modelling and translation of English to sign languageand translation of English to sign language

Encoding established and extended sign language Encoding established and extended sign language notation with standard description model (XML)notation with standard description model (XML)

Already close to closing the feedback loop to allow Already close to closing the feedback loop to allow native signers evaluation of our language native signers evaluation of our language production systemproduction system

Page 32: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Presentation by StreamsPresentation by Streams

Animation and LinguisticsAnimation and Linguistics

User Applications : Evaluation of broadcast User Applications : Evaluation of broadcast transmission for DTV transmission for DTV

Exploitation and DisseminationExploitation and Dissemination

Page 33: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Presentation by Streams - Presentation by Streams - TelevisionTelevision

WP1 Television WP1 Television Closed signing for Broadcast DTTClosed signing for Broadcast DTT Enhanced signing experienceEnhanced signing experience Regulation and StandardsRegulation and Standards

WP2 Internet WP2 Internet Information and Education for Deaf PeopleInformation and Education for Deaf People

WP3 Face to FaceWP3 Face to Face High Street Post Office Counter ServicesHigh Street Post Office Counter Services

Science Museum Trial - Summer 2001Science Museum Trial - Summer 2001

Page 34: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Low transmission rate < 25 kbit/sLow transmission rate < 25 kbit/s Compatibility with signing on other media andCompatibility with signing on other media and foreign deaf languagesforeign deaf languages Precise, sharp representation of signerPrecise, sharp representation of signer Open display optionsOpen display options Compliance with international standards: MPEG, DVBCompliance with international standards: MPEG, DVB Future-proof:Future-proof:

cost savingcost saving allows vast no. of signed programmesallows vast no. of signed programmes no transition from video-based to VH signingno transition from video-based to VH signing

VH on TV: The AdvantagesVH on TV: The Advantages

Page 35: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Integrated TX system for broadcast to STBsIntegrated TX system for broadcast to STBs demonstrator complete end of 2000demonstrator complete end of 2000

Implementing virtual human s/w in STBImplementing virtual human s/w in STB

Incorporating a compression layerIncorporating a compression layer

Using MPEG-2 delivery layer for maximum compliance: Using MPEG-2 delivery layer for maximum compliance: with existing hardwarewith existing hardware with MPEG & DVB standardswith MPEG & DVB standards with proprietary formatswith proprietary formats

Broadcast VH Signing:Broadcast VH Signing:AchievementsAchievements

Page 36: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Broadcast VH Signing:Broadcast VH Signing:Functional architectureFunctional architecture

MUXMUX

PacketPacket

MPEG-2MPEG-2AVAV

encoderencoder

MPEG-4MPEG-4SNHCSNHC

encoderencoder

BAFBAFencoderencoder

MPEG-2MPEG-2AVAV

decoderdecoder

MPEG-4MPEG-4SNHCSNHC

decoderdecoder

BAFBAFdecoderdecoder

MPEG-4MPEG-4SNHCSNHCplayerplayer

BAFBAFplayerplayer

CCOOMMPPOOSSEE

dedePacketPacket

dedeMUXMUX

EncoderEncoder DecoderDecoderCompositorCompositor

SystemSystem SystemSystemDeliveryDelivery

normativenormative

proprietaryproprietaryMPEG-2MPEG-2

TSTS

Page 37: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Broadcast VH Signing:Broadcast VH Signing:System layer implementationSystem layer implementation

UDP/TCPUDP/TCPpacketiserpacketiser

ThomsonThomsonMPEGMPEG

encoderencoder

RFRFmodulatormodulator

DVBDVBreceiver receiver

cardcard

IPIPfilterfilter

SystemSystem SystemSystemDeliveryDelivery

EncoderEncoder DecoderDecoderCompositorCompositor

MPEG-2MPEG-2TSTS

Page 38: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

MPEG-2 Transport Stream (TS)

MPEG-2 Packetized Elementary Stream (PES) Section PES

Broadcast VH Signing:Broadcast VH Signing:Versatile delivery architectureVersatile delivery architecture

BAFAudioVideo

FlexMUX

Scenedesc.

Audio Video SNHC

MPEG-4MPEG-2

Proprietary

SiGML SiGML

Text

MPEG-7ContentdescriptionContentdescription

CodingCoding

DeliveryDelivery

DVB compliantDVB compliant

Page 39: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Advanced TX system for broadcast to STBsAdvanced TX system for broadcast to STBs

Open, MPEG & DVB compliant architectureOpen, MPEG & DVB compliant architecture

Improved synchronisation layer Improved synchronisation layer

Integrating a compositing layerIntegrating a compositing layer

Implementing a complete MPEG-4 multimedia player Implementing a complete MPEG-4 multimedia player

Integrating SiGML streamIntegrating SiGML stream

Broadcast VH Signing:Broadcast VH Signing:PerspectivesPerspectives

Page 40: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

MPEGMPEG

CompositorCompositor

Broadcast VH Signing:Broadcast VH Signing:Targeted architectureTargeted architecture

MUXMUX

PacketPacket

MPEG-4MPEG-4SNHCSNHC

encoderencoder

BAFBAFencoderencoder

MPEG-4MPEG-4SNHCSNHC

decoderdecoder

BAFBAFdecoderdecoder

MultimediaMultimediaplayerplayer

dedePacketPacket

dedeMUXMUX

EncoderEncoder DecoderDecoderCompositorCompositor

SystemSystem SystemSystemDeliveryDelivery

normativenormative

proprietaryproprietaryMPEG-2MPEG-2

TSTS

MPEG-MPEG-AVAV

encoderencoder

22 44

MPEG-MPEG-AVAV

decoderdecoder

22 44

Page 41: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Presentation by Streams - Presentation by Streams - WWWWWW

WP1 Television WP1 Television Closed signing for Broadcast DTTClosed signing for Broadcast DTT

Enhanced signing experienceEnhanced signing experience Regulation and StandardsRegulation and Standards

WP2 Internet WP2 Internet Information and Education for Deaf PeopleInformation and Education for Deaf People

WP3 Face to FaceWP3 Face to Face High Street Post Office Counter ServicesHigh Street Post Office Counter Services

Science Museum Trial - Summer 2001Science Museum Trial - Summer 2001

Page 42: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Weather Forecast AWeather Forecast Applicationpplication

First WWW application:First WWW application:daily weather forecast in 3 sign daily weather forecast in 3 sign languageslanguages content creationcontent creation example forecastexample forecast evaluationevaluation

Page 43: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

DemoDemo

Page 44: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Evaluation with Deaf usersEvaluation with Deaf users

Subjective quality of signing rated as Subjective quality of signing rated as ‘reasonable’ or ‘good’‘reasonable’ or ‘good’

68% correct or partially correct68% correct or partially correct

Improvement possibilities Improvement possibilities mouthingmouthing facial expressionsfacial expressions

Page 45: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

MouthingMouthing

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

independent partiallydependent

mainlydependent

partially correctcorrect

Scores for signs depending in various degrees on mouthing

Page 46: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Facial ExpressionsFacial Expressions

0%

10%

20%

30%

40%

50%

60%

70%

80%

independent partiallydependent

mainlydependent

partially correctcorrect

Scores for signs depending in various degrees on facial expressions

Page 47: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Next StepsNext Steps

ImprovementsImprovements

Beta-testingBeta-testing on lineon line larger user group larger user group user feedbackuser feedback

Exploitation planningExploitation planning

Page 48: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Presentation by Streams – Presentation by Streams – Face to FaceFace to Face

WP1 Television WP1 Television Closed signing for Broadcast DTTClosed signing for Broadcast DTT Enhanced signing experienceEnhanced signing experience Regulation and StandardsRegulation and Standards

WP2 Internet WP2 Internet Information and Education for Deaf PeopleInformation and Education for Deaf People

WP3 Face to FaceWP3 Face to Face High Street Post Office Counter ServicesHigh Street Post Office Counter Services Science Museum Trial - Summer 2001Science Museum Trial - Summer 2001

Page 49: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

WP3: Face-to-face transactions WP3: Face-to-face transactions

Research concentrated on TESSA (Text and Research concentrated on TESSA (Text and Sign Support Agent)Sign Support Agent) Enables Post Office counter clerks to “translate” from Enables Post Office counter clerks to “translate” from

(English) speech to sign language(English) speech to sign language

System developments:System developments: Autumn 2000: New system software completed, Autumn 2000: New system software completed,

incorporating IBM “Via Voice” speech recognition and incorporating IBM “Via Voice” speech recognition and improved avatarimproved avatar

Spring 2001: Spring 2001: 200 new signs recorded, processed and 200 new signs recorded, processed and added to systemadded to system

Spring/Summer 2001: Development and testing of Spring/Summer 2001: Development and testing of “unconstrained system”“unconstrained system”

Page 50: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

First System using Constrained First System using Constrained Speech RecognitionSpeech Recognition

Page 51: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

““Unconstrained” Speech SystemUnconstrained” Speech System

Page 52: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

DemoDemo

Page 53: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Testing the Speech Recognition Testing the Speech Recognition Accuracy of the Unconstrained SystemAccuracy of the Unconstrained System

Single speaker Single speaker 200 “constrained” phrases 200 “constrained” phrases

Three recording conditions:Three recording conditions: studio microphone in acoustic boothstudio microphone in acoustic booth boom microphone I in labboom microphone I in lab boom microphone II in Science Museum Post Officeboom microphone II in Science Museum Post Office

Three conditions for recogniser:Three conditions for recogniser: UntrainedUntrained Acoustic models fully trained on boom microphone II in labAcoustic models fully trained on boom microphone II in lab Acoustic and language models fully trainedAcoustic and language models fully trained

Page 54: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Speech recognition accuracy Speech recognition accuracy of unconstrained systemof unconstrained system

Word accuracy

0

20

40

60

80

100

Untrained user Fully trained user,acoustic models

only

Fully trained user,acoustic and

language modelstrained

Pe

rce

nta

ge

ac

cu

rac

y

PO Test

Lab Test Mic 1

Lab Test Mic 2

Page 55: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Testing the Phrase Retrieval Accuracy Testing the Phrase Retrieval Accuracy of the Unconstrained Systemof the Unconstrained System

10 speakers10 speakers For each speaker and each of 200 phrases:For each speaker and each of 200 phrases:

record one utterance of the “constrained” phraserecord one utterance of the “constrained” phrase ask speaker to write down another way of expressing the phraseask speaker to write down another way of expressing the phrase record speaker saying this phraserecord speaker saying this phrase

Training of recogniser not possible for 10 different Training of recogniser not possible for 10 different speakersspeakers

Hence measure phrase retrieval accuracy on Hence measure phrase retrieval accuracy on texttext of of unconstrained phrases onlyunconstrained phrases only

Page 56: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Phrase recognition Results on Phrase recognition Results on Text of Alternative UtterancesText of Alternative Utterances

Name Correct Wrong Correct meaning (but wrong phrase selected)

Money formatted incorrectly

Phrase not in system

Percentage Accuracy

Alex 150 33 10 3 4 75

Cara 162 20 14 0 4 81

Matt I 148 23 12 13 4 74

Matt II 133 57 6 - 4 66.5

Jo 141 70.5

Briony 146 73

David 146 73

Average accuracy = 73.3%

Page 57: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Future WorkFuture Work

Unconstrained SystemUnconstrained System Investigate use of partial string matching of word Investigate use of partial string matching of word

sequences and phoneme sequencessequences and phoneme sequences Investigate use of Latent Semantic AnalysisInvestigate use of Latent Semantic Analysis Add spoken language(s) translationAdd spoken language(s) translation

Sign recognitionSign recognition Collect dataCollect data Configure baseline systemConfigure baseline system

Page 58: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Exploitation and Exploitation and Dissemination HighlightsDissemination Highlights

Exploitation and DisseminationExploitation and Dissemination BBC Collaboration for BBC Collaboration for closed signing solution closed signing solution

for broadcasting DTVfor broadcasting DTV TESSA BCS IT Award & Gold Medal TESSA BCS IT Award & Gold Medal WWW Weather Forecasting in 3 European Sign WWW Weather Forecasting in 3 European Sign

LanguagesLanguages Close Involvement of Deaf PeopleClose Involvement of Deaf People

Page 59: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Dissemination Highlights Dissemination Highlights

November 2000: TESSA wins British Computer November 2000: TESSA wins British Computer Society Gold Medal for ITSociety Gold Medal for IT

February 2001: TESSA exhibited at Royal SocietyFebruary 2001: TESSA exhibited at Royal Society March 2001: TESSA appears on “Computer Club” March 2001: TESSA appears on “Computer Club”

(German TV)(German TV) July–September 2001: TESSA on exhibition at July–September 2001: TESSA on exhibition at

Science Museum, LondonScience Museum, London October 8th 2001: TESSA appears on “Blue Peter” October 8th 2001: TESSA appears on “Blue Peter”

(BBC TV)(BBC TV) November 2001: TESSA on show at COMDEX, Las November 2001: TESSA on show at COMDEX, Las

VegasVegas

Page 60: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Exploitation Highlights / Short Term Exploitation Highlights / Short Term

Bandwidth efficient closed signing Bandwidth efficient closed signing Excessive in-vision signing disliked by hearing Excessive in-vision signing disliked by hearing

peoplepeople Impacts on DTT multiplexes where bit-rate is Impacts on DTT multiplexes where bit-rate is

already at a premiumalready at a premium BBC investigation of closed signing for DTVBBC investigation of closed signing for DTV

Demonstration of Avatar-based signingDemonstration of Avatar-based signingBody suit capture technologiesBody suit capture technologies

Page 61: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Short Term- WWW strategyShort Term- WWW strategy

Give away basic web browserGive away basic web browser

Sell SiGML authoring tool presentedSell SiGML authoring tool presented

De factoDe facto standard standard

Page 62: ViSiCAST 2001 Technical Audit 8 October 2001, Brussels Michele Wakefield - Project Manager, ITC

Exploitation Highlights Exploitation Highlights Medium to Long Term Medium to Long Term

Conversion of subtitlesConversion of subtitles high % of programmes subtitledhigh % of programmes subtitled supports wide range of deaf signing languagessupports wide range of deaf signing languages subtitles translated in set top boxsubtitles translated in set top box overcomes spectrum capacity & scheduling restrictionsovercomes spectrum capacity & scheduling restrictions

Requirements:Requirements: reliable unconstrained translator reliable unconstrained translator next generation DVB-compliant STB with in-built signing decodernext generation DVB-compliant STB with in-built signing decoder