![Page 1: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/1.jpg)
Signing for the Deaf using Virtual Humans
Ian Marshall
Mike Lincoln
J.A. Bangham S.J.Cox
(UEA)
M. Tutt M.Wells
(TeleVirtual, Norwich)
![Page 2: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/2.jpg)
SignAnim
School of Information Systems, UEA
Televirtual, Norwich
Subtitles to Signing Conversion
Funded by
Independent Television Commission (UK)
![Page 3: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/3.jpg)
Tessa
School of Information Systems, UEA
Televirtual, Norwich
Speech to Signing of Counter Clerk Turns in PO Transactions
funded by Post Office
![Page 4: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/4.jpg)
ViSiCAST
School of Information Systems, UEA
Televirtual, Norwich
Independent Television Commission (UK)
Post Office (UK) RNID (UK)
IvD (Holland) University of Hamburg (Germany)
IST (Germany) INT (France)
EU funded 5th Framework Project
![Page 5: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/5.jpg)
Background – Deaf Community
Deaf v Hard of Hearing
Signing v. Subtitles
60,000 v. 1 in 8 of population
300 Level 3 signers
![Page 6: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/6.jpg)
Background – Sign Language
Signed Sign Supported British Sign
English English Language
(SE) (SSE) (BSL)
educated deaf community preferred first language
![Page 7: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/7.jpg)
SignAnim – Aims and Aspirations
Exploration of (semi-)automatic conversion of subtitles to sign language …
… to increase access for the Deaf ...
… with a potential of providing access to up to 50/80% of TV broadcasts.
![Page 8: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/8.jpg)
SignAnim – Natural Language Processing
Subtitle stream up to 180 words min -1
Sign rates typically 50% of speech rate (100 signs min-1)
SE – too verbose to be signed in full
SSE – elision of low information words
BSL – translation to multi-modal signs
![Page 9: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/9.jpg)
SignAnim Components – Simon the Avatar
Data Capture
Sign Dictionary
Avatar
Sign Stream
![Page 10: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/10.jpg)
Motion Capture
Cybergloves
Magnetic Sensors
Video face tracker
![Page 11: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/11.jpg)
Schematic of SignAnim system
TV Capture Card
Teletext Stream
Audio/Video Stream
Avatar Software MixerEliser
P1 P2… Pn
D1 D2
![Page 12: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/12.jpg)
SignAnim Components – Eliser
RequirementsResolution of Lexical Ambiguity
Elision
If @ receiver Timeliness of signing
v
If @ transmitter prioritising of parts of sign sequence
![Page 13: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/13.jpg)
Eliser - Summary
PrioritiserCMU Parser Elision Level
Subtitle Frames Sign Stream
Sign Dictionary
![Page 14: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/14.jpg)
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim and had to learn while a guest of the RAF in Norfolk.’
26 wordsin 2 subtitle framestime to speak / time subtitles on screen 7 secstime to sign in full 18 / 14 / 9 secsfinger spelling significant overhead
![Page 15: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/15.jpg)
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim and had to learn while a guest of the RAF in Norfolk.’
Resolution of some lexical ambiguity by p.o.s. tagging
- duck noun/ verb- had auxiliary/ verb- swim noun/ verb- in participle/ preposition
to facilitate correct sign selection
![Page 16: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/16.jpg)
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim and had to learn while a guest of the RAF in Norfolk.’
Potential elisiondeterminersauxiliary verbsmodifying phrasesadjectives and adverbs
in extreme cases jettison entire sentences
![Page 17: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/17.jpg)
SignAnim – Natural Language Processing
‘Last night we brought you the tale of the duck that could not swim and had to learn while a guest of the RAF in Norfolk.’
Additional problems
structural ambiguity
appropriate signno sign for guest, default finger spell
![Page 18: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/18.jpg)
SignAnim – CMU link grammar
Positive features
Lexically driven sentence parserRobustPrioritorises multiple analysesOn failure returns partially parsed word sequenceModifiability
![Page 19: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/19.jpg)
SignAnim – CMU link grammar example
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
![Page 20: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/20.jpg)
CMU link grammar parser - a shell
<noun> : ( {A-} & {D-} & Wd- & S+ ) or
( {A-} & {D-} & O- ) or
( {A-} & {D-} & PN- );
<adj> : A+ ;
<det> : D+ ;
<verb> : S- & O+ & {@PP+};
<prep> : PP- & PN+;
book.n books.n report.n reports.n room person : <noun> ;
yellow green : <adj> ;
the a : <det> ;
book.v books.v report.v reports.v brings : <verb> ;
on in : <prep> ;
CAPITALIZED-WORDS : <noun> or <adj> or <det> ;
"." : FS- ;
LEFT-WALL : (Wd+ & FS+) ;
![Page 21: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/21.jpg)
CMU link Grammar Parser - link construction
books .n : ( {A-} & {D-} & Wd- & S+ ) or
( {A-} & {D-} & O- ) or
( {A-} & {D-} & PN- ) or
.v : (S- & O+ & {@PP+});
books
{ A }
{ D }
WdS
books
{ A }
{ D }
O
books
{ A }
{ D }
PN
books
SO
{@PP}
person
{ A }
{ D }
WdS
books
SO
{@PP}
![Page 22: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/22.jpg)
linkparser> A person reports the book.
Found 1 linkage (1 had no P.P. violations) Unique linkage, cost vector = (UNUSED=0 DIS=0 AND=0 LEN=7)
+----------------FS---------------+
+---Wd---+ +------O-----+ |
| +--D-+---S---+ +--D--+ |
| | | | | | |
///// a person reports.v the book.n .
///// FS <---FS----> FS .
(m) ///// Wd <---Wd----> Wd person
(m) a D <---D-----> D person
(m) person S <---S-----> S reports.v
(m) reports.v O <---O-----> O book.n
(m) the D <---D-----> D book.n
![Page 23: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/23.jpg)
Eliser - elision stategy
Augment CMU dictionary with further p.o.s. information
e.g. has.aux v. has.v
Rules for word and path priorities
#Link Weight Left Path Left Word Right Path Right Word
CO 3 X X - -
D 1 - X - -
Ds 1 - X - -
G 4 X X - -
AN 4 - X - -
A 4 - X - -
RS 4 - X X X
![Page 24: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/24.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
![Page 25: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/25.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
10 10 10 10 10 10 10 10 10 10
![Page 26: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/26.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 10 10 10 10 10 10 10 10 10
![Page 27: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/27.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 10 10 10 10 10 10
![Page 28: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/28.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 2 10 10 10 10 10
![Page 29: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/29.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 2 10 9 9 9 9
![Page 30: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/30.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 2 10 9 1 9 9
![Page 31: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/31.jpg)
Eliser - Prioritorising
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 2 10 9 1 4 9
![Page 32: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/32.jpg)
Eliser - Elision
Perhaps the hen was actually reared by a broody duck !
Wd
Xp
Pv
Ds Ssi
CO
Jp
MVp
Dsu
AE
3 1 10 10 2 10 9 1 4 9
![Page 33: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/33.jpg)
TESSA - Overview
Aim : To give access to Post Office services for those whose first language is not English.
![Page 34: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/34.jpg)
TESSA Input : Speech Recognition
• Restricted Number of sentences (115)
• Variable quantities (monetary amounts, days of the week)
• Grammar defined as FSN
• MLLR acoustic adaptation
• Entropic recognition engine
![Page 35: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/35.jpg)
TESSA Output : BSL and Foreign Language
• BSL sign sequences
• Signs for variable quantities blended into standard phrases
• Customer may ask for phrases to be repeated
• Text translations into 4 languages for non-English speakers
• English text for the hard of hearing
![Page 36: Signing for the Deaf using Virtual Humans Ian Marshall Mike Lincoln J.A. Bangham S.J.Cox (UEA) M. Tutt M.Wells (TeleVirtual, Norwich)](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649dfe5503460f94ae693d/html5/thumbnails/36.jpg)
Conclusions
SignAnim and Tessa demonstrated
+ replay of motion captured sequences readable
+ usefulness of existing NLP and speech recognition technologies
+ desirability of BSL (rather than SSE)