-2-

Outline

IntroductionMethodsProblems to be solvedDemos

-3-

Speech Assessment

Speech assessment: How to assess an utterance for the purpose of learning a spoken language? Assessment levels: syllables, words, sentences,

paragraphs Assessment criteria: timbre, tone, energy, rhythm,

co-articulation, … Feedbacks: High-level correction and suggestions

-4-

Related Disciplines

Related disciplines for speech assessment: Language learning:

CALL: Computer Assisted Language LearningCAPT: Computer Assisted Pronunciation Training

Speech technology:UV: Utterance Verification

-5-

Our Approach

Basic approach to timbre assessment Lexicon net construction (Usually a sausage net) Forced alignment to identify phone boundaries Phone scoring based on several criteria, such as

ranking, histograms, posterior prob., etc. Weighted average to get syllable score Weighted average to get sentence score

-6-

Basic Assessment Criteria

Timber Based on acoustic

models

Tone Based on tone

recognition (for tonal language)

Based on pitch similarity with the target utterance

Energy Based on energy

comparison with the target utterance

Rhythm Based on duration

comparison with the target utterance

Fluency

-7-

Additional Assessment Criteria

English Stress

Levels (word or sentence) Meanings

IntonationDeclarative sentenceInterrogative sentence

Co-articulationA red apple.Did you call me?Hit and run

Mandarin Tone Retroflex or not Co-articulation

兒化音

-8-

Problems to be Solved

Score related Optimization Consistency Interpretability

Confusing phone id. ( 日本人的發音 )Slightly adaptationParagraph-level assessmentContents construction

-9-

Demo: Practice of Mandarin Idioms of Length 4 ( 一語中的 )

Level (difficulty) of an idiom is based on it’s freq. via Google search:孤掌難鳴 ===> 260,000鶼鰈情深 ===> 43,300亡鈇意鄰 ===> 22,700舉案齊眉 ===> 235,000

Can be adapted for English learning

Next step: multi-threading, fast decoding via FSM

-10-

Demo: Recitation Machine （唸唸不忘）

Support Mandarin & English

Support user-defined recitation script

Next step: multithreading for recording & recognition

-11-

Demo: Dialog Practice via Videos

Dialog-based practice and evaluation

-12-

Demos on PC and PMP

PC 軟體 Lucy’s Café: Speech and Score

PMP 華語練習機

-13-

Demo: Embedded Systems

Chicken run (落跑雞)

Penguin for Tang Poetry (唐詩企鵝)

Robot Fighter (蘿蔔戰士)

Singing Bass & Dog (大嘴鱸魚和唱歌狗)

-14-

On-going WorkOn-going work:

Tone recognition and assessment Retroflex & nonretroflex recognition Detection of “ 兒化音”

Demo page: http://mirlab.org/mir_main/demo.htm