Statistical Error Correction Methods for Domain-Specific ASR Systems

Horia Cucu, Andi Buzo, Laurent Besacier, Corneliu Burileanu

University “Politehnica” of Bucharest, Romania

LIG, University Joseph Fourier, Grenoble, France

Paper Overview

● (Preliminary) Work on Statistical Error Correction

● Using Statistical Post-Editing (SPE)● Trained on a « parallel corpus »

● ASR outputs + Human Corrections● Can be applied to « black-box » systems

● Applied to Romanian ASR for a domain-specific task

● Not (yet) an interactive approach

Outline

● Related Works

● ASR Output Correction

● Experiments and Results

● Towards Machine Assisted Error Analysis

Outline

● Related Works

● ASR Output Correction

● Experiments and Results

● Towards Machine Assisted Error Analysis

Automatic Error Correction (1)

● Error correction (not detection)● Do not deal with word confidence estimation for error

detection, etc.● Focus on cases where ASR is a black-box

● User does not have any hooks to modify ASR models● ASR is called as an external API, 1 single engine is used

for several domains, etc.

● ASR error correction methods apply a post-processing block to correct errors in raw ASR transcripts

● Few related works in the litterature

Automatic Error Correction (2)● Use of a fertility model (derived from SMT theory) to correct 1-to-1, 1-to-2 and 2-to-1 errors

● Ringger, E.K., and Allen, J.F. A Fertility Channel Model for Post-Correction of Continuous Speech Recognition. ICSLP 1996

● Apply a similar fertility channel model (using syllables instead of words) along with an improved LM on Korean ASR

● Jung, S., Jeong, M., Lee, G.G. Speech recognition error correction using maximum entropy language model Interspeech 2004

Automatic Error Correction (3)● Using character co-occurrence and replacement rules trained on corrected ASR outputs (4300 utterances from a Japanese travel task)

● Kaki, S., Sumita, E., Iida, H. A method for correcting errors in speech recognition using the statistical features of character co-occurrence COLING-ACL 1998.

● Using ASR confusion networks to define rules that specify when the second candidate in a confusion set should be preferred over the first one.

● Mangu, L., Padmanabhan, M. Error corrective mechanisms for speech recognition ICASSP 2001

Automatic Error Correction (4)● See also

● Quaero ASR Error Analysis & Recovery Workshop (2011)

● To Summarize...● Some approaches from SMT theory● Rules extracted from « parallel » corpora● ASR output // Human corrections of ASR transcriptions

Statistical Post Edition (SPE) to train an automatic corrector

Statistical Post Edition (SPE)● SPE : originally designed for post-processing of Machine

Translation (MT) outputs● SPE is a Phrase-Based Machine Translation (PBMT) system

where source corpus consists of raw MT outputs and target corpus consists of post-edited version of raw translations

● Introduced in 2007 by P. Isabelle & M. Simard ● SPE trained on domain-specific data to adapt a general RBMT system to a

new specialized domain.● P. Isabelle, C. Goutte, and M. Simard,Domain adaptation of MT systems

through automatic post-editing in NAACL 2007● M.Simard, N.Ueffing, P. Isabelle, and R.Kuhn,Rule-based translation with

statistical phrase-based post- editing Statistical MachineTranslation Workshop 2007.

Outline

● Related Works

● ASR Output Correction

● Experiments and Results

● Towards Machine Assisted Error Analysis

Scenarios

● A « general domain » LVCSR system is available● A « domain specific » speech corpus has to be

transcribed● Scenario 1 : raw ASR transcripts are used to adapt

the initial LVCSR system● Scenario 2 : a medium-size amount (<2k) of ASR

output utterances is manually corrected● Scenario 2.a : LVCSR is blackbox => SPE● Scenario 2.b : LVCSR can be updated => LM

adaptation●

Using SPE to correct ASR transcripts

Scenario 2.a

Outline

● Related Works

● ASR Output Correction

● Experiments and Results

● Towards Machine Assisted Error Analysis

LVCSR System Overview● Acoustic model

● Based on the CMU Sphinx toolkit● 54h of Romanian read speech (17 speakers only)● 36 phonemes● CD-modelling – 5 states HMMs - 4000 senones - 16 GMs

● “General Domain” Language model● 169M words collected on the Web with diacritics restored● 3-gram model with a 64k words vocabulary

● Pronunciation model● G2P system based on PB-SMT approach

● For more details● H. Cucu & al. SMT-Based ASR Domain Adaptation Methods for Under-

Resourced Languages: Application to Romanian. Accepted to Speech Communication Journal, 2013.24.07.13 16University POLITEHNICA of Bucharest

Other Setups● Domain Specific Speech Corpus

● Weather news● Dev Set

● 2000 utterances transcribed by LVCSR and then corrected● 1 speaker only

● Test Set● 600 new utterances● 3 speakers

● Baseline LVCSR performance● 11.4% WER

● SPE trained using MOSES toolkit24.07.13 17University POLITEHNICA of Bucharest

Results (1)

Scenario 1 : raw ASR transcripts used to adapt initial LVCSR system

usable only if access to LVCSR internals

Scenario 2.a: blackbox LVCSR system + SPE

Results (2)

Scenario 2.b : updated LVCSR system – LM adaptation (interpolation)

and lexicon improvment (reduce OOV)

Results (3)

● LM adaptation using corrected transcripts remains the best

● Statistical Error Correction useful when ASR is a blackbox

● ASR domain adaptation by SPE● Phrase-Table of the SPE system

● Analyzing the most frequent errors

Outline

● Related Works

● ASR Output Correction

● Experiments and Results

● Towards Machine Assisted Error Analysis

Error analysis (1)● 1-to-1 replacements

● OOV replacement● masive muntoase (mountains) → masivele muntoase (the

mountains): 2 corrections out of 3,● averse (showers) → aversele (the showers): 1 correction out of

3,

● Non OOV replacement● ceata (the fog) → ceata (fog): 6 corrections,● noi (we) → norii (the clouds): 6 corrections,● continua (will continue) → continua (continues): 5 corrections, ● sint ([they] are, old form) → sunt ([they] are): 7 corrections,

Error analysis (1)● many-to-many replacements

● climatul logica/logice (logical climate) → climatologice (climatologically): 6 corrections,

● va sta la dispozitie (will be available)→ va sta la dispozitie (is available): 7 corrections,

● Wrong rules learned● ce (what) → aceste (these)● cinci (five) → in jur de (around)

Need for error detection and replacement rule filtering

Conclusion and Perspectives

● Conclusion● SPE as a post-process for ASR error correction● Useful for domain adaptation of a black box ASR● Allows ASR error analysis (as a by-product)

● Perspectives● Use ASR N-best to increase training data for SPE● Need to filter the SPE phrase-table

● Removing wrong replacement rules using phonetic di-similarity

● Use of an error detection● Correct only if an error was detected on ASR output