PG 1

Caroline V NelsonSpeech Solutions Team Technical LeadNortel Enterprise Multimedia Professional ServicesAugust 20, 2007

Success Criteria for Speech Applications

NORTEL CONFIDENTIAL PG 2

Zen and the Art of Speech Applications

> We all want to deploy “successful” speech systems

> But what exactly is the definition of “success”?

> That depends on who you ask…

NORTEL CONFIDENTIAL PG 3

“Success” depends on who you ask…

Three different perspectives:

Purchasing Customer

Speech Scientist

End User (Caller)

NORTEL CONFIDENTIAL PG 4

Speech Scientist (designs/develops the app)

> The Micro View:

• Recognition Accuracy

• In/Out of Grammar Utterances

• Per-state “success” (retries, etc)

• Host database interaction

• Caller transactions (tasks across multiple dialog states)

NORTEL CONFIDENTIAL PG 5

“Micro” can’t see the forest for the trees…

> The recognition accuracy numbers look great!• (but there’s a very high rate of

callers speaking out-of-grammar…)

> The overall speech numbers look nice!• (but broken out into states shows

a hotspot down the flow path…)

> The speech is working like a charm!• (but the back-end host is

experiencing some serious issues…)

NORTEL CONFIDENTIAL PG 6

End Use Caller (calls into the app)

> The “Middle-Management” View:

• Task Completion

• Quick

• Easy

• “usability”

NORTEL CONFIDENTIAL PG 7

Now that we have the foundation…

> Need speech scientist metrics to achieve caller goals

• Callers prompted to speak in-grammar

• Utterances recognized• Error recovery through effective

strategies• Reliable back-end host data

> Caller “task” start & end points must be defined and agreed upon in advance to be properly logged.

> But if it’s awkward and cumbersome to use…

• Which is why we do usability testing – right?

NORTEL CONFIDENTIAL PG 8

Customer (purchases the app for business)

> The Macro View:

• Self-service call containment

• Cost reduction

• Customer (caller) satisfaction = loyalty/retention

> In the end it’s all about the bottom line $$$

NORTEL CONFIDENTIAL PG 9

Comparison is Key to “Macro”

> Usability testing for caller satisfaction• Benchmarking against “industry

standards”

> Need metrics from previous system (dtmf?) for comparison • To demonstrate level of

improvement

> If no previous system…• Then each self service completed

task is automation gain• Compare automation level

against benchmarks per industry

NORTEL CONFIDENTIAL PG 10

It’s all connected…

Macro

Micro

Mid-level

> Recognition accuracy

> In/Out of Grammar

> State success (retries, etc)

> Reliable host data

> Task completion

> Quick and easy

> “Usability”

> Customer satisfaction

> Self-service automation

> Call containment

> Bottom line $$$

NORTEL CONFIDENTIAL PG 11

Be careful what you measure…

> …because what you measure is what you’ll get.

> Metrics need to be identified, agreed upon and planned in advance – including any comparison measures.