ad hoc evaluation of sds along project lifecylcle in orange
DESCRIPTION
Laurent, M., Bretier, P. and Manquillet, C. (2010). Ad-hoc evaluations along the lifecycle of industrial spoken dialogue systems: heading to harmonisation?. In LREC 2010. MaltaTRANSCRIPT
Homonymy
Traceability
Nomadismof evaluation
methodologies !!
and Synonymy
Need : One evaluation
per evaluator
per situation
Difficult communicationbetween actors
No standard forinformation sharing,
cooperationand collaboration !
Local Contraints: Budget,
data collection, etc.
Different norms
depending on the
Communitiesof Practice
Need to refine indicators
+ collected data
Call-flow: entry, path, exit
Callerhistory Call
motive
Referencevalues
Analysis
Framing
Development
Experiments
Pilot
Generalisation
Exploitation
Specification of project scope, specifications and
project plan.
Feasibility evaluation.
The service is maintained by both the hosting
providers and the initial developers (for
corrections). Access is granted to the
overall flow of traffic. .
Service tested under real conditions, with a
fragment of the real flow of traffic.
The evaluation constitute a comprehensive
validation of the service before its access is
extended to all users. It includes the relevance of
the technical architecture.so as to detect and fix the last beta-solution issues.
Test the service so as to identify design issues
and refine the development to the users
practices. Alignment of observation with a priori
projections.
Tests organised on a panel of recruited users.
Development of a prototype integrating the initial design, linguistic, usability and technical
considerations.
Test within the development team to
detect major bugs and enrich the vocal
recongition model,
Analysis of end-users' needs and practices. Solutions benchmark.
Collect the users experience feedback in live conditions.
The service keeps constantky monitored. Dashboards containing high-level indicators are
sent daily to the various interested teams (number
EvaluationContexts
What is done?
Evaluation context?
Marketingexperts
Hostingproviders
Ergon
omic
s
expe
rts
Vocal
experts
Use
r ex
perie
nce
Pla
tfo
rm in
tera
ctio
ns
Tec
hn
ical
per
form
ance
SD
S- U
sers
inte
ract
ion
sS
erv
ice
%
46,2
3
12
3658
14
What is aSpoken
Dialogue System?
Viewpoints
Métier TechnicalDevelopers
ErgonomicsExperts Marketing
Experts
BusinessManagers
HostingProviders
GoalsEvaluationsIndicators
ContractingOwners
Hard/software
platform delivering
a service to
the customer
Automate routing service satisfying
the customer relationship
management
Hard / software platform
Interactive systemsimplying a human-amchine dialogue,
a QoS & an individualised
access to information
A technology basedservice tailored to
the various business customers' needs
Hard / software platform
Service automating
the human
operators low
added-value
tasks
Enhance the customer service with
automated vocal solutions
Deliver a service to the customer with respect to
the customer relationship policy
Ensure the respect of the predefined QoS
commitments and the end-user satisfaction.
ROI (project financing decisions), QoS,
dialogue performance, user satisfaction
Evaluate the service quality and its adequacy
to the organisation
Track the end-user experience
Expert evaluation campaign (field tests
and real users)
Measure of the service performance and the
users' perception.
Measure of the service performance, monitor
live services and technical support
Measure of the service performance and the
users' perception
Monitor the good functionning of the live
services they host
Responsible of the service hosting and
maintenance.
Enhance the organisation
Compare the adequation of the service
performance with the organisation needs
Compare the adequation of the service
performance with the users needs
Answer the users' expectations.
Tailor the development with a user-centric
development approach
Monitor the service perceived quality
Control the service performance, monitor
and correct anomalies.
Indicators linked to the service design
Develop the service
Marianne Laurent, Philippe Bretier, Carole Manquillet
Ad Hoc evaluations along the lifecycle of industrial spoken dialogues systems: heading to harmonisation?
[email protected], [email protected], [email protected] Labs, 2 avenue Pierre Marzin, 22307 Lannion, France