■ Customer service is defined as the ability of acompany to afford the service requestor withthe expressed need. Due to the increasingimportance of service offerings as a revenuesource and increasing competition among serv-ice providers, it is important for companies tooptimize both the customer experience as wellas the associated cost of providing the service.For more complex interactions with higher val-ue, mixed-initiative systems provide an avenuethat gives a good balance between the twogoals. This article describes a mixed-initiativesystem that was created to improve customersupport for problems customers encounteredwith their appliances. The tool helped call tak-ers solve customers’ problems by suggestingquestions aiding the diagnosis of these prob-lems. The mixed-initiative system improved thecorrectness of the diagnostic process, the speedof the process, and user satisfaction. The toolhas been in use since 1999 and has providedmore than $50 million in financial benefits byincreasing the percentage of questions thatcould be answered without sending a field serv-ice technician to the customers’ homes. Anoth-er mixed-initiative tool, for answering e-mailfrom customers, was created in 2000.

One of the more frustrating experiencesin calling an automated answering sys-tem for information or help is being

forced to listen to all the menu options andthen being directed through the system, some-times requiring a redial (only to start again atthe top menu option). These systems are ratherpopular with companies because they savemoney—the companies’ money, that is. How-

ever, in a competitive service industry, thesefully automated systems may not be the bestchoice for a service provider because customersatisfaction plays an increasingly importantrole. Therefore, more sophisticated tools areemployed that attempt to balance the need ofmaking the customer happy—and perhapsincreasing some direct tangible benefit at thesame time. This is where mixed-initiative sys-tems play an important role.

General Electric (GE) Consumer & Industri-al’s Appliances Division manufactures and sellsa wide range of home appliances. GE’s sub-sidiary, Advanced Services Inc. (ASI), providescustomer service call centers that help solvecustomer issues over the telephone and sched-ule field service visits when needed. One of thetelephone services ASI provides is a group ofmore than 300 field service call takers whoschedule field service personnel to visit cus-tomers’ homes. Around 1.4 million home vis-its are scheduled per year. The call takers’ pri-mary goal was to schedule service visits, notdiagnose problems with appliances over thetelephone. However, they were able to help thecustomer without sending a field service tech-nician in 3.9 percent of the calls. There was anopportunity to increase this percentagebecause about 20 percent of the time all thefield service representative who visited thehome needed to do was to inform the customerabout a particular setting or remedial action. Ifthis information could have been obtainedover the telephone, it would have saved timefor both the customer and the field service rep-resentative. However, training the call takers todiagnose appliances was difficult because ofthe large number of appliances that can be

Articles

SUMMER 2007 89Copyright © 2007, Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602

Appliance Call Center: A Successful Mixed-Initiative Case Study

William Cheetham and Kai Goebel

AI Magazine Volume 28 Number 2 (2007) (© AAAI)

serviced, the complexity of modern appliances,and the high turnover in the call takers. Thesolution for this was to create a software toolthat acts as a mixed-initiative assistant (Allen1999) for the call takers called Support the Cus-tomer (STC).

The STC system is just one part of the call-taking process. The full process is shown in fig-ure 1. In this process, the customer, in the bot-tom left, calls an 800 number provided forscheduling home visits of field service techni-cians. A customer call-tracking system (Agent1) receives the call, accesses a customer data-base to retrieve information, such as the cus-tomer’s address and registered products, thendisplays this information to the call taker whoactually answers the telephone. STC is Agent 2.Before STC was developed, the call takerswould answer the questions to the best of theirability using their personal knowledge, train-ing, paper manuals, frequently asked questionslists, and weekly paper flyers on new issues.

STC uses a case base, a rule base, and a decisiontree to assist the call takers in helping the cus-tomers. STC stores cases of problems and theirsolutions, a decision tree of questions that areused in a diagnostic process to differentiate theactual case from all others, and rules that canautomatically answer questions. After theappliance has been diagnosed, the results ofthe diagnosis are stored in the call-schedulingsystem, Agent 3. The call-scheduling systemcreates a planned routing for the field servicetechnician the night before the service call is totake place and suggests parts to stock on therepair person’s truck. The call-taking systemalso sends data about the customer to STC, andthe field service technician can provide feed-back on the success or failure of the service vis-it to STC.

The next section discusses STC in moredetail. Then the mixed-initiative issues of STCare described. The final section gives resultsfrom multiple years of usage for STC.

Articles

90 AI MAGAZINE

Retrieves userstatistics,

appliance, andwarranty

Information

Providesmixed-initiativedecision support

Schedulesfield visit

Agent 1

Call CenterOperator

Customer Field Technician

Agent 2 Agent 3

Call Tracking IngelligentSoftware

CallScheduling

Figure 1. Customer Support Process.

Support the Customer Application

The call-center tool is designed to help call tak-ers solve customers’ problems by suggestingquestions that can be used to diagnose the cus-tomers’ problems. The primary goals of the calltaker are the correct diagnosis of the appliance,speed of diagnostics, and customer satisfaction.The purpose of the software tool is to improvemetrics associated with these goals. In order todo this, there were multiple requirements forthe software tool. These requirements are sim-ilar to the principles of mixed-initiative userinterfaces described by Eric Horvitz (1999). Therequirements are (1) suggest correct resolutionsto customer issues; (2) suggest questions thatwill diagnose the problem; (3) increase thespeed of the diagnosis process; (4) increase usersatisfaction with the process; (5) explain to theuser why something was suggested; and (6)have the tool learn from its experiences and beable to adapt to a changing environment.

Related Work and Tool UsageCase-based reasoning (CBR) (Aamodt and Plaza1994) has been used to automate customer-support help desks by many companies such asCompaq (Acorn and Walden 1992) and Broder-bund (Watson 1997). Using CBR for customersupport (Simoudis 1992) has become of inter-est to many other companies with help desks.We used a CBR tool to assist GE ConsumerProducts customer-support personnel. STC wascreated using a CBR tool from Inference Cor-poration called k-commerce. (Inference was lat-er acquired by eGain.) K-commerce allowed formixed initiative between the call taker and theautomated assistant. It provided standard userinterfaces for the knowledge engineer and enduser (that is, call taker). The knowledge engi-neer interface included forms for creating cas-es and rules. During project development,Inference Corporation also released a tool forcreating decision trees. These tools greatlyreduced the development time for the projectand allowed us to focus on knowledge acquisi-tion instead of tool creation.

The k-commerce form for creating a newcase allows the knowledge engineer to enter atitle, description, multiple questions, and a res-olution for the case. The system selects themost relevant cases using two criteria. The firstcriterion is the degree of match between thetext description of the problem typed in by thecall taker and the text in the title and descrip-tion of the case. The second criterion is the per-centage of questions in the case that have beenanswered correctly. When the diagnostic

process is started (before any questions havebeen answered) the selection of the most rele-vant cases is based on the text match. Then, asquestions from the highest matching cases aresuggested and answered, the correct answersplay a larger roll in determining the most rele-vant cases. The appropriate case is selected byanswering the questions that differentiate thepossible cases. For our application we wantedto be able to suggest one and only one case asthe solution. A way to guarantee that thesequestions can differentiate every case in thecase base is to form a tree of questions over thecase base where each case is a leaf in the treeand each internal node in the tree is a ques-tion. We consider this approach as a hybridapproach that employs both decision tree andCBR methodology where the decision tree issimply the selection mechanism of the CBRsystem. Some problems can be represented bycases without creating a decision tree (FAQs arean example of these), but for others it is betterto have a decision tree created for them (any-thing with a significant diagnostic process usedthe decision tree approach). Figure 2 shows aportion of a decision tree where rectangles arequestions, the arrows are answers, and ovals arecases.

K-commerce also allows the creation of rulesthat can automatically answer questions. Anexample of a rule is, if the answer to the ques-tion “What type product is your appliance?” is“Refrigerator” and the third and fourth charac-ter in the model number are both “S” then theanswer to the question “What type of refriger-ator do you have?” is “Side-by-Side.”

User InterfaceThe current interface for STC is shown in figure3. The critical information tab at the top hasthe product line and model number, which areboth passed in from the call-taking system. Italso has the problem description, which istyped in by the call taker. The model group canbe determined by a rule that uses the modelnumber. The symptom is a keyword phrase thatis selected by the call taker. The questions tabhas a set of questions the call taker can ask thecustomer to diagnose the problem. The resultstab has a set of solutions. Selecting the correctresult is the goal of the process.

Application DevelopmentThe development of the STC system was a five-step process. These same five steps can be fol-lowed for deploying other applications of AI:first, standardize the process and knowledge;second, digitize the inputs and outputs; third,

Articles

SUMMER 2007 91

automate the process as much as possible;fourth, control the quality of the system; andfifth, leverage the system and knowledge forimproved impact.

Standardizing the call-taking processinvolved identifying all cases that can and can-not be solved on the telephone and determin-ing the correct questions to ask and the correctorder in which to ask them. Much of thisknowledge was tacit knowledge (that is, per-sonal experience of the call takers and engi-neers). Since different call takers would ask dif-ferent questions to diagnose the same problem,we formed teams of call takers and engineers todetermine what should be the correct cases,questions, and order of questions. The visual-izations of the decision trees were very usefulin reviewing and optimizing this knowledge.The knowledge was entered into the case baseonly after the engineers and call takers createdthese visualizations and safety, legal, quality,and consumer-service personnel validatedthem. This took the majority of the time need-ed to create the STC system.

Digitizing the inputs and outputs of the STCsystem was the next step. The call-taking sys-tem was modified to start up the STC systemand send relevant data (for example, modelnumber) for every telephone call. K-commercewas modified slightly so that it could acceptthe data. The call-scheduling agent, from fig-ure 1, was modified to accept a large amount ofdata from the STC system in addition to thedata it was receiving from the call taker. TheSTC data was captured every time the case base

was used to answer a call. The data includedstart time, end time, customer telephone num-ber, call-taker ID, type of appliance, a short textdescription of the issue, all questions asked byCBR, all answers given to these questions, thecase suggested, and whether this answer wasaccepted by the customer.

Automating the entire telephone conversa-tion was not possible with the state of naturallanguage processing technology, but a team ofa call taker working with the STC system caneffectively automate the application of thestandardized and optimized process. Anattempt was made to automate the applicationas a web-based customer self-service tool. Thiswas discussed in Cheetham (2003).

Controlling the quality of the system after itwas created was more expensive than the ini-tial creation but also continued to improve thesuccess rate. Initially, each week a case authorwould analyze the data in the call-schedulingsystem for every call that was taken for theweek using standardized database queries. Theauthor was looking for any trends and espe-cially any times a caller would not be satisfiedby a suggestion. Any trends or outstandingitems would be discussed in a weekly feedbackmeeting with the call takers. The result of thefeedback meetings would be a few changes inthe case base, decision trees, and rules. Thesechanges would be made immediately andreviewed in the next week’s meeting. The fre-quency of these reviews decreased as the casebase, decision trees, and rules stabilized.

Leveraging the STC system, knowledge in

Articles

92 AI MAGAZINE

Which compartment is too cold?

Fresh FoodFreezer

What is freezer temperature setting?

<= 3 > 3

Schedule service Lower temperature,wait 24 hours

Figure 2. Decision Tree for “Refrigerator too Cold.”

the case base, and data in the call-record data-base provided additional benefits that were notall foreseen when the first version of STC wascreated. One benefit was the ability to auto-mate the response to e-mail. It takes longer fora person at the call center to answer an e-mailmessage than a telephone message, and in2000 GE’s e-mail volume was growing rapidly.

We used Cisco’s e-mail manager tool to storereplies to frequently asked questions so thatthey can be reused on future similar questions.In order to select the appropriate reply, a set ofdecision trees was created where each node inthe tree is a rule about metadata regarding thee-mail. Each leaf node in the decision treespecifies a set of replies that can be used toanswer that specific type of e-mail message.These trees are similar but not identical to thetrees used by STC. The e-mail handler does notautomatically answer e-mail, it suggests previ-ous replies that can be reused for a person whothen just selects the correct response from alist of suggestions. The person can edit the sug-gested reply, create a new one, or send the sug-

gested reply unchanged with one button click.Figure 4 shows a case from the e-mail man-

ager. A customer e-mailed GE saying the cus-tomer had a 50-year-old refrigerator in perfectworking condition and asked if we would liketo purchase it. The reply to this e-mail wasstored in the case base and is now used abouttwo times per week. Customer-support per-sonnel either send this exact reply or edit it ifthe request was slightly different.

Understanding a free text e-mail message isdifficult for an automated system, so we hadthe decision tree use only metadata that wassent with the e-mail. The metadata was cre-ated by not allowing customers to send e-mail directly to us; we made them enter theinformation on a form on our web site. All e-mail addresses on our products were replacedwith web pages. In addition to the free text ofthe e-mail, customers needed to classify theirmessages based on a few pull-down menuson the web site and provide a telephonenumber. This self-classification of the e-mailcreated the metadata that made it much eas-

Articles

SUMMER 2007 93

Figure 3. Call Taker User Interface.

ier for an automated system to select similarcases.

The e-mail handling and STC tool worktogether. If we determine an e-mail would bebetter answered with a telephone call then wecan use STC to help the customer. A new groupof customer-support personnel was created.They do not answer calls. They call the cus-tomers themselves. We keep track of the suc-cess rate for each product/nature-of-problemcombination so we know the predicted chanceof solving the problem over the telephone. Ifthe customer does not answer the telephone atour first attempt, we can repeatedly try to con-tact customers with a high chance of successusing callback before replying with an e-mail.

Mixed-Initiative Issues of STCSTC is a mixed-initiative conversational CBRsystem that acts as an intelligent assistant forthe call taker. Together, STC and the call takerform a team in which each provides a neededstrength where the other has a weakness. Thissection describes how each mixed-initiativeissue (Tecuci, Boicu, and Cox 2007) wasaddressed in the STC application.

Task IssueThe task issue addresses the division of respon-sibility between the human and the agents. Todecide which tasks humans should do andwhich intelligent agents should do, one shoulddetermine where humans are more appropriatethan intelligent agents and where agents haveadvantages over humans. Humans are hard tobeat at common sense, interacting with otherhumans, and creativity. Intelligent agents, onthe other hand, are good at many things thathumans find cumbersome or tedious, such asfollowing a precise process, storing and retriev-ing large quantities of information, and math-ematical computations. While humans have amoderate cost to train and a moderate to highcost to perform a task, agents have a high costto program for a task but a low cost to imple-ment that program.

Tasks can be performed by a human (with-out an agent), an agent (without a human), ora human and an agent. Before computers, alltasks were performed without an agent. In therecent past, AI has been used to create systemsin which an agent operates without a human.However, there are many tasks that can benefitfrom a human performing a portion of the taskand an agent performing a different portion ofthe task. Tasks like this are good candidates formixed-initiative systems.

In a call center, the task of helping customersrequires interaction with the customer andcommon sense but also adherence to a processand accessing a large amount of information ina timely fashion. The call takers are good at thenatural language processing that is needed tointeract with the customers, but it is difficultfor them to store and correctly retrieve thelarge amount of technical information that isneeded to help the customer. Luckily, STC isvery good at storing and retrieving this infor-mation even though it cannot do any naturallanguage processing. The user (call taker) per-forms the interactions with the customer, andthe agent (STC) stores and retrieves the stan-dardized knowledge about diagnosing appli-ances. Before this system was fielded, much ofthe information needed to diagnose a problemwas in manuals and tacit knowledge of a fewcall takers. Customers would often be placedon hold while the call takers looked for the cor-rect manual and then searched for the perti-nent information. Having an agent that couldstore this information, provide it automatical-ly, and guide the call taker in the diagnosisremoved many frustrating delays.

Another task that the agent can perform isautomatically answering the questions that itis confident it can answer correctly. Some of

Articles

94 AI MAGAZINE

Figure 4. E-mail Example Case.

these questions include information that canbe obtained from other sources than the cus-tomer. Information about the customer, suchas models owned and previous diagnostics ses-sions, can be stored in a database. When ques-tions require this information, it can beanswered automatically. One future goal forhigh-end appliances is to include a telephoneor Ethernet connection so additional diagnos-tic data, such as measurements from a temper-ature sensor, can be sent directly to the diag-nostic tool, allowing it to take initiative on awider set of questions.

Control IssueThe control issue addresses the shift of initia-tive and control between the human and theagents. This primarily focuses on how theagents should show initiative, including proac-tive behavior. Humans often show initiativeand can generally strike a good balance in judg-ing when initiative will be useful and when itwill not be useful. To that end, humans con-sider a range of guiding principles including:(1) Have an understanding of the goals and pri-orities. (2) Have an understanding of the cur-rent situation. (3) Identify a task from the cur-rent situation that can help goals. (4) Identifypotential benefits and problems from doing atask. (5) Determine the confidence at which atask should be performed (high—do task,low—do not do task, and medium—ask if taskshould be done). (6) Have the ability to per-form a task. (7) Inform others that a task hasbeen done.

Items one, two, and three are needed to cor-rectly identify opportunities for taking initia-tive. Item four gathers information about theadvantages and disadvantages of taking the ini-tiative. Care must be taken to identify the dis-advantages. Possible disadvantages includebothering others, doing an unwanted task,doing a low-priority task instead of a high-pri-ority one, and being destructive or wasteful.

The fifth item is weighing the advantagesand disadvantages. If the advantages greatlyoutweigh the disadvantages then the initiativeshould most likely be taken. If the disadvan-tages outweigh the advantages then the initia-tive should not be taken. If neither the advan-tages nor the disadvantages outweigh the otherthen a human can ask another person (forexample, a supervisor) if the initiative shouldbe taken. The human needs the ability to per-form the task and should inform others affect-ed that the task has been completed.

Software agents can use the same set ofguidelines when they attempt to show initia-tive. They should understand the goals, priori-

ties, current situation, and set of possible ini-tiative actions. They should be able to identifyadvantages and disadvantages. Disadvantagesof an automated assistant taking inappropriateinitiative include bothering the user, lockingthe user out (stealing cycles), acting in anunknown way, undoing the user’s desiredactions, making the user undo the agent’sactions, and keeping the user from doing tasks.

The agent should be able to determine theconfidence that the initiative should be taken(Cheetham and Price 2004). Then, the agentexecutes the action with highest confidenceand informs the user of the action taken.

Since speed of call is a requirement in STC,we do not want to have the call taker ever waitfor the agent. All of the agent’s actions takeplace so quickly that they appear to be instan-taneous to the call taker. This is also importantfor the customer’s satisfaction, because we donot want to have the customer wait for theagent.

Finally, the action suggested after diagnosisis completed can be automated. The call takercould step the user through a precreated repairprocess or e-mail or fax the process to the user.If parts are required, the agent can place anorder for the parts to be sent to the address ofthe customer. If a service technician is required,the time for that visit can be scheduled and adescription of the problem automatically sentto the call-scheduling system.

Awareness IssueThe awareness issue addresses the maintenanceof a shared awareness with respect to the cur-rent state of the human and agents. This canbe difficult. The humans and agents have com-pletely difference senses. Humans have sight,hearing, touch, taste, and smell. Any informa-tion gathered with these senses will need to berelayed to the agent. Agents have access to dig-itized data sources such as sensors or databases.Any pertinent information discovered by theagent needs to be relayed to the human.Humans have experiences and knowledge thatcan be used to make conclusions from existinginformation. Agents may include an expert sys-tem that also makes conclusions, most likely ina different way from the human.

One good way to keep a shared awareness isto share three types of information: facts andbeliefs, reasoning, and conclusions. To obtain aclear understanding of an agent’s awareness, allthree information types are needed by thehuman. Showing facts and beliefs allows thehuman to check for any that are not correct,current, or consistent with the human’s factsand beliefs. The conclusions are the value that

Articles

SUMMER 2007 95

the agent produces and need to be shown tothe human. The reasoning is the bridgebetween the facts and beliefs and the conclu-sion. It can be difficult to share the reasoningof black-box techniques, such as neural nets.Therefore, black-box techniques should beused in mixed-initiative systems only whentheir reasoning does not need to be inspected(for example, this process does not need to beaudited or is guaranteed to be correct). Twoother rules of thumb on having a shared aware-ness are (1) do not make the agents guess at thesituation, as incorrect assumptions can causemany problems in a mixed-initiative system,and (2) allow the human to inspect and changethe agent’s facts and beliefs.

For STC, in order for the agent to be able tomake valid suggestions, it needs to have aware-ness of all the information that the call takerhas about the problem. This requires the STCsystem to have all possible questions that thecall taker can ask and all possible answers tothose questions. We quickly found that eachquestion needed an answer of “unknown” andthat the agent needed to be able to correctlydeal with these answers. For instance, the cus-tomer may not be able to answer the question“What is the model number of your appli-ance?” This question also led to the creation ofa model number locator agent that can be usedto locate the model number on any appliance.

The user must also have awareness aboutwhat the agent is doing. Since the agent cantake the initiative to answer questions, the usermust be able to inspect the conclusions thatthe agent has made. In STC, all questions thatthe agent has answered automatically are listedfor the agent so they can be easily reviewed,but answering them does not interrupt theuser. It is also important that the call taker beable to change the answers when a question isanswered automatically. The user can click onthe question that the agent has answered andthen select a different answer. A system admin-istrator can also inspect the facts and rules thatcaused a question to be automaticallyanswered.

Communication IssueThe communication issue addresses the proto-cols that facilitate the exchange of knowledgeand information between the human and theagents. These protocols include mixed-initia-tive dialog and multimodal interfaces.

The communications from human to agentin a mixed-initiative system need to be as effi-cient as possible for the human and as com-plete as possible for the agents. These are oftencompeting goals. The human has limited time

to update the awareness of the agents. Onegoal of a mixed-initiative system is often tosave time for the human and having thehuman spend a large amount of time or effortupdating the agents’ awareness defeats thispurpose. However, the agents need as completean awareness of the situation as possible inorder to provide assistance to the human.

The communication from the agents to thehuman needs to be easy and unobtrusive forthe human, but the human also needs to haveaccess to all information that could be valu-able. If the communication is not easy andunobtrusive for humans, they will be unlikelyto use the system because the benefits to themdo not outweigh these costs. One way to makethe communications less obtrusive is to reducethe false alarm rate (that is, the percentage ofcommunications when the agent needlesslycommunicates to the human). These needlesscommunications just bother the human. Onereal deployed example of possibly bothersomecommunication is a voice-based warning sys-tem for the F-16 combat aircraft. The femalevoice tells the pilot to “pull up” from a dive or“warning, warning” when the pilot needs topay attention to various panel displays (thenagging female voice got nicknamed “BitchingBetty” by pilots).

For STC, the primary communication fromthe agent to the user is a suggested question. Itwould be bad if there was only one questionsuggested and it was not correct for the givensituation. Multiple methods can be used toreduce the cost of an incorrect suggested ques-tion. Some of these are providing multiple sug-gestions from which the user can pick, notforcing the user to act on the suggestion, andunobtrusively presenting the suggested ques-tions to the call taker.

The primary communications from the userto the agent are the answers selected for thequestions. The set of answers to a question iscreated so that each answer is appropriate for adifferent case. For instance, we do not ask foranswers that are continuous variables like tem-perature. We do ask for ranges of temperatureswhere each range would have a meaning asso-ciated with it. This way no postprocessing isneeded to turn the answer into a diagnosticfact.

Call takers and customers often wonder whythe system is suggesting a specific question.User trust is enhanced if there is a clear expla-nation for why the system is taking someaction. When the questions are created, weoften also create an explanation for why thequestion should be asked. This explanation canbe displayed for the call taker by clicking on

Articles

96 AI MAGAZINE

the question in the user interface. Creatingthese explanations in CBR systems can be dan-gerous. One project to create automated expla-nations produced explanations that were notalways appropriate for the customer. For exam-ple, the explanation for why we are asking thequestion “Is the power cord plugged in?” was“We are trying to confirm the problem cause‘Idiot User.’”

Evaluation IssueThe evaluation issue addresses the human andagent contribution to the overall system’s per-formance. Evaluation of a mixed-initiative sys-tem involves evaluation of the users, agents,and the agent/user interaction. One field ofstudy for evaluating user (that is, employee)performance is called human resource man-agement (HRM). The basic premise of HRM isthat humans should not be treated likemachines. In the evaluation of an agent, itwould also be good to evaluate the agent lesslike a machine and more like a person. In acompany setting, HRM evaluates employees ontwo major areas: their individual abilities andhow they work with others. The agent shouldbe evaluated on both its individual abilitiesand how it interacts with the user. An agentwill not be as effective as possible unless it iscompetent in both of these areas. The individ-ual effectiveness of the agent can be measuredwith quantitative indicators such as speed (sec-onds), coverage (number of tasks the agent cancomplete), false alarm rate, and accuracy ofresults. However, the quality of its user interac-tions will probably need to be evaluated withqualitative indicators such as user satisfaction,ease of interaction, trustworthiness, friendli-ness, manners, and value provided to the user.

A mixed-initiative system can be evaluatedon its speed, cost, and correctness in perform-ing its tasks. Different systems will place differ-ent importance on each of these issues. In amixed-initiative system there is a question ofwho will do each task, the user or the agent.The simple answer to this is to have the onewith the lowest cost, highest speed, and high-est correctness perform the task. The agentalmost always has the highest speed, so itwould be good to focus on how to have theagent also have a low cost and high correct-ness. The cost of the agent performing a taskfor the user can be thought of as the sum of thecosts of the following five activities: (1) creat-ing the agent; (2) the user telling the agent todo the task; (3) the agent doing the task; (4) theagent communicating the result to the user;and (5) the user receiving and validating theresult of the agent.

All of these costs should be reduced to makethe agent as useful as possible. A simple way tohave the agent be as correct as possible is tohave it only do tasks (or specific instances of agiven task) where it is likely to be correct.

Figure 5 shows a generalized flow of controlthrough a mixed-initiative system. The processcan start and go to either a user or an agent.Then users and agents will work the task untileither the user or the agent ends the task. Somemixed-initiative systems may require either theuser or agent to start and end the task. Oneextreme of this flow of control is to have theuser start the task, work on the task withouthelp from the agent, and then end the task.This is the way many tasks are done before themixed-initiative system is created. The otherextreme is to have an agent start the task, workon the task without help from the user, andthen end the task. This is how many tradition-al AI systems operate. This would usually bethe lowest cost and highest speed approach,but there are many domains where the correct-ness of such a system would not be acceptable.So, the proper trade-off for a mixed-initiativesystem is to have the agent do all of the taskswhere it can have an acceptable correctness ata cost that is lower than the cost of the userperforming the task. This would move as muchof the task as possible away from the user andto the agent.

One way to evaluate job performance frompeople is with a three-level scale: (1) Poor—does not finish tasks correctly or on time. (2)Good—finishes tasks correctly and on time. (3)Excellent—finishes tasks correctly and on timeplus shows initiative to suggest or do othertasks.

Agents could be evaluated on a similar scale.This would require initiative from the agent inorder for it to be an excellent agent.

With STC, we found that call takers quicklylost trust in the system if it gave poor suggest-ed questions or initiative. That is why we wentthough the additional effort of creating a deci-sion tree on top of the case base. K-commerceper se does not require a decision tree; it canautomatically suggest questions from the mostlikely cases. However, at the start of the diag-nosis process, it is often unclear which cases arethe most likely ones. We created diagnostictrees for each possible symptom of each type ofappliance to improve the correctness of the sys-tem and retain the users’ trust. Figure 4 showsthe tree for a side-by-side refrigerator where acompartment is too cold.

The customers’ satisfaction is primarilybased on the correctness, speed, and profes-sionalism of the help provided by the call tak-

Articles

SUMMER 2007 97

er. We have already described how the systemhelps the correctness and speed of the process.The system also helps the professionalism ofthe call taker by providing the exact wordingthat should be used to ask a question of thecustomer. Call takers are given two weeks oftraining before they are able to take calls froma customer. This training used to involve edu-cation about the appliances and how to pro-fessionally interact with a customer. Now,training focuses mainly on professional inter-action, and the system acts as “on the job”training about the appliances.

Architecture IssueThe architecture issue addresses the designprinciples, methodologies, and technologiesfor different types of mixed-initiative roles andbehaviors. Horvitz (1999) described 12 criticalfactors for mixed-initiative systems. These fac-tors include considering uncertainty about auser’s goals, employing dialogue to resolve keyuncertainties, employing socially appropriatebehaviors for agent-user interactions, main-taining working memory of recent interac-tions, and continuing to learn by observing.

These are excellent design principles for amixed-initiative system. However, even withgreat architecture principles it is still difficult towrite a good mixed-initiative agent. For exam-ple, many people do not like the MicrosoftOffice assistant, sometimes called Clippy.

Clippy did have many good properties, but itwas removed from Microsoft Office because thevalue of its help was not greater than the costof the incorrect interruptions. Many users con-sidered a small number of incorrect interrup-tions or even a single incorrect interruption tobe too much. It was very difficult for Clippy togive correct and timely advice at all times.

However, there is another form of mixed-ini-tiative interaction taking place in MicrosoftOffice. The spelling and grammar checkers actas a simple mixed-initiative system. MicrosoftWord puts a wavy line under words that maynot be correctly spelled or may have inappro-priate grammar. The goal of this is to help theuser write with proper spelling and grammar.The user should usually want to have properspelling and grammar, so there is little uncer-tainty about the user’s goals. There is a dialogueavailable if the user clicks on the underlinedword. The dialogue is a pop up that includes aguess at the correct word, the ability to“ignore” the suggestion, which removes theline, the ability to “add” the word to a person-al dictionary, and a description of the gram-matical rule that caused this to be underlined ifit was a grammar issue. Underlining a word issocially appropriate interaction since it doesnot interrupt a user. Memory of the usage ofthe “ignore” and “add” options can allow thegrammar and spelling agents to continue tolearn the users’ preferences.

The STC architecture puts extra emphasis onthe maintainability of the system. Every timethe case base was used to answer a call, adescription of the call was written to a report-ing database. The description included starttime, end time, caller telephone number, call-taker ID, type of appliance, a short text descrip-tion of the issue, all questions asked by the sys-tem, all answers given to these questions, thefinal diagnosis suggested, and whether thisanswer was accepted by the caller. Each week acase author would analyze this information inthe reporting database for every call that wastaken for the week. The call takers would helpwith this analysis and were a valuable part ofthe development team.

Personalization IssueThe personalization issue addresses the adapta-tion of the agent’s behavior to the preferencesof its user. Agents can be more useful if they arepersonalized for the user. All users are not thesame, so it can be difficult for one agent tomake all users happy all of the time. One com-mon way to personalize the actions of an agentfor a specific user is to store information aboutthe user and use this information to determinethe type and timing for help the agent pro-vides. One other way for the agent and user towork together more effectively is to have com-mon shared experiences. This means the agentneeds to remember past experiences with theuser and use these to benefit the user. One sim-ple example of this is when the tool Matlab (orother software development environments)

Articles

98 AI MAGAZINE

start

end

useri agentk

Figure 5. User-Agent Interaction.

remembers past user commands and allows theuser to quickly select and reissue these com-mands. This has the effect of allowing the userto quickly say, “Do it again.” The past experi-ences can be both short-term and long-termexperiences.

There can be problems with personalization.An agent should be careful about makingassumptions about a user and assuming thesepreferences do not change. Preferences canchange over time or be misunderstood by theagent, so the agent should allow users to edittheir personal preferences. For example, twoyears ago one of the authors purchased twoDVDs on potty training for his daughter fromamazon.com. Amazon was still annoyingly rec-ommending potty-training books and videostwo years later. Fortunately, Amazon now doesallow users to improve their recommendationsby editing the list of items they have purchasedor previously rated. Another example is fromthe TiVo digital video recorder, which suggestsand takes initiative to record shows it thinks itsuser would enjoy. Many people have com-mented that from recording shows like Will &Grace and Queer Eye for the Straight Guy theirTiVo now thinks they are gay (Zaslow 2002).One person reportedly tried to teach his TiVothat he was not gay by recording many thingsa straight man would watch like World War IImovies and documentaries. Now the TiVothinks he is a Nazi.

STC has some limited personalization capa-bility. There is no personalization for the calltaker and some personalization for the cus-tomer. The ideal personalization scenario iswhen customers initialize their personal infor-mation by registering the products they pur-chase. Alternatively, call takers will update theinformation for the customer when the cus-tomer calls ASI. Information about customersand the appliances they own is stored in a data-base. This information is used by STC when itsuggests questions and solutions. In addition,past experiences of telephone conversations(questions asked and answers given) are alsostored. This is beneficial if, for example, a cus-tomer is forwarded from a first-level call takerto a supervisor. In that case, all answered ques-tions are visible to the supervisor. Having thispersonalized information readily availablereduces customers’ frustration that comes fromrepeating the same information. The sameprinciple allows personalized information to beshared between different call takers. If a cus-tomer calls multiple times, then previous ses-sions can be reviewed to get the background onthe current call. All questions and answers arestored for 30 days in one database (short-term

memory) and a brief summary of the call isstored permanently in another database (long-term memory).

ResultsSTC has been in constant use since the end of1999. More than 300 call takers at multiplelocations in the United States use the system.The percentage of calls that are correctlyanswered over the telephone has increasedeach year. Figure 6 shows the success rate foreach year. Frequent customer surveys showcustomer satisfaction is higher with cases thatare answered using STC than with calls that arenot.

This system has also been a financial successfor GE. The initial development in 1999 cost $1million for the software tool, Inference Corpo-ration professional services, two person yearsof effort by GE personnel, and the cost of hard-

Articles

SUMMER 2007 99

19990

5

10

15

20

25

2000 2001 2002

Year

Perc

enta

ge H

elp

ed

2003 2004 2005

Figure 6. STC Success Rate.

0

0.6

0.4

0.2

0.8

1

1.2

1.4

1.6Vo

lum

e (M

illio

ns)

1999 2000 2001 2002

Year2003 2004 2005

Figure 7. Call Volume.

William Cheetham is asenior researcher in theartificial intelligence lab-oratory of the GeneralElectric Global ResearchCenter in upstate NewYork, where he hasworked since 1985. Hisjob is to invent or

improve knowledge-based products andservices. This often involves the applicationof artificial intelligence and related tech-niques. He has led the development ofmore than a dozen intelligent systems thatare in use throughout the General ElectricCompany. Since 1998, he has been anadjunct professor at Rensselaer PolytechnicInstitute, where he now teaches the classApplied Intelligent Reasoning Systems.

Kai Goebel is a seniorscientist at RIACS, wherehe is coordinator of thePrognostics Center ofExcellence for NASAAmes Research Center.Prior to that, he workedat General Electric’sGlobal Research Center

in Niskayuna, New York, from 1997 to 2006as a senior research scientist. He has carriedout applied research in the areas of artificialintelligence, soft computing, and informa-tion fusion. His research interest lies inadvancing these techniques for real-timemonitoring, diagnostics, and prognostics.He has fielded numerous applications foraircraft engines, transportation systems,medical systems, and manufacturing sys-tems. He holds half a dozen patents andhas published more than 75 papers. Since1998, Goebel has been an adjunct professorof the Computer Science Department atRensselaer Polytechnic Institute (RPI), Troy,New York, where he has been coteachingclasses in soft computing and applied intel-ligent reasoning systems with WilliamCheetham.

ware to deploy the system. The main-tenance in each of the next six yearshas averaged $0.5 million for ongoingmaintenance of the cases and a majorupgrade to a new version of the eGainsoftware in 2004. The benefit of notsending a field service technician to acustomer’s home when the product isin warranty is $50 for GE. The dollarsavings for GE can be calculated by thefollowing formula

Savings = increase in success rate * call volume * $50

Figure 7 shows the call volume peryear. The sum of the savings for eachyear from 2000 to 2005 is $44.5 mil-lion. This is a project that can bothprovide better service for customersand reduce the cost of this service.

Other benefits include the follow-ing: (1) Higher first-call success rate.This increases customer satisfactionand decreases the number of calls GEneeds to handle. (2) Early identifica-tion of new types of customer prob-lems. This feedback can be sent todesign teams, who can fix the problemin future releases and reduce futurecustomer problems. (3) The ability toenforce policies such as “If the modelis sold by a particular retailer S, thenask if it was purchased at S.” S pays forthis service, but call takers often forgotto ask, and GE ended up shoulderingthe cost. (4) The increased ability tomail parts out with instructions andavoid a service call for items that can-not be fixed over the telephone but arean easy fix and the customers want todo it themselves. (5) STC identifiesparts needed when a field service tech-nician is sent to the home so more fix-es can be made on the first trip, savingtime for the customer and field tech-nician (the part needed is referred to as“intercept” in figure 3). (6) Theincreased consistency of call takers hasreduced repeat calls from customers“fishing” (a practice in which cus-tomers call repeatedly in hopes to getdifferent answers).

Our implementation of Cisco’s e-mail tool went into use in June of2000. The tool reduced the averagetime a person needs to answer an e-mail message by 40 percent. Since ouryearly projected cost to answer e-mailswas $1.5 million, the savings of thistool is $600,000 per year.

ConclusionGE Consumer Products has been suc-cessful using the STC mixed-initiativesystem to provide customer support.The system contains exhaustiveknowledge about appliance problemsand the questions needed to diagnosethem. The human call taker providessome knowledge about appliances anda professional interface between ourcustomers and the diagnostic knowl-edge stored in the tool. The team ofcomputer system and human call tak-er performs better than the call takershad before the system was created.

ReferencesAamodt, A., and Plaza, E. 1994. Case-BasedReasoning: Foundational Issues, Method-ological Variations, and System Approach-es, AICOM 7(1).

Acorn, T., Walden, S. 1992. SMART: SupportManagement Cultivated Reasoning Tech-nology for Compaq Customer Service. InInnovative Applications of Artificial Intelli-gence 4, ed. C. Scott and P. Klahr. Cam-bridge, MA: AAAI Press/The MIT Press.

Allen, J. 1999. Mixed-Initiative Interaction.IEEE Intelligent Systems 14(6): 14–16.

Cheetham, W., 2003. Lessons LearnedUsing CBR for Customer Support. In Pro-ceedings of the Sixteenth International FloridaArtificial Intelligence Research Society Confer-ence. Menlo Park, CA: AAAI Press.

Cheetham, W., Price, J. 2004. Measures ofSolution Accuracy in Case-Based ReasoningSystems. Paper presented at the SeventhEuropean Conference on Case-Based Rea-soning, Madrid, August 30–September 2.

Horvitz, E. 1999. Principles of Mixed-Initia-tive User Interfaces. In Proceedings of theConference on Human Factors in ComputingSystems, 159–166. New York: Associationfor Computing Machinery.

Simoudis, E. 1992. Using Case-Based Rea-soning for Customer Technical Support.IEEE Expert 7(5): 7–13.

Tecuci, G.; Boicu M.; and Cox, M. 2007.Seven Aspects of Mixed Initiative Reason-ing: An Introduction to this Special Issueon Mixed Initiative Assistants. AI Magazine28(2).

Watson, I. 1997. Applying Case-Based Rea-soning: Techniques for Enterprise Systems. SanFrancisco: Morgan Kaufmann Publishers.

Zaslow, J. 2002. What to Do When YourTiVo Thinks You’re Gay. The Wall Street Jour-nal, November 26.

Articles

100 AI MAGAZINE

Stay Informed with AAAI Member Announcements

If you are a member of AAAI andwould like to receive periodicannouncements from AAAI,please go to aaai.org/cgi-

dada/mail.cgi, choose AAAI-Members, enter your email

address, and select “subscribe.”You can also register

for the semimonthly AI Alert.