leveraging the crowd to support the conversation design ... · design cba need to include paypal...
TRANSCRIPT
Author KeywordsIterative design; Conversationdesign; Chatbot design pro-cess; Early-stage design sup-port; Crowdsourcing
CCS Concepts•Human-centered computing→ Systems and tools for in-teraction design; Empiricalstudies in interaction design;User interface design;
Leveraging the Crowd to Support theConversation Design Process
Yoonseo ChoiSchool of ComputingKAISTDaejeon, South [email protected]
Nyoungwoo LeeSchool of ComputingKAISTDaejeon, South [email protected]
Hyungyu ShinSchool of ComputingKAISTDaejeon, South [email protected]
Jeongeon ParkSchool of ComputingKAISTDaejeon, South [email protected]
Toni-Jan Keith MonserratInstitute of Computer ScienceUniversity of the Philippines LosBañosLos Baños, Laguna, [email protected]
Juho KimSchool of ComputingKAISTDaejeon, South [email protected]
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
Copyright held by the owner/author(s).CHI’20,, April 25–30, 2020, Honolulu, HI, USAACM 978-1-4503-6819-3/20/04.https://doi.org/10.1145/3334480.XXXXXXX
AbstractBuilding a chatbot with human-like conversation capabilitiesis essential for users to feel more natural in task comple-tion. Many designers try to collect human conversation dataand apply them into a chatbot conversation, aiming that itcould work like a human conversation. To support conver-sation design, we propose the idea of inviting the crowd intothe design process, where crowd workers contribute to im-proving the designed conversation. To explore this idea, wedeveloped ProtoChat, a prototype system that supports aconversation design process by (1) allowing the crowd toactively suggest new utterances based on designers’ pre-written design and (2) visually representing crowdsourcedconversation data so that designers can analyze and im-prove their conversation design. Results of an exploratorystudy indicated that the crowd is helpful in providing insightsand ideas as designers explore the design space.
IntroductionWhen building a chatbot, designers aim to embed human-like conversation capabilities so that users feel more naturalin completing their tasks with a chatbot. When designing aconversation, many designers attempt to collect from smallto large scale human conversation data so that their chatbotmodels a desired human conversation. Existing approacheseither collect conversation data from humans or formulatethe conversation by analyzing existing data sources. Ex-
amples include Wizard-of-Oz [7, 4], workshops [5], Twitterconversation data [3, 8], mail threads of DBpedia [1], andexisting chatbot logs [9].
To understand the challenges designers face during theconversation design process, we conducted semi-structuredinterviews with two professional conversation designers(at least 1 year of experience) and seven amateur design-ers with prior experiences in conversation design. Resultsshow that designers find it difficult to discover possible con-versations in the design process, especially when they arenot familiar with the chat domain. Also, it is overwhelmingto rapidly iterate the conversation design as the iterativeprocess requires not only the design of the conversationbut also prototyping and testing the working chatbot, whichcan be built using frameworks such as Google Dialogflow 1,BotKit 2 and Chatfuel 3. Supporting the iterative design ofconversation helps designers to get a sense of how poten-tial users might follow the current design of conversation.Yet, there has not been much investigation on how design-ers iterate the conversation design, challenges within theprocess, and how to design a system to support the pro-cess.
To address these challenges, we suggest crowdsourcingas a solution to supporting iterations in the design pro-cess. Inviting crowd workers in the design process poten-tially lessens the burden of designers to quickly test andimprove the design. There have been studies that incorpo-rated crowdsourcing in real-time chat scenarios. Chorus [6]demonstrated how the crowd could come up with not onlya diverse set of responses but also a diverse set of varia-tions of descriptions on a given topic, where they expected
1https://dialogflow.com/2https://botkit.ai/3https://chatfuel.com/
crowdsourcing as a potential approach to explore diverseconversations in the chat domain. Extending this line ofwork, we aim to utilize the crowd during the conversationdesign process, not during the chat session. We believethat the crowd could empower the iterative design processby enabling rapidly testing intermediate designs. By testingwith the crowd, designers can easily test their design in alightweight manner compared to Wizard-of-Oz, a workshop,or a lab study.
To explore the idea of using crowdsourcing in the conversa-tion design process, we developed ProtoChat, a prototypesystem for supporting designers by (1) allowing the crowdto actively suggest new utterances and turn-taking conver-sation based on designers’ scenarios and (2) representingcrowdsourced data in effective ways to assist designersanalyze and improve the conversation. We decided to setthe role of the crowd as the active suggester of conversa-tion design so that designers could get enough evidence indecision making. We first designed the crowd-testing inter-face to test the designer’s draft conversation with the crowd.Crowd workers are provided with the interface to either fol-low the designed conversation by responding or suggestingnew messages between the conversation. Other than thecrowd-testing interface, we designed a separate interfacefor designers where they can design the conversation flowwith a set of chatbot utterances and their correspondingtopics. Furthermore, after crowd-testing, designers couldreview and analyze the collected responses from the crowdwith the designer interface to iteratively improve their de-sign.
Through our exploratory study with four conversation de-signers, we found that ProtoChat enables designers toquickly design and test their ideas about a conversationwith the crowd and analyze the data provided by the crowd
Design
Need to include PayPal option
Remove event topic
Add option for options
Crowd-test Review
Collective view
Individual view
Figure 1: Conversation design process with ProtoChat. Designers can design a conversation, test with the crowd, and browse and analyze thecrowdsourced conversation.
so that they could improve the design. The crowd enabledneed-finding of the chat domain which is important in theearly stage of conversation design. Moreover, the crowdprovided evidence for decision making regarding later ver-sions of the design. This research aims to help chatbot de-signers with crowd power, from collecting conversation sce-narios by crowd contribution to effectively representing thecrowdsourced output, which enables designers to explorepotential scenarios, important in understanding the needsand use cases of real users.
c
b
a
Figure 2: Chabot’s turn on the crowdinterface. The crowd could either (a)proceed conversation with the designedscenario, (b) insert a new scenario, or (c)follow new scenarios that the crowdcreated.
This work is based on our ongoing research [2], in whichwe introduce the ProtoChat system and its preliminaryevaluation. In this workshop paper, we briefly summarizethe system and our findings so far, and discuss the use ofcrowdsourcing as a method for helping the conversationdesign process. Through our work, we expect to promotea discussion in the community of CUI (Conversational UserInterface) researchers about applying crowdsourcing fordata collection, decision making support, and design spaceexploration and confirmation in a broad set of CUI applica-tion scenarios.
ProtoChat: Crowd + Conversation DesignProtoChat supports a conversation design process by (1)allowing crowds to actively suggesting the design based on
designers’ scenarios and (2) providing designers effectiverepresentation of crowdsourced data to assist designers an-alyze and improve the conversation. The system consists oftwo interfaces: the designer interface and the crowd-testinginterface.
The designer interface supports designers to design low-fidelity conversations, plan how to test the conversationdesign, review the crowdsourced data, and refer to the pre-vious version design and analysis as well. Designers candesign a conversation with ‘topic’ and ‘utterance’, two ba-sic building blocks of designing a conversational flow. Topicrefers to what kind of questions need to be asked in the do-main (e.g. Payment) and utterance refers to how the topicis addressed within the conversation (e.g. "How would youlike to pay?"). Three main features are provided in designerinterface: Draft, Review (See Figure 1-1, 3), and History.In the Draft page, designers can create a conversation bydefining a set of topics and utterances (See Figure 1-1).Designer could also configure parameters and launch cus-tomized tests. In the Review page, designers can browseand review the crowdsourced conversations with collec-tive view and individual view (See Figure 1-3). Additionally,ProtoChat provides the History page, where designers canreview their previous design versions and test settings.
The crowd-testing interface is a separate auxiliary inter-face to quickly test the designed conversation. Crowds areguided to proceed the conversation with the designers’ con-versation flow in the form of chatbot, with pre-instructionsprovided. The crowd-testing interface(See Figure 1-2) hasa unique feature that enables the crowd to either proceedconversation with responding to pre-written utterances ofa chatbot or insert new conversations within the existingconversation flow (See Figure 2) to elaborate on the conver-sation. If the crowd chooses to insert a new conversation, itwill be performed as a self-dialogue.
Evaluation
Day 1 Day 2 Day 3
Pre-Interview
Post-Interview Final Interview
Crowd-Test
Design
Export
Design
Export
Review Review
Post-Interview
Crowd-Test
Design
Export
Review
Pre
Main
Post
Figure 3: The procedure of the 3-dayexperiment.
(Day 1) Create a conversation flow to
successfully guide users to finish the task,
in the form of chatbot.
(Day 2, 3) Revise their previous version of
conversation design after reviewing and
analyzing crowd data in the Review phase.
Crowd-test
Design
Review
Export
Configure test settings after designing the
overall conversation flow and run the test.
The testing method was fixed to MTurk.
Browse and analyze the data crowdsourced
by the Crowd-testing interface. Participants
could move back-and-forth between the
individual and the collective view.
Export the design into a hi-fidelity design
prototype each day. By requiring
participants to export their conversational
flow into a design prototype that supports
end-to-end scenarios, they were able to
recognize whether their ongoing design
needs more editing or iterations.
Figure 4: Detailed explanation of eachtask performed in the experiment.
Experiment procedureThe experiment focused on exploring the role and contri-butions of the crowd in conversation design. To examineProtoChat’s role during the overall design process, weconducted a 3-day long experiment. We recruited 4 de-signers (3 female, 1 male) who work on research topicsinvolving conversation design and have prior experienceof chatbot conversation design. Participants received $45for the whole experiment. During the experiment, we askedparticipants to mainly work on three tasks: 1) Design, 2)Crowd-test, and 3) Review the conversation for each day sothat they work on three iterations. The overall procedure ofthe 3-day experiment is shown in Figure 3. Figure 4 showsdetailed explanation of each task performed in the overallexperiment.
ResultParticipants chose different domains for their conversationdesign (P1: movie reservation, P2: ice cream order, P3:hotel reservation, P4: house fixing). We found that severalinteractions supporting conversation design were enabledby crowd power. With the current version of crowd-testing
interface, we were able to observe two types of crowd con-tribution toward the design of chatbot conversation.
Need-finding of a new conversationWith crowd-testing, participants were able to observe boththe desired flow of conversation by reviewing the collec-tive view and the needs from the crowd by analyzing indi-vidual chat logs. Specifically, participants were able to doneed-finding within the context of a conversation. Need-finding includes collecting responses about specific top-ics/questions or collecting new questions that can be askedfor a more natural conversation. They agreed that the re-sponses collected from the crowd helped them discoveruser needs and make internal decisions for next step de-signs. For example, P1 and P4 mentioned that based onthe responses from open-ended questions, they were ableto make UI decisions in the Export step such as decidingbetween button UI or free-formed question, or even the con-tent of answer format. P2 realized that they missed out onthe option between choosing a cup or cone, after seeingthe new crowd-added utterance – “A cup or a cone?”, whichthey felt was an essential topic that needs to be dealt duringan ice cream order.
Design confirmation with majority votingWhen participants tried to make decisions about their de-sign, they checked the path that the majority of crowds fol-lowed in the Sankey diagram during the Review phase andverified whether their conversation flow makes sense andis easy to follow. Based on the the majority responses ofthe crowd in the chatbot’s each utterance, participants wereable to refine their utterances during the iterative process,not the overall sequence or the order of topics. Detailedrevisions were made such as changing the tone of the botutterances (e.g. format of the questions), but the overallsequence of topics was similar for multiple iterations.
Discussion and Future WorkIn the workshop, we want to discuss ways in which crowd-sourcing could be applied in CUI research and ways tomake it more effective.
Further investigation toward realistic conversation designIn a real-world setting, it is hard to run iterations in early-stage chatbot design process, as it involves prototypingand testing a working chatbot with potential users. P2 men-tioned that “This tool helps in collecting feedback for myown design with a large number of crowds in a lightweightmanner, which enables experiencing a quick iterative pro-cess.”. We believe that the design space of supportingiterative design of conversation needs to be further inves-tigated, especially by focusing on how to support more di-verse forms of conversations. For example, our system didnot support branching interactions in a conversation, whichis one of the fundamental building blocks in chatbots. Also,it is important to investigate not only the iteration of theconversation design but also the whole iteration of chatbotbuilding process that incorporates the iterative conversationdesign.
Supporting branching interactions with the crowdDesigners pointed out that branching out the conversationthreads based on user input would be necessary to sup-port a more realistic conversation design than the currentversion. For example, if a chatbot asks the user “Wouldyou like to order snacks?” then an adaptive conversationflow can respond differently based on user input between“yes" or “no". With increased flexibility of the role of crowdworkers such as providing adaptive responses with spe-cific intents, we envision the crowd-driven expansion ofthe scenario tree with moderate designer intervention. Thebranching support can increase the the level of complexityin conversation design that our system can handle.
Collecting alternative expressionsBy observing the ability to collect diverse topics of conver-sation within each chat domain, we wish to improve thesystem to collect diverse alternatives for a specific expres-sion. Collecting various alternatives with the same meaningis crucial because those expressions will be able to providea more dynamic, natural conversation experience to users,by displaying utterances of the same meaning in differentways. For instance, designers can build a chatbot that givesdifferent responses every time even if the user asks similarquestions. Plus, the collected conversation can be used astraining data to build a machine learning model that pow-ers a human-like chatbot. One possible way of collectingdiverse expressions would be to improve the promptingmechanism in the crowd-testing interface, where the crowdis provided with questions that ask them to paraphrase thesentence that they just provided.
All in all, we believe that exploring how crowdsourcing canbe applied in CUI design can lead to constructive discus-sions and novel research directions. In the workshop, weplan to further discuss challenges of using the crowd in CUIdesign and effective ways to address them.
AcknowledgmentsThis work was supported by Samsung Research, SamsungElectronics Co.,Ltd.(IO180410- 05205-01).
REFERENCES[1] Ram G. Athreya, Axel-Cyrille Ngonga Ngomo, and
Ricardo Usbeck. 2018. Enhancing CommunityInteractions with Data-Driven Chatbots–The DBpediaChatbot. In Companion Proceedings of the The WebConference 2018 (WWW ’18). International WorldWide Web Conferences Steering Committee, Republicand Canton of Geneva, Switzerland, 143–146. DOI:
http://dx.doi.org/10.1145/3184558.3186964
[2] Yoonseo Choi, Hyungyu Shin, Toni-Jan Keith PalmaMonserrat, Nyoungwoo Lee, Jeongeon Park, and JuhoKim. 2020. Supporting an Iterative ConversationDesign Process (To appear). In Extended Abstracts ofthe 2020 CHI Conference on Human Factors inComputing Systems (CHI EA ’20). Association forComputing Machinery, New York, NY, USA, ArticlePaper LBW1458, 6 pages.
[3] Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You,Yufan Guo, Vibha Sinha, Jiebo Luo, and RamaAkkiraju. 2018. Touch Your Heart: A Tone-awareChatbot for Customer Care on Social Media. InProceedings of the 2018 CHI Conference on HumanFactors in Computing Systems (CHI ’18). ACM, NewYork, NY, USA, Article 415, 12 pages. DOI:http://dx.doi.org/10.1145/3173574.3173989
[4] Meng-Chieh Ko and Zih-Hong Lin. 2018. CardBot: AChatbot for Business Card Management. InProceedings of the 23rd International Conference onIntelligent User Interfaces Companion (IUI ’18Companion). ACM, New York, NY, USA, Article 5, 2pages. DOI:http://dx.doi.org/10.1145/3180308.3180313
[5] Rafal Kocielnik, Lillian Xiao, Daniel Avrahami, andGary Hsieh. 2018. Reflection Companion: AConversational System for Engaging Users inReflection on Physical Activity. Proc. ACM Interact.Mob. Wearable Ubiquitous Technol. 2, 2, Article 70
(July 2018), 26 pages. DOI:http://dx.doi.org/10.1145/3214273
[6] Walter S. Lasecki, Rachel Wesley, Jeffrey Nichols,Anand Kulkarni, James F. Allen, and Jeffrey P. Bigham.2013. Chorus: A Crowd-Powered ConversationalAssistant. In Proceedings of the 26th Annual ACMSymposium on User Interface Software andTechnology (UIST ’13). Association for ComputingMachinery, New York, NY, USA, 151–162. DOI:http://dx.doi.org/10.1145/2501988.2502057
[7] Archana Prasad, Sean Blagsvedt, Tej Pochiraju, andIndrani Medhi Thies. 2019. Dara: A Chatbot to HelpIndian Artists and Designers Discover InternationalOpportunities. In Proceedings of the 2019 onCreativity and Cognition (C&C ’19). ACM, New York,NY, USA, 626–632. DOI:http://dx.doi.org/10.1145/3325480.3326577
[8] Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, andRama Akkiraju. 2017. A New Chatbot for CustomerService on Social Media. In Proceedings of the 2017CHI Conference on Human Factors in ComputingSystems (CHI ’17). ACM, New York, NY, USA,3506–3510. DOI:http://dx.doi.org/10.1145/3025453.3025496
[9] Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum.2018. The Design and Implementation of XiaoIce, anEmpathetic Social Chatbot. CoRR abs/1812.08989(2018). http://arxiv.org/abs/1812.08989