Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 265–268July 29–31, 2021. ©2021 Association for Computational Linguistics

265

A Cloud-based User-Centered Time-Offset Interaction Application

Alberto Chierici, Tyeece Hensley, Wahib Kamran, Kertu Koss,Armaan Agrawal, Erin Collins, Goffredo Puccetti and Nizar Habash

Computational Approaches to Modeling Language (CAMeL) LabNew York University Abu Dhabi, UAE

{alberto.chierici,nizar.habash}@nyu.edu

AbstractTime-offset interaction applications (TOIA) al-low simulating conversations with people whohave previously recorded relevant video ut-terances, which are played in response totheir interacting user. TOIAs have great po-tential for preserving cross-generational andcross-cultural histories, online teaching, sim-ulated interviews, etc. Current TOIAs ex-ist in niche contexts involving high produc-tion costs. Democratizing TOIA presents dif-ferent challenges when creating appropriatepre-recordings, designing different user sto-ries, and creating simple online interfaces forexperimentation. We open-source TOIA 2.0,a user-centered time-offset interaction applica-tion, and make it available for everyone whowants to interact with people’s pre-recordings,or create their pre-recordings.

1 Introduction

Stories are the most pervasive medium people useto make sense of themselves and the surroundingworld. In the past twenty years, social media devel-opment has enabled people to document narrativesof their daily experiences online in unprecedentedmeasure (Page, 2013). TOIAs explore the nextevolution of narrative sharing devices exploitingadvances in artificial intelligence, as well as peo-ples’ increasing comfort with virtual communica-tion methods, to simulate face-to-face interactionsbetween humans and previously-recorded videosof other humans.

Democratizing time-offset interaction has manychallenges, including but not limited to: creatingappropriate videos, robust human-computer interac-tion (HCI) design, and lack of intuitive online inter-faces. We open-source TOIA 2.0, a user-centeredTOIA, and make it available for everyone whowants to interact with people’s pre-recordings, cre-ate their own, and researchers in dialogue, machinelearning, and HCI to gather original datasets.

2 Related Work

The usefulness of TOIAs has been demonstrated inmany practical scenarios. For example, for keepinghistorical memories (Traum et al., 2015), job inter-view practice for young adults with developmentaldisabilities,1 and building digital humans acrossdifferent industries. Storyfile, Typeform’s videoaskare some examples of commercial applications.2

TOIAs may also be reminiscent of virtual assistantslike Siri and Alexa and digitally animated charac-ters like Digital Humans,3 and Soul Machines;4

however, these are not authentic representations ofhuman beings which is TOIAs’ goal.

The general public cannot afford current TOIAdeployments due to their high production costs:creating a character (aka avatar) may require pre-recording about 2,000 video answers (Nishiyamaet al., 2016; Jones, 2005). Chierici et al. (2020)proposed a more streamlined avatar developmentprocess, but it is still impractical for the everydayuser: it involves transcribing and recording conver-sations based on brainstormed plausible utterances.Their work resulted in creating more than 400 pre-recordings and manual annotations that took sev-eral days. Research into time-offset interactionsneeds to generalize and to streamline the avatardevelopment process to make a mass use system.A first attempt made by Abu Ali et al. (2018) goestowards this direction and includes the possibil-ity to chat with the avatars in different languages.Their system implementation is not simple to usebecause it has two separate, non-communicatingcomponents for recording videos and interactingwith them. It also requires local installation anddoes not support multiple users.

1https://ict.usc.edu/prototypes/vita/2www.storyfile.com,www.videoask.com3www.digitalhumans.com4www.soulmachines.com

266

3 Design Principles

We designed TOIA 2.0 so that the time-offset in-teraction feels like a natural interaction to all users:the user who interacts with other people’s pre-recordings (henceforth, interactor), and the userwho records their own narratives (henceforth TOIAmaker). As per best practices in UX design, eachstep of interaction with the interface should mini-mize the user’s cognitive load and be psychologi-cally satisfying. We followed the human-computerinterface heuristics established by Nielsen (1994).Some design decisions to reach this goal included:the creation of a common visual framework acrossthe whole system for all types of users; use of famil-iar layouts and vocabulary; creation of affordancesby auto-generating suggested questions to answerand to suggest particular kinds of interactions; pro-viding psychological satisfaction with profiles thatemphasize social interaction; giving users agencyand autonomy over aspects of their TOIA expe-rience; and providing reassurance with repeatedlanguage and layouts.

An important extension to the previous workby Abu Ali et al. (2018) is the introduction of asocial network aspect. This does not just serve thegoal of creating a community of TOIA makers, butalso provides helpful feedback to them includingwhat additional questions are asked, what answersare liked or disliked. Another extension is givingthe TOIA maker control over which recordings areplayable individually, and as part of streams (i.e.collections or albums) that define different contextsand intentions for the interactions.

4 TOIA 2.0 System Architecture

The cloud-based centralized TOIA 2.0 system con-sists of six components, broadly speaking, that areinterconnected via a web server and componentAPIs (Figure 1). In the rest of this section, we dis-cuss the back-end components of TOIA 2.0. Wethen proceed to discussing the user interface whichis the highlight of this demo paper in Section 5.

TOIA Database The central repository of theTOIA 2.0 system is a relational database manage-ment system that stores all user information fromthe TOIA makers’ name, password, biographicaldescription, and language preferences, to links to,and meta-data of, all their video recordings, andtheir stream organization, as well as suggestedquestions.

Figure 1: TOIA 2.0 system architecture

Dialogue Manager The Dialogue Manager isimplemented as microservice that takes as inputthe ASR output of last utterance of the interac-tor conversing with a specific TOIA stream. Itreturns the video ID of the video recording con-taining the response. We use Chierici and Habash(2021)’s BERT q-q retrieval index as implementedby Haystack.5

Question Suggestion One of the most crucialsteps in creating a TOIA is building a knowledgebase consisting of questions and recorded videoanswers (pre-recording). Although users have com-plete creative freedom when it comes to creatingpre-recordings, it might still be tedious and chal-lenging to come up with and record hundreds ofresponses while trying to predict the hypotheticalpaths of future conversations.

We use transformer models such as GPT-2 togenerate personalized suggestions (Radford et al.,2018; Mishra et al., 2020). The generative pre-trained transformer fine-tuned on dialog-specificdata outputs a collection of question suggestions,given the pre-recordings maker’s profile settingsand the history of already recorded question-answer pairs. In that way, new personalized ques-tions can be dynamically generated and suggestedat any given point during the avatar creation pro-cess to any user given the same base model.

Web Server The web server is the main compo-nent that connects all the TOIA 2.0 parts. It man-ages information flow from front-end to back-endand vice versa.

5haystack.deepset.ai/

267

Figure 2: TOIA maker User Account

Figure 3: A TOIA maker recording a video answer tothe question “what’s your favorite food?”

5 User Interface

The user interface (UI) comprises a home page,login and sign-up screens, the recorder page, theplayer, and various menu and navigation aid items.While the TOIA 2.0 main page allows browsingprofiles of different public TOIA streams, all users(TOIA makers and Interactors) must register andhave an account to be able to make their own TOIAstreams or converse with public TOIA streams. Afull walk through the user flow is available on theapplication page.6 We next present specific compo-nents.

User Account Figure 2 presents the user accountfor a registered TOIA maker. The user is greeted bya message and shown statistics about the numberof people that interacted with their TOIA alongwith a notification of any new questions that wereautomatically generated by the system or identified

6http://toia.camel-lab.com/

though failed interactions (e.g., when an answerwas not found with high confidence).

Streams To the mid right of the figure, there is alink to a management page for the streams associ-ated by the TOIA maker. Each stream can get itsown profile that specifies the functional purpose ofthe stream, e.g., a business stream may target jobinterviews, while a family stream can focus on shar-ing family histories. Streams can be made publicor be only shared with specific users. All viewablestreams can be accessed on the Talk to TOIA page(see Figure 4).

Videos The bottom half of Figure 2 shows a col-lection of recorded videos, and some entries withquestions and no recordings. New questions arepresented as videos that have no content and arewaiting to be filled. The TOIA maker can deleteexisting video entries or suggested questions, orclick or Edit to record a response or change an ex-isting response. The TOIA maker can also create acompletely new video by clicking on the big plussign to the bottom right of the figure. Edit and Addactions will take the TOIA maker to the recorderview. The checkboxes at the top right corner of thevideo allow the TOIA maker to select a number ofrecordings and assign them to a stream. All videosin the TOIA maker’s account can be filtered fordisplay using keyword search.

Recorder Figure 2 shows the recorder page,which is opened when a TOIA maker chooses toedit a video or create one from scratch. The main

268

Figure 4: A view of a number of public TOIA streams that are ready for interactors to converse with.

component of the recorder page is recording screenconnected to the user’s camera. The question (orprompt) is at the bottom of the screen. Automaticspeech recognition output is shown to the TOIAmaker, and they have the option of editing the text.To the left of the recording screen are a set of la-bels. First are the video recording type, e.g., fillervideo or Question & Answer video. Next is a tog-gle for making the video public. Finally a list ofavailable streams. All videos are automatically as-signed to the All stream. A video can be assignedto more than one stream. Creating a stream is aseasy as writing a new stream name. After recordinga video, the TOIA maker can test play it, rerecordit, or delete it.

The Player The player interface can be accessedthrough the Talk to TOIA page, which lists all thepublicly viewable streams which the interactor userhas access to (see Figure 4). The player interface isintentionally as simple as a Facetime or Skype inter-face: the interactor speaks, and the player plays anappropriate response from the list of stream videosas determined by the dialogue manager.

6 Conclusion and Future Work

We presented TOIA 2.0, a user-centered time-offsetinteraction application system, which we plan tomake publicly available to users – TOIA makersand interactors – as well as researchers interestedin TOIA systems.

In the future, we plan to increase the robustnessof the system and its membership. We are also con-sidering additional enhancements to both the TOIAmaker and interactor’s experience, e.g., providingsemi-automatically generated videos or guessinganswers on behalf of a TOIA maker given theirhistory. In all cases, we keep the TOIA maker incomplete control of their recordings and streams.

ReferencesDana Abu Ali, Muaz Ahmad, Hayat Al Hassan, Paula

Dozsa, Ming Hu, Jose Varias, and Nizar Habash.2018. A bilingual interactive human avatar dialoguesystem. In Proc. of SIGdial Meeting on Discourseand Dialogue.

Alberto Chierici and Nizar Habash. 2021. A view fromthe crowd: Evaluation challenges for time-offset in-teraction applications. In Proc. of the Workshop onHuman Evaluation of NLP Systems.

Alberto Chierici, Nizar Habash, and Margarita Bicec.2020. The margarita dialogue corpus: A data set fortime-offset interactions and unstructured dialoguesystems. In Proc. of Language Resources and Eval-uation Conference.

Karen Sparck Jones. 2005. Some Points in a Time.Computational Linguistics, 31(1).

Shlok Kumar Mishra, Pranav Goel, Abhishek Sharma,Abhyuday Jagannatha, David Jacobs, and HalDaume. 2020. Towards automatic generation ofquestions from long answers. arXiv:2004.05109.

Jakob Nielsen. 1994. Enhancing the explanatory powerof usability heuristics. In Proc. SIGCHI Conferenceon Human Factors in Computing Systems.

Masashi Nishiyama, Tsubasa Miyauchi, HirokiYoshimura, and Yoshio Iwai. 2016. Synthesizingrealistic image-based avatars by body sway analysis.In Proc. of International Conference on HumanAgent Interaction.

Ruth E. Page. 2013. Stories and Social Media: Identi-ties and Interaction. Routledge.

Alec Radford, Karthik Narasimhan, Tim Salimans, andIlya Sutskever. 2018. Improving language under-standing by generative pre-training. Preprint short-url.at/fzBC4.

David Traum, Andrew Jones, Kia Hays, Heather Maio,Oleg Alexander, Ron Artstein, Paul Debevec, Ale-sia Gainer, Kallirroi Georgila, Kathleen Haase, et al.2015. New dimensions in testimony: Digitallypreserving a holocaust survivor’s interactive story-telling. In Proc. of International Conference on In-teractive Digital Storytelling.