Human Improvised TheatreAugmented with Artificial Intelligence

Piotr MirowskiHumanMachineLondon, UKpiotr.mirowski@computer.org

Kory Wallace MathewsonUniversity of AlbertaEdmonton, AB, Canadakorymath@gmail.com

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

Copyright held by the owner/author(s).C&C ’19, June 23–26, 2019, San Diego, CA, USAACM 978-1-4503-5917-7/19/06.https://doi.org/10.1145/3325480.3326547

AbstractImprovisational theatre (improv) has been proposed as agrand challenge for general artificial intelligence (AI) [7].Current state-of-the-art conversational intelligence modelslack proper grounding, language understanding, and gener-ate meaningless meandering responses [8]. Utilizing themas improvised comedy partners (improvisors) is doomedto fail - curiously, this limitation makes their use particularlyappealing. Improv theatre celebrates risk taking and fail-ure by inviting performers to express themselves withouthesitation or fear of being judged [11]. Our installation is aninteractive improv workshop for a group of interested partic-ipants, culminating in a live public performance. Attendeesare invited to observe and interact with AI-based improvi-sational theatre technology. The workshop is facilitated bytwo improv theatre professionals with a combined 30 yearsof experience in teaching, training, and touring. The perfor-mance features various AI tools for augmented creativity.

Author KeywordsImprovisation; Theatre; Storytelling; Performance; ActorTraining; Chatbot; Language Models; Conversational AI

CCS Concepts•Human-centered computing → Interaction techniques;•Applied computing → Performing arts;

Session: Poster & Demo Reception CC ’19, June 23–26, 2019, San Diego, CA, USA

527

Motivation

Figure 1: Performers receivinglines from AI chatbot via headset.

Figure 2: Example of improvperformance (downstage) with oneimproviser receiving lines from anAI chatbot (upstage).

Figure 3: Example AI improvperformance technical setup.

Improvisational TheatreImprovisation (impro or improv ) is a complex theatrical artform modelled on natural human interaction and demand-ing constant adaptation to an evolving context; it has beenqualified as “real-time dynamical problem solving” [4, 10]in the settings of both jazz music and theatre. Improv re-quires performers to exhibit acute listening to both verbaland non-verbal suggestions coming from the other improvi-sors, split-second reaction, rapid empathy towards the otherperformers and the audience, short- and long-term memoryof narrative elements, and practiced storytelling skills [11].From an audience point of view, improvisors must expressconvincing raw emotions and act physically to reproducethe experience of a scripted play.

The Three Minds of an ImprovisorIndividual improvisors, or improv troupes, typically com-bine several behavioural channels (or “minds”) over thecourse of a performance. Upright Citizens Brigade teacherBilly Merritt introduced three concepts of pirate, robot andninja [15] to qualify players who fearlessly initiate new sceneswith “no idea of what will happen next” (pirates), playerswho use their sense of logic and acting skills to ground thescene into reality (robots), and players who support theimprovised storytelling by introducing characters and situa-tions necessary to move the story forward or by reincorpo-rating narrative elements to bring the story towards a con-clusion (ninjas). An alternative subdivision into “realms ofthe body” [16] could be seen in the practice of French actor,dancer and trainer Francois Delsarte (1811-1871), namelyinto head, heart and gut. In the context of improv, the headfocuses on storytelling, narrative elements, interest, hu-mor, analogy, metaphor, and reincorporation; the heart con-cerns itself with reacting truthfully in the moment [14]; andthe gut is the wild-card channel through which spontane-

ity emerges [1]. Good improvisers or improv troupes canbalance these three channels, effortlessly switching as theperformance progresses.

Using AI as an Actor Training Tool?AI models have been used for generating narrative struc-tures by playing a role similar to the head [3]: an AI sto-ryteller can be used in improv exercises where the actorsfocus on being in the scene (i.e. similar to a director callingedits). AI models have also been used for putting perform-ers with challenging and novel situations (gut) [12, 13]. Thelatter study [13] collected feedback of a large number of hu-man performers who qualified the AI conversational partneras an “X factor”. The pirate-like conversational AI forcedthem to take care of the narrative, and enabled them to fo-cus on the emotional aspects of the performance. TheseAI models can therefore serve challenging educational andinspirational tools for performance development.

Engaging in Human-Machine ConversationsTo learn how to perform alongside AI-based improvisors,we have developed several exercises which channel thegolden rule of improvisational theatre, saying: “Yes, and. . . ”The first exercises focus on conversational dialogue. Weuse several conversational AI-based chatbot systems:

• rule-based systems (e.g., ELIZA [19]);

• retrieval based models such as Jann (Just Approxi-mate Nearest Neighbour) 1, combining the UniversalSentence Encoder [5] embeddings of Cornell Movie-Dialogs Corpus [2] with approximate nearest neigh-bor search;

1https://github.com/korymath/jann

Session: Poster & Demo Reception CC ’19, June 23–26, 2019, San Diego, CA, USA

528

• generative language such as A.L.Ex (Artificial Lan-guage Experiment) [12], based on sequence-to-sequence recurrent neural networks [6, 18] trainedon the OpenSubtitles corpus [17].

Figure 4: Example of improvperformance with two humans andone robot. On the right, anaudience volunteer curates thesuggestions generated by the AIchatbot before sending them to thePA system of the theatre (audio)and to the robot (embodiment).

Figure 5: Example of improvperformance between a humanand a robot (EZ-Robot).

This progression illustrates the limitations and benefits ofthe different models. It invites participants to explore howeach AI-based system might contribute to performance the-atre in different ways. In these interactions, we focus on thedynamics necessary for successful improvisation alongsideAI. We discuss timing, status, physical dynamics, and theundesirable—albeit quite common—dismissal of nonsen-sical lines from conversational AI models. Teachings fromour AI-centered workshop are transferable, for instance fordeveloping performance and public speaking skills [20].

Exploring AI Stage Partner EmbodimentOver the course of the workshop, we explore how the em-bodiment of the conversational dialogue system can affecttheatricality and communicative dynamics. We illustrateand demonstrate the differences by comparing a roboticplatform [9] to a video projection-based system. We alsoexplore how the lines generated from the conversationalmodels can be supplied (via headphones) to the humansin the performance (Fig. 1). In this way, a subset of the hu-man performers can deliver nonsensical AI-generated linesusing uniquely human emotion, instinct, and timing, whilethe other human performers justify and ground the scene.By modifying the embodiment we are able to compensatefor timing limitations of computational systems through non-verbal acting and emotional subtext.

Introducing New Modes of Interaction in ImprovWe will demonstrate several novel interactive systems forhuman-machine collaboration in improvisational theatre.AutomaTED invites performers to deliver an improvised

TED style talk based on slides generated by an AI-basedsystem [20]. dAIrector generates a sequence of plot pointsfor the human(s) to improvise with [3]. We also presentexplorations in multi-modal generation, specifically, we in-troduce “emotion driven music generation”, where a sys-tem provides underscoring for improvised scenes based onemotional content of the dialogue.

DiscussionWe aim to augment and enhance human performance bybuilding and deploying challenging AI-based improv sys-tems. The interaction between the humans and the ma-chines has been explicitly explored in previous research.We discuss these findings and open the discussion to con-ference participants to provide their own reflections on in-teracting with the systems. We particularly look forward toconference participants’ engagement in discussions on giv-ing up control to AI-based collaborators [14].

We also wish to explore audience reactions to these tech-nologies, namely, the balance between public excitementsurrounding AI and the fear and misinformation about thecapabilities of such systems. One of the major goals of ourwork is to invite the public into the conversation: we give in-tuitive explanations about AI during our shows and value lis-tening to the audience’s perspectives on AI. Our past workand performances have illustrated the limitations of the cur-rent state-of-the-art models, to great comedic success.

AcknowledgementsWe would like to thank Adam Meggido, Chris Mead, KatySchutte, and Matt Schuurman for direction as well as pastand present cast members of Improbotics from London,Edmonton, Sweden, and Belgium. For full cast listings seehttps://improbotics.org.

Session: Poster & Demo Reception CC ’19, June 23–26, 2019, San Diego, CA, USA

529

References[1] Jean Benedetti. 1999. Stanislavski: his life and art.

Methuen Drama.

[2] Cristian Danescu-Niculescu-Mizil and Lillian Lee.2011. Chameleons in imagined conversations: A newapproach to understanding coordination of linguisticstyle in dialogs.. In Workshop on Cognitive Modelingand Computational Linguistics, ACL.

[3] Markus Eger and Kory W Mathewson. 2018. dAIrector:Automatic Story Beat Generation through KnowledgeSynthesis. arXiv preprint arXiv:1811.03423 (2018).

[4] Brian Magerko et al. 2009. An Empirical Study ofCognition and Theatrical Improvisation.. In Creativityand Cognition.

[5] Daniel Cer et al. 2018a. Universal sentence encoder.arXiv preprint arXiv:1803.11175 (2018).

[6] Ilya Sutskever et al. 2014. Sequence to sequencelearning with neural networks. In Advances in NeuralInformation Processing Systems. 3104–3112.

[7] Lara J Martin et al. 2016. Improvisationalcomputational storytelling in open worlds. InInternational Conference on Interactive DigitalStorytelling. Springer, 73–84.

[8] Nouha Dziri et al. 2018b. Augmenting NeuralResponse Generation with Context-Aware TopicalAttention. arXiv preprint arXiv:1811.01063 (2018).

[9] EZ-Robot Inc. https://www.ez-robot.com Accessed 8March 2019.

[10] Philip N Johnson-Laird. 2002. How jazz musiciansimprovise. Music Perception 19, 3 (2002), 415–442.

[11] Keith Johnstone. 1979. Impro: Improvisation and theTheatre. Faber and Faber Ltd.

[12] Kory W Mathewson and Piotr Mirowski. 2017.Improvised Theatre Alongside Artificial Intelligences. InAAAI AIIDE.

[13] Kory W Mathewson and Piotr Mirowski. 2018.Improbotics: Exploring the Imitation Game usingMachine Intelligence in Improvised Theatre. In AAAIAIIDE.

[14] Sanford Meisner and Dennis Longwell. 2012. SanfordMeisner on acting. Vintage.

[15] Bill Merritt and Stephanie Carrie. 2012. Pirate, Robotor Ninja? UCB Vet Billy Merritt’s Theory on the ThreeTypes of Improv Performers. https://goo.gl/NF56JeAccessed 8 March 2019.

[16] Ted Shawn. 1963. Every little movement. Printed bythe Eagle Print. and Binding Co.

[17] Jörg Tiedemann. 2009. News from OPUS-A collectionof multilingual parallel corpora with tools andinterfaces. In Recent Advances in Natural LanguageProcessing, Vol. 5. 237–248.

[18] Oriol Vinyals and Quoc Le. 2015. A neuralconversational model. arXiv preprint arXiv:1506.05869(2015).

[19] Joseph Weizenbaum. 1966. ELIZA—a computerprogram for the study of natural languagecommunication between man and machine. Commun.ACM 9, 1 (1966), 36–45.

[20] Thomas Winters and Kory W Mathewson. 2019.Automatically Generating Engaging Presentation SlideDecks. In 8th International Conference onComputational Intelligence in Music, Sound, Art andDesign.

Session: Poster & Demo Reception CC ’19, June 23–26, 2019, San Diego, CA, USA

530