collaborative storytelling with human actors and ai

Collaborative Storytelling with Human Actors and AI NarratorsPaper type: Event Report

Boyd BranchUniversity of KentUnited Kingdom

[email protected]

Piotr MirowskiImprobotics

United Kingdomimprobotics.org

Kory MathewsonImprobotics

Canadaimprobotics.org

Abstract

Large language models can be used for collaborativestorytelling. In this work we report on using GPT-3(Brown et al. 2020) to co-narrate stories. The AI systemmust track plot progression and character arcs while thehuman actors perform scenes. This event report detailshow a novel conversational agent was employed as cre-ative partner with a team of professional improvisers toexplore long-form spontaneous story narration in frontof a live public audience. We introduced novel con-straints on our language model to produce longer narra-tive text and tested the model in rehearsals with a teamof professional improvisers. We then field tested themodel with two live performances for public audiencesas part of a live theatre festival in Europe. We surveyedaudience members after each performance as well asperformers to evaluate how well the AI performed in itsrole as narrator. Audiences and performers respondedpositively to AI narration and indicated preference forAI narration over AI characters within a scene. Per-formers also responded positively to AI narration andexpressed enthusiasm for the creative and meaningfulnovel narrative directions introduced to the scenes. Ourfindings support improvisational theatre as a useful test-bed to explore how different language models can col-laborate with humans in a variety of social contexts.

Improv Theatre and AIImprovisational theatre is increasingly being used as an en-vironment and platform for testing and exploring the cre-ative potential of computationally creative systems (Bruceand others 2000; Baumer and Magerko 2010; O’Neill andothers 2011; Jacob 2019) and of artificial intelligence lan-guage models in particular (Martin, Harrison, and Riedl2016; Mathewson and Mirowski 2017; 2018; Cho and May2020). Turing test inspired experiments focus on evaluat-ing how well language models can perform natural-soundinghuman language (Turing 1951). Conversely, improvisa-tional theatre is uniquely positioned to explore the collab-orative potential of language models. Collaborative story-telling includes both on-stage performance (e.g. improvisedtheatre) and off-stage games (e.g. table-top role-playinggames, card games). Collaborative storytelling in collabora-tion with artificial agents has been studied previously (Per-lin and Goldberg 1996; Hayes-Roth and Van Gent 1996;

Riedl and Stern 2006; Magerko and others 2011), most of-ten in the context of virtual environments where the humanplayers interact with digital avatars, with the exception ofplot generation tools like dAIrector (Eger and Mathewson2018) informing live improv on stage. Recent advances inlarge language models enable richer text-based interactionbetween human and AI players (Nichols, Gao, and Gomez2020), as illustrated by the success of online role-playinggame AI Dungeon1. Our case study pulls away from the vir-tual world and situates AI and human collaborators togetheron-stage. This shared narrative is then interpreted by live hu-man actors, expressing the full range of emotional, physicaland verbal human creativity.

Improvised theatre explores how interesting narrativescan emerge from establishing rules for simple social dy-namics and rhetorical conventions. In contrast to scriptedtheatre, improv is built from spontaneity (Spolin and Sills1963). Improvisers are trained to disengage executive cog-nition in order to allow their automatic responses to guideand justify a given and emerging social context (Johnstone1979). Narratives emerge by assuming the presence ofmeaning. The performer only needs to accept offers: whatis said and done on stage. There are no ‘wrong’ things aperformer can say to invalidate the emerging narrative. Formeaning to emerge for an audience however, each novel nar-rative statement must be followed up with some degree ofagreement and justification. An improvisational scene is ul-timately judged on the degree to which novel statements canbe integrated back into the previous given circumstances.That process is called justification and is synonymous withan ongoing adaptation, by the actors–thrown out of theircomfort zone–to the changing dynamics of an improvisednarration. This practice is what makes improv theatre sucha useful platform to explore the creative capacity of artifi-cial intelligence. For the AI to perform ‘well’ it cannot sim-ply introduce novel narrative subjects, but must also be ableto adapt to the emerging given circumstances, akin to thedesiderata of AI systems capable of generalising to unseendata.

Previously, (Mathewson and Mirowski 2017; 2018; Choand May 2020) explored how an AI trained on movie or im-prov dialogue could generate interesting narratives as a per-

1https://play.aidungeon.io/

Proceedings of the 12th InternationalConference on Computational Creativity (ICCC ’21)ISBN: 978-989-54160-3-5

97

Figure 1: Left: Operator of the AI narrator and virtual avatar.Right: Example of an improvised scene. Photos: Erika Hughes

former within an improvised scene, and demonstrated thatconversational agents built using recurrent neural networksor transformers, e.g., GPT-2 (Radford and others 2019),could indeed move a given narrative forward when humanagents were operating to accept and justify the statements.In their setup, the AI only functioned as a character in agiven scene. In our study, we examine how AI performs inthe role of the narrator.

MethodsDatasets for AI improvBefore large language models, conversational agents weretrained on datasets geared towards dialogue, like the Cor-nell Movie Dialogs Corpus (Danescu-Niculescu-Mizil andLee 2011) and OpenSubtitles (Tiedemann 2009). The latterwas used to develop conversational models such as (Vinyalsand Le 2015) and the improv theatre-specific chatbot A.L.Ex(Mathewson and Mirowski 2017) from HumanMachine2.Later on, an improv-specific dataset of yes-and exchangesfrom improv podcasts was curated in (Cho and May 2020).

While employed in the context of improvised storytelling,our work departs from generating dialogue and focuses onstorytelling from the perspective of a narrator. We hypothe-sized that the best datasets would come from general fictionnovels as well as synopses and plot summaries. Coinciden-tally, Large Language Models (LLMs) are now pre-trainedon comprehensive and diverse sets of corpora and are ca-pable of memorising diverse linguistic patterns from books,novels, movie scripts, newspapers or blog posts (Radfordand others 2019; Brown et al. 2020). From the perspectiveof thematic diversity and specificity, the need for collectingspecific training data seems to have become less of an issue.

Live curation, mitigating of model biasThere are however trade-offs between the predictive powerof large language models, and their embedded biases or theirmisalignment with desired societal values, which have beendiscussed in (Bender et al. 2021; Kenton et al. 2021).

Our approach to mitigating these biases and to the re-moval of offensive content relies on a combination of au-tomated filters and human curation, performed in real timein the context of a live show. First, we remove sentencesthat contain known offensive words from a blocklist, andall generated sentences are validated using multiple filters

2https://humanmachine.live

for inflammatory, hateful or sexual content by the Perspec-tive API3. Second, the human who operates the storytellerinterface has agency in both how they formulate and typethe context, and in what sentences produced by the AI theychoose to read, with a possibility to omit or reword parts ofthose sentences.

Interactive live AI narration on stageOur interface works in the following way: each time a sen-tence is typed by the operator, it is concatenated to thecontext of the scene. GPT-3 is then run 3 times on thewhole context, thus generating three sets of sentences of to-tal length up to 100 characters for each set. The operator hasthe choice of selecting none, one, or several of these sen-tences, in the order they choose4.

An important aspect of the human-machine interaction onstage is that the actors’ performance and the operation of theAI happen simultaneously, i.e. that the operator types con-text prompts and chooses AI-generated suggestions at thesame time as scenes unfold. The human operator may theninterrupt the scene, in a similar way to an improviser ‘edit-ing‘ the scene. This delegates to the human cast and to theoperator the artistic choices of timing–a crucial element ofcomedy–and maintains the liveness of the performance.

Story initiation We initially experimented with a systemfor automated selection of initial writing prompts from thenovel-first-lines-dataset (a crowdsourced dataset of first sen-tences of novels)5. The single-word audience suggestionwould be matched with a fixed set of 11k sentences us-ing sentence-level embeddings computed using the Univer-sal Sentence Encoder (Cer et al. 2018) combined with ap-proximate nearest neighbor search6. Early trials during im-prov rehearsals demonstrated that the first lines of novelswere not informative enough for the actors performing muchshorter scenes, and that the actors preferred to initiate thestory themselves.

Avatar for the AI narrator We designed a virtual avatarthat personified the AI narrator. That avatar consisted of a3D model of a robot, inspired by Aldebaran Robotics’ Nao,built using Cinema 4D7 and imported into Adobe Charac-ter Animator8 as a puppet controlled by facial expressionsof the operator as they are reading the AI-generated lines.Instead of using computer-generated voice, we relied on hu-man voice for expressive interpretation. The operator was

3https://www.perspectiveapi.com/4While it has been observed that GPT-3 is capable of some

amount amount of meta-learning, such as recognising and generat-ing analogies, or responding to commands (e.g., ”translate the fol-lowing sentence from English to French”) (Brown et al. 2020), wedecided to limit this work to using GPT-3 as a statistical languagemodel and to leave, for future work, hierarchical text generation oradditional prompt engineering, such as “expand on what happensnext” or “let’s look back at this character”.

5https://github.com/janelleshane/novel-first-lines-dataset6https://github.com/korymath/jann7https://www.maxon.net/8https://www.adobe.com/products/character-animator.html


98

standing behind the TV screen that was projecting the avataron the stage.

Evaluating AI in performanceWe worked with a small team of professional improvisers tobuild and rehearse an original 50 minute performance thatincluded a series of short AI-assisted scenes followed bya 12-minute AI narrated long-form improvisational scene.We then presented 2 performances for public audiences.We present a partial transcript from one of the AI narratedperformances and discuss how well the AI was able to of-fer contextually relevant suggestions that advanced the plot.We also administered anonymous surveys to the performers(p1-p5) as well as 9 audience members after each perfor-mance (a1-a9). The surveys consisted of a series of openended questions regarding how they experienced the AI onstage. Our surveys were conducted in accordance with theapproved ethical standards of our public research institution.

ResultsWe relate a 12-minute long-form improvisation between 6actors. GPT-3 generated altogether 455 sentences of sug-gestions, but only a subset was selected by the operator. Thefollowing extract is an exact transcript of only the inputs orcontext given to the GPT-3 language model. Sentences inbold correspond to GPT-3 outputs, which are fed backagain as context. (Notes in italic and between parenthesesare scene descriptions that were not input to GPT-3). Thetranscript of performer lines and interactions is not includedbecause of the difficulty of transcribing an improvised per-formance where many actors may speak on top of one an-other. The audience gave the suggestion: “Pizza Hut”.

At the Pizza Hut. Brian and his date lost patience. (Theoperator misunderstood the relationship between thetwo protagonists.)There was always a reason for them to admire eachother. Brian was an expert at making pizza. Sally foundher vocation, making pizza like Brian. Brian startedlisting all the products... Baguettes, patisserie... Sallyasked Brian for help. (The operator made a confusionin the name, as it was Sandra, not Sally.)The door opened and a burly man entered, followedby his wife. (A couple entered the pizzeria, the manspoke with a heavy voice.)The husband and the wife entered the pizzeria. Theyasked for supremes, with garlic bread. Both womenhad crushes on Brian. (The unnamed wife briefly ap-proached Brian.)Sally searched for pastries. The husband and the wifeasked for vodka. (Unused suggestion.)They got creme patissiere... Brian apologized.(Sally/Sandra was rolling pizza on the floor.)Sally was dreaming about becoming a master patissier.She continued to look for pastries. (Sally/Sandra saidshe was done working at Pizza Hut and wanted to re-sign. Scene transition, with an angry boss entering thestage.)

Brian’s boss told him he would let her go. Sally gaveher notice. The boss refused. The boss was cruel.Brian asked the boss for her resignation. The bossmade a mistake. (A confrontation took place betweenBrian and the boss, the boss later started behavingapologetically.) Brian and Sally left the pizzeria.(A male actor stepped in to play the newly introducedSally.) (Scene transition to Sandra at a restaurantowned by the burly man and his wife.)Sandra pursued her dream of being a pastry chef. San-dra was serving the old burly couple. The burly manwas impressed. The burly man and his wife compli-mented Sandra.Even though Sandra was violating safety regulations.Sandra was getting tired. Sandra’s dream wouldsoon come true.They loved it! With her sweat, she impressed them.Sandra was now a great pastry chef. (Scene transi-tion to the boss joining the group.)The boss came to apologise to Sandra. Sandra saidthat she remembered him. He was diminished. Hewas wondering if it was safe to do it on the floor...She heard about Brian. Can you come back, he asked.The boss was apologetic. Sandra thanked the boss,who helped her. Brian and Sandra were both happy.Sandra was proud. The boss was really clear. Theboss was jealous. (Group scene.) He agreed. (Endscene.)

As the 12-minute scene unfolded, the operator was typinga summary of it as inputs to an interface to GPT-3. For eachline of context that was input by the operator, there weremany alternative suggestions that could have been selected,and this transcript shows only the ones that were actuallychosen and presented to the cast and to the audience. The de-cision to intervene in the narration and the timing and deliv-ery of each intervention were choices made by the operator,who was simultaneously voicing and animating the virtualavatar, as well as observing the live improvised scene.

Just like in the first show (for which we do not report thetranscript in this paper), the AI-assisted narrator’s interac-tions became more frequent as the scene was unfolding andthe characters established. The motivation for this was tolet the actors establish the characters and their relationshipsfirst, and to start intervening only once the cast had an initialguess of the narrative arc of the story.

Audience and performer responseWe provide the following small sample of 9 audience re-sponses as useful observations to guide discussion ratherthan evidence of findings that can be generalised. 7 of therespondents indicated the presence of AI itself as the mostsignificant motivational factor in attending the events. 6 re-ported overall satisfaction with AI narration, 1 reported neu-tral satisfaction, and two reported dissatisfaction. The AInarrated scene was the most frequently cited (5/9) responseto the question ‘What did you enjoy most about the show.’

All 5 performers reported satisfaction with the ability ofthe AI to move the story forward. 3 however also ‘slightly


99

agreed’ that the AI ‘mainly introduced absurd or random in-formation’ into the scenes. We present the following quotesfrom performers about their experience to advance discus-sion about the relationship between the insertion of surpris-ing plot points experienced as both ‘random’ and useful inadvancing the story arc.

• As a performer I had to physically become the character(the AI) described in the narration. This pushed me to acertain pov / voice/ physicality which I probably wouldn’thave chosen i.e. a gruff, muscly patisserie store owner.(p1)

• (The AI) added a level of randomness and craziness dif-ferent from a human brain. (p2)

• (The AI) really helped the plot move forward, but with-out being too prescriptive, and enabled me to focus oncharacter development, relationships, emotions and ob-ject work. (p3)

• I did a few scenes as the protagonist where I was sad, andthen (the AI) would say ’she was happy’ or similar, but Iloved that as I has to justify it and it was funny! (p4)

• Generally the narrative direction (of the AI) helped theshow move forward in a good direction (p5)

DiscussionIn the above exchange between the operator’s inputs and theAI suggestions, one can notice that the AI introduced twokey characters (the burly man and his wife) who played therole of mentors for the main protagonist, Sandra, and en-abled the resolution of the story by complimenting Sandra’swork. The AI’s suggestions also satisfied a classical narra-tive arc by allowing her “dream to come true” and achievingher transformation into a “great pastry chef”. This illustratesthe capacity for an AI-based narrator–operating in tandemwith a human curator who makes timing decisions–to gen-erate novel and meaningful plot points.

Interestingly, as (p4) noted, the AI-provided suggestionsdid not consistently keep the affect or motivational stance ofsome characters (e.g., the boss was first cruel, then apolo-getic and even helping Sandra). Where this inconsistencymight invalidate a progressing story when uttered from acharacter (and subsequently fail a Turing-test), in the mouthof a narrator it can encourage performers to maintain clas-sical story arcs that require characters to change and adaptover time (Aristotle 350 BC). The inconsistencies of the AI-generated text were interpreted by the cast as narrated rever-sals of feelings, and challenged the performers (as p1 sug-gests) to allow themselves to change and be affected by eachother in surprising but meaningful ways. In improv theatrethis is described as a ‘status’ reversal where the ‘low’ statusof a character at the beginning of a scene becomes ‘high’status by the end (Giebel 2019). Such reversals are in prac-tice often difficult for human improvisers to execute, as oneinstinctively attempts to maintain their given status or fightto maintain ‘high’ status. In this instance, the AI drove theplot more aggressively forward and motivated the perform-ers to shift and adapt status to the evolving circumstance that

in effect provided a more clear beginning, middle, and endto the story.

As a creative partner, rather than simply providing strangeor absurd plot points to challenge the human performers tomake sense out of, the AI seems to have removed someof the cognitive load for improvisers (as with p3) allowingthem to concentrate on relationships. Without a narrator,improvisers must both react spontaneously in the moment,and remember to engage narrative techniques such as statuschanges to move the story forward. This important practiceof narrative making can be understood as “a carefully ar-gued process of removing and adding participants” (Kumaret al. 2008). The practices of theatrical improvisation andacting techniques such as Meisner actor training (Moseley2012) explicitly ask performers “not to be in their heads”,meaning not to withdraw from the live performance in orderto plot or to reflect and comment on the scene, but rather todedicate their entire attention to what is happening in frontof them on the stage. We believe that one of the potentialapplications of computational creative systems could be toalleviate the cognitive load of performers to shift their focusfrom plotting to reacting.

Strikingly, the seemingly random characters introducedby the AI were often the result of human error. But evenwhen such errors were introduced the performers playfullyaccepted the offers that resulted in comic relief, skilfullytransforming an error into a serendipitous opportunity to‘break the fourth wall’ and to connect with the audience.For instance the wrong naming of the main protagonist (asthe operator mistakenly and repeatedly typed “Sally” insteadof “Sandra”) led the improvises to quickly introduce, thenshelve, a temporary character. This tight collaboration be-tween improvisers and AI prevented the introduction of anew character that may have otherwise been considered a‘less carefully argued’ addition to the narrative, to still per-form a useful function (comic relief) without disrupting theevolving story.

ConclusionsNarrative theatrical performances encapsulate human cul-ture, social interaction, physical expression and natural hu-man emotion. Improv is an ideal test-bed to explore ques-tions about the human-AI collaborative creative capacity. Ithas been proposed as a grand challenge for artificial intelli-gence (Martin, Harrison, and Riedl 2016). We believe thatAI-as-collaborator, as in this current study, uplifts artists, asopposed to challenging them.

Language models capture statistics of written corpora ofhuman culture, and thus provide human audiences with amirror of typical narrative tropes and biases. Thus, theyhighlight the need for human interpretation and curation ofAI-generated content. Our two-pronged approach of auto-mated filters followed by human operator selection of sen-tences, illustrates a transfer of responsibility from the lan-guage model to the (human) narrator–not unlike a typicalimprov show, where the human cast are responsible for thestory they tell (e.g., “punching up, not down”) and adapt totheir audiences (e.g., family-friendly vs. late-night shows).


100

This work is, to the best of our knowledge, the first stag-ing of an AI narrator co-creating improvised theatre along-side humans for a live audience. Timing and aesthetics aresignificant factors for the human experience of AI by audi-ences and cast members. The ease of use of the narrativeinterface for the human operator impacts how quickly theycan add to the language model context or choose from itsoutputs. Finally, the imagined ‘personality’ of the AI narra-tor play a role in co-creation. These are important avenuesfor future research on human-AI co-creation.

AcknowledgmentsThe authors wish to acknowledge the anonymous audienceparticipants who filled the questionnaires, and above all, tothank the actors from improv theatre troupe Improbotics9

who collectively created the artwork that enabled this spe-cific study, namely Harry Turnbull, Julie Flower, MarouenMraihi, Paul Little and Sarah Davies.

ReferencesAristotle. 350 B.C. Poetics. Translated by S. H. Butcher.Baumer, A., and Magerko, B. 2010. An analysis of narra-tive moves in improvisational theatre. In JICIDS, 165–175.Springer.Bender, E. M.; Gebru, T.; McMillan-Major, A.; andShmitchell, S. 2021. On the dangers of stochastic par-rots: Can language models be too big? In Proceedings ofthe 2021 ACM Conference on Fairness, Accountability, andTransparency, 610–623.Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.;Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell,A.; et al. 2020. Language models are few-shot learners.arXiv preprint arXiv:2005.14165.Bruce, A., et al. 2000. Robot improv: Using drama to createbelievable agents. In IEEE ICRA.Cer, D.; Yang, Y.; Kong, S.-y.; Hua, N.; Limtiaco, N.; John,R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar,C.; et al. 2018. Universal sentence encoder. arXiv preprintarXiv:1803.11175.Cho, H., and May, J. 2020. Grounding conversations withimprovised dialogues. arXiv preprint arXiv:2004.09544.Danescu-Niculescu-Mizil, C., and Lee, L. 2011.Chameleons in imagined conversations: A new approachto understanding coordination of linguistic style in dialogs.arXiv preprint arXiv:1106.3077.Eger, M., and Mathewson, K. W. 2018. dairector: Automaticstory beat generation through knowledge synthesis. arXivpreprint arXiv:1811.03423.Giebel, J. D. 2019. Improvising tactical choices based onstatus or “who’s driving the dramatic action bus?”. In Ob-jectives, Obstacles, and Tactics in Practice. Routledge. 141–150.Hayes-Roth, B., and Van Gent, R. 1996. Improvisationalpuppets, actors, and avatars. In Proc Comp Game Dev Conf.

9https://improbotics.org

Jacob, M. 2019. Improvisational Artificial Intelligence forEmbodied Co-creativity. Ph.D. Dissertation, GIT.Johnstone, K. 1979. Impro: Improvisation and the theatre.London: Faber and Faber Ltd.Kenton, Z.; Everitt, T.; Weidinger, L.; Gabriel, I.; Mikulik,V.; and Irving, G. 2021. Alignment of language agents.arXiv preprint arXiv:2103.14659.Kumar, D.; Ramakrishnan, N.; Helm, R. F.; and Potts, M.2008. Algorithms for storytelling. IEEE Transactions onKnowledge and Data Engineering 20(6):736–751.Magerko, B., et al. 2011. Employing fuzzy concept fordigital improvisational theatre. In AIIDE, 53–60.Martin, L. J.; Harrison, B.; and Riedl, M. O. 2016. Im-provisational computational storytelling in open worlds. InInternational Conference on Interactive Digital Storytelling,73–84. Springer.Mathewson, K. W., and Mirowski, P. 2017. Improvisedtheatre alongside artificial intelligences. In AIIDE.Mathewson, K. W., and Mirowski, P. 2018. Improbotics:Exploring the imitation game using machine intelligence inimprovised theatre. In AAAI AIIDE.Moseley, N. 2012. Meisner in practice. London, Nick HernBooks.Nichols, E.; Gao, L.; and Gomez, R. 2020. Collaborativestorytelling with large-scale neural language models. In Mo-tion, Interaction and Games. 1–10.O’Neill, B., et al. 2011. A knowledge-based framework forthe collaborative improvisation of scene introductions. InICIDS, 85–96. Springer.Perlin, K., and Goldberg, A. 1996. Improv: A system forscripting interactive actors in virtual worlds. In Conf onComp. G. & Int. Tech. ACM.Radford, A., et al. 2019. Language models are unsupervisedmultitask learners. OpenAI Blog 1(8).Riedl, M. O., and Stern, A. 2006. Believable agents andintelligent story adaptation for interactive storytelling. InIntl Conf on Tech for Int Dig St and Ent, 1–12. Springer.Spolin, V., and Sills, P. 1963. Improvisation for the theater:A handbook of teaching and directing techniques. North-western University Press.Tiedemann, J. 2009. News from opus-a collection of multi-lingual parallel corpora with tools and interfaces. In RecentAdvances in NLP, volume 5.Turing, A. M. 1951. Can digital computers think? The Tur-ing Test: Verbal Behavior as the Hallmark of Intelligence.Vinyals, O., and Le, Q. 2015. A neural conversationalmodel. arXiv preprint arXiv:1506.05869.


101

collaborative storytelling with human actors and ai

Documents