a new approach to the design of interactive storytelling engines

A New Approach to A New Approach to the Design of the Design of Interactive Interactive Storytelling Storytelling

EnginesEnginesTeongJoo Ong and John LeggettTeongJoo Ong and John Leggett

Texas A&M UniversityTexas A&M University

OverviewOverview

Introduction and BackgroundIntroduction and Background Motivation and Scenarios of UseMotivation and Scenarios of Use Architecture of the Interactive Architecture of the Interactive

Storytelling EngineStorytelling Engine Story GenerationStory Generation Encoding and Evaluating the Story Encoding and Evaluating the Story

ComponentsComponents Story ExampleStory Example Evaluation Criteria, Contributions Evaluation Criteria, Contributions

and Conclusionsand Conclusions

IntroductionIntroduction Stories are used to convey information, Stories are used to convey information,

cultural values, and experiencescultural values, and experiences New technologies have constantly New technologies have constantly

provided increasingly sophisticated means provided increasingly sophisticated means to tell storiesto tell stories

Recent trend is the convergence of Recent trend is the convergence of technology, entertainment, and art in the technology, entertainment, and art in the computercomputer

The ancient art of storytelling can be used The ancient art of storytelling can be used to efficiently convey information in our to efficiently convey information in our increasingly computerized worldincreasingly computerized world

BackgroundBackground

Interactive storytelling is a major research Interactive storytelling is a major research areaarea

Many overlapping approaches have been Many overlapping approaches have been used:used:

Immersive storytellingImmersive storytelling Emergent storytellingEmergent storytelling Plot-based systemsPlot-based systems Interactive authoring of storiesInteractive authoring of stories Character-based systemsCharacter-based systems

These approaches differ in design These approaches differ in design paradigms and technical solutionsparadigms and technical solutions

Background (cont.)Background (cont.) Approaches used in interactive Approaches used in interactive

storytelling fall into the field of Narrative storytelling fall into the field of Narrative IntelligenceIntelligence

Narrative Intelligence commonly refers to Narrative Intelligence commonly refers to research on human narratives and research on human narratives and storytelling, and related technologiesstorytelling, and related technologies

Related research projects from this field:Related research projects from this field: CHOROS (N. M. CHOROS (N. M. Sgouros) Sgouros) Fuzzy Cognitive Maps (FCM) (E. Maffre)Fuzzy Cognitive Maps (FCM) (E. Maffre) HyperSoap (J. Dakss)HyperSoap (J. Dakss) Terminal Time (M. Mataes)Terminal Time (M. Mataes)

OverviewOverview





MotivationMotivation Each of the different approaches used in Each of the different approaches used in

interactive storytelling has its own interactive storytelling has its own strengths and weaknessesstrengths and weaknesses

A hybrid approach that can combine the A hybrid approach that can combine the strengths while reducing or eliminating the strengths while reducing or eliminating the weaknesses of several different approaches weaknesses of several different approaches could potentially be very importantcould potentially be very important

Certain characteristics of the approaches Certain characteristics of the approaches will also be discarded to keep complexity will also be discarded to keep complexity and understandability in checkand understandability in check

Motivation (cont.)Motivation (cont.)

Our hybrid approach proposes the use of an Our hybrid approach proposes the use of an evolutionary algorithm and story templates to evolutionary algorithm and story templates to construct story componentsconstruct story components

Current system states and interaction Current system states and interaction between fuzzy rules impose constraints, between fuzzy rules impose constraints, modifications and evaluations on the story modifications and evaluations on the story components generated by the evolutionary components generated by the evolutionary algorithmalgorithm

The end result is a template-based The end result is a template-based storytelling engine offering flexibility and storytelling engine offering flexibility and robustness in storytellingrobustness in storytelling

Scenarios of UseScenarios of Use

A murder mystery that can yield different A murder mystery that can yield different conclusions depending on the reader’s conclusions depending on the reader’s interaction with computer agents using interaction with computer agents using dynamically generated story componentsdynamically generated story components

Sub-module of a computer game engine Sub-module of a computer game engine that generates story scripts dynamicallythat generates story scripts dynamically

Interactive learning environment that Interactive learning environment that presents concepts in a narrative thread presents concepts in a narrative thread that includes recombination and that includes recombination and indeterminacyindeterminacy

Scenarios of Use in Scenarios of Use in EducationEducation

Multiple experiences (points of view) Multiple experiences (points of view) through changing plots, story lines, and through changing plots, story lines, and character relationships while adhering to character relationships while adhering to a common threada common thread

Student involvement in unfolding the Student involvement in unfolding the story through interactive exploration of story through interactive exploration of alternatives and role-playingalternatives and role-playing

Encourages personalization of the storyEncourages personalization of the story Children as authors of their own storiesChildren as authors of their own stories

OverviewOverview





System ArchitectureSystem Architecture

Evolutionary Programming Subsystem (EPS)

Knowledge Base

Fuzzy Decision Based Subsystem (FDBS)

Graphical/Object Library (GOL)

Integrated Authoring Environment (IAE)

evaluates/modifies parameters

processes/generates story components

monitors/modifies parameters of rules

user interactions

sends combined output

rendering instructions

authors/interactors

Interactive Storytelling EngineInteractive Storytelling Engine

Combines the global search and emergent Combines the global search and emergent characteristics of evolutionary programming characteristics of evolutionary programming and approximate reasoning of fuzzy logic to and approximate reasoning of fuzzy logic to generate new story components generate new story components

The solution can be any valid combination of The solution can be any valid combination of story components in accordance with the story components in accordance with the author’s rules, system states and templatesauthor’s rules, system states and templates

TemplatesTemplates are used to describe story are used to describe story components and the valid contextual sets of components and the valid contextual sets of actions, scripts, objects and agentsactions, scripts, objects and agents

Interactive Storytelling Engine Interactive Storytelling Engine (cont.)(cont.)

Story ComponentsStory Components are the representation of are the representation of a particular time step in the story. They a particular time step in the story. They facilitate the encoding process of the facilitate the encoding process of the Evolutionary Programming Subsystem (EPS)Evolutionary Programming Subsystem (EPS)

Contextual SetsContextual Sets are used to categorize are used to categorize various plot units, events, scripts, actions various plot units, events, scripts, actions and characters within the story componentsand characters within the story components

The The Evolutionary AlgorithmEvolutionary Algorithm explores the explores the problem space using solution instances of a problem space using solution instances of a particular story componentparticular story component

The The Fuzzy Decision Based Subsystem Fuzzy Decision Based Subsystem (FDBS) (FDBS) imposes constraints, modifications and imposes constraints, modifications and evaluations on the solution instances with evaluations on the solution instances with rules and variables in the knowledge base rules and variables in the knowledge base

Interactive Storytelling Engine Interactive Storytelling Engine (cont.)(cont.)

The The Knowledge BaseKnowledge Base stores user stores user interactions, events, fuzzy rules, discrete interactions, events, fuzzy rules, discrete rules and various variables to keep track rules and various variables to keep track of the story’s progressof the story’s progress

In order to ensure proper development In order to ensure proper development and conclusion of the story, underlying and conclusion of the story, underlying story threads must be created by the story threads must be created by the author to act as guidelines for the engineauthor to act as guidelines for the engine

Underlying story threads are coded as a Underlying story threads are coded as a collection of story components so that collection of story components so that solution instances generated by the EPS solution instances generated by the EPS can be evaluated by the FDBScan be evaluated by the FDBS

Integrated Authoring Integrated Authoring Environment (IAE)Environment (IAE)

A collection of tools and interfaces for A collection of tools and interfaces for authors to create interactive storiesauthors to create interactive stories

The collection of editing tools and library The collection of editing tools and library consists of:consists of: Editing ToolsEditing Tools

Agent CharactersAgent Characters Action ScriptsAction Scripts Story ComponentsStory Components The EPSThe EPS The FDBSThe FDBS

LibraryLibrary The Graphical/Object Library (GOL) The Graphical/Object Library (GOL)

OverviewOverview





Story Generation: Step 1Story Generation: Step 1

User interacts with the storyUser interacts with the story Knowledge base records user Knowledge base records user

interactionsinteractions

User


Interacts

Knowledge Base

Modifies/Records user interactions


The knowledge base stores parameters The knowledge base stores parameters from previous execution of the EPSfrom previous execution of the EPS

Evolutionary Programming Subsystem (EPS) Knowledge Base

Stores Parameters from previous execution

Story Generation: Step 3 Story Generation: Step 3

Knowledge Base Fuzzy Decision Based Subsystem (FDBS)

Provides rules/parameters for merging and checking

Knowledge base provides various Knowledge base provides various information and rules for the FDBSinformation and rules for the FDBS

Output of FDBS determines the firing Output of FDBS determines the firing strength of the various fuzzy rules strength of the various fuzzy rules


Knowledge Base

Fuzzy Decision Based Subsystem (FDBS)

Evolutionary Programming Subsystem (EPS)

Provides parameters and information from rules to generate story components from templates

Provides templates and contextual sets used to populate story components

Information from the knowledge base Information from the knowledge base and FDBS are merged by the EPS to and FDBS are merged by the EPS to generate new story components generate new story components


Knowledge BaseEvolutionary Programming

Subsystem (EPS)

Stores story component and other necessary data for historical record

State information about the system is State information about the system is stored in the knowledge base to allow stored in the knowledge base to allow the user to return to a particular scene the user to return to a particular scene of the storyof the story


Knowledge Base

Provides story component used to generate rendering instructions

The resultant story component is sent The resultant story component is sent to GOL to be interpreted into a series of to GOL to be interpreted into a series of instructionsinstructions





Renders various graphical objects and agents

Various objects and agents are displayed Various objects and agents are displayed in the IAE based on the rendering in the IAE based on the rendering instructions generated in the previous stepinstructions generated in the previous step

These steps are repeated until the story These steps are repeated until the story terminatesterminates

OverviewOverview





Encoding Function Encoding Function Parameters in Floating Parameters in Floating

PointPoint

Each of the floating point numbers Each of the floating point numbers represent a gene in a chromosomerepresent a gene in a chromosome

They can encode the unknown variables They can encode the unknown variables in a function (z = v + w + x) or the in a function (z = v + w + x) or the partition sizes of a surface partition sizes of a surface

1.234 2.345 1.11 3.211 0.121 2.3222 0.982Floating point format

Encoding the Encoding the ChromosomesChromosomes

Chromosomes are created by Chromosomes are created by concatenating the floating point concatenating the floating point numbersnumbers

For the EPS, a chromosome For the EPS, a chromosome consists of a series of floating point consists of a series of floating point numbers numbers

Each floating point number Each floating point number represents a particular story represents a particular story sequence in the templatesequence in the template

1.234 2.345 1.11 3.211 0.121 2.3222 0.982Floating point format

Encoding the Story Encoding the Story ComponentsComponents

2 3 2 …

……………………………….……Valid sets of action and object instances

Gene in a chromosome

For example, a template sequence For example, a template sequence numbered 2, with two object instances numbered 2, with two object instances will be encoded as a floating point will be encoded as a floating point number 2.xx depending on the set being number 2.xx depending on the set being chosenchosen

CrossoverCrossover

Chromosome A

Chromosome B

Crossover operator allows chromosomes Crossover operator allows chromosomes to exchange genetic materialsto exchange genetic materials

Single-point and Multi-point crossovers Single-point and Multi-point crossovers are supported by the EPSare supported by the EPS

Chosen offset for crossover

Chromosome A*

Chromosome B*

MutationMutation

Chromosome A

Chromosome A*

Mutation operator randomly changes the gene in a chromosome

Introduces variations in the populationIntroduces variations in the population

Mutation will happen to this gene

After mutation

Fitness EvaluationFitness Evaluation

Chromosome in a population

Story thread given by the author which may indicate preferences or avoidance of certain actions or object instances

The story components are The story components are evaluated against the story evaluated against the story thread given by the authorthread given by the author

OverviewOverview





The Three Little Pigs Story The Three Little Pigs Story ExampleExample<storyline><storyline>

<timestep order="1" name="introduction" sequence="and"><timestep order="1" name="introduction" sequence="and"> <set type="introduction" pickatmost="1"> <set type="introduction" pickatmost="1"> <element name="act_10" addfitness="1"/><element name="act_10" addfitness="1"/> </set></set> </timestep></timestep> <timestep order="2" name="wolf threat" sequence="and"><timestep order="2" name="wolf threat" sequence="and"> <set type="wolf threat" pickatmost="1"/><set type="wolf threat" pickatmost="1"/> </timestep></timestep> <timestep order="3" name="wolf attack" sequence="and"><timestep order="3" name="wolf attack" sequence="and"> <set type="wolf attack" pickatmost="2" <set type="wolf attack" pickatmost="2" placement="remove"/>placement="remove"/> </timestep></timestep> <timestep order="4" name="wolf's plan" sequence="and" <timestep order="4" name="wolf's plan" sequence="and" loop="true" condition="{isPigStillAvailable}==true OR loop="true" condition="{isPigStillAvailable}==true OR {isWolfDead}==true">{isWolfDead}==true"> <set type="wolf's plan" pickatmost="1" <set type="wolf's plan" pickatmost="1" placement="noremove">placement="noremove"> <element name="act_13" addfitness="2"/><element name="act_13" addfitness="2"/> </set></set> </timestep></timestep></storyline></storyline>

The Three Little Pigs Example The Three Little Pigs Example (cont.)(cont.)

Introduction

Wolf’s Threat

Wolf’s Attack

Wolf’s Plan

While the wolf is still alive or a pig is available for the wolf, these template sets provide various alternatives for the wolf to attack the pigs and the pigs’ response to those attacks

Provides background description of the story to the reader. It also determines the template sets that will be used in the rest of the story by selecting a particular variant of the story

The wolf actor shows up. Depending on how the pigs react, certain action instances could dictate that the pigs are paralyzed with fear, and thus eaten by the wolf which can lead to an early conclusion to the story

Conclusion

The wolf may have been killed or cooked by one of the pigs, or all of the pigs were eaten by the wolf, yielding a conclusion

OverviewOverview





Evaluation CriteriaEvaluation Criteria

Quality of story is based on user’s Quality of story is based on user’s evaluation of the events, deviation evaluation of the events, deviation from the author’s story line, and from the author’s story line, and congruence of the transitions in the congruence of the transitions in the storystory

Responsiveness, performance and Responsiveness, performance and execution speed of the engineexecution speed of the engine

Stability and consistency in the Stability and consistency in the knowledge baseknowledge base

ContributionsContributions

Simple and integrated environment Simple and integrated environment for authors to create interactive and for authors to create interactive and recombinant storiesrecombinant stories

Materials used in one story can be Materials used in one story can be reused in a related story settingreused in a related story setting

A useful educational tool when rules A useful educational tool when rules are integrated into the engine to are integrated into the engine to reflect certain physical or social reflect certain physical or social phenomenaphenomena

ConclusionsConclusions

The hybrid approach used in The hybrid approach used in storytelling is not perfect since it lacks storytelling is not perfect since it lacks the ability to automatically generate the ability to automatically generate story templates out of a knowledge basestory templates out of a knowledge base

Gives authors the power to have full Gives authors the power to have full control of the cyborg authoring control of the cyborg authoring environment and provides tools to environment and provides tools to author various aspects of the interactive author various aspects of the interactive storiesstories

a new approach to the design of interactive storytelling engines

Documents

story templates

story lines

different approaches

based systemsthese approaches

scenarios of usearchitecture

different conclusions

weaknessesa hybrid approach

new approach