Enhancing Event-Based Semantics in the

Ontology of Picture Books 2

Karen Ang De La Salle University

2401 Taft Avenue, 1004 Manila, Philippines

karenang0903@yahoo.com

Ethel Ong De La Salle University

2401 Taft Avenue, 1004 Manila, Philippines

ethel.ong@delasalle.ph

ABSTRACT

Picture Books 2 generates fable-like stories for children age 6 to

8 years old based on an input picture comprising of three scenes.

Currently the system is able to generate stories that show

transitions and movement of characters and objects across the

adjacent scenes of the input picture. However some of the stories

produced are illogical. This study explores an approach to

enhancing the ontology structure and the reasoning engine of the

system to produce stories that follow a logical flow.

Keywords

Picture Books 2, Story Generation, Ontology, Reasoning Engine,

World State

1. INTRODUCTION

1.1 Storytelling for Children Many people use storytelling for various reasons, such as sharing

their beliefs, culture, and experiences. Also, storytelling has been

widely recognized to develop a child’s language and literary

skills. It plays a vital role in shaping a child's creativity skills.

Children get a chance to practice their imaginations and also

relate the stories to their daily lives.

During the early years of children, they use drawings to share

their stories. Storybooks which contain pictures or picture books

are usually the first contact of a child with the world. The

combination of art and words is a powerful experience for them.

This triggers their imagination and introduces concepts for their

cognitive and language development [6]. This idea led to the

development of a story generation system, Picture Books [7].

1.2 Motivation Picture Books 1 [7] takes a single picture as input and generates

a story based on the elements (background, characters and

objects) in the picture story. However, a single picture may not

capture the intended story that has been conceptualized by the

child when he/she drew the picture. This motivated the

development of Picture Books 2 [1], which takes an input picture

containing a sequence of multiple scenes. The system generates

fable-like stories that also feature common animals possessing

traits as the main characters of the generated story. These

character traits comprise the main factor in determining the

theme for the story. Furthermore, the system relies on the causal

chain of events that may occur in the given scenes to generate a

story. The chain of events is defined in the storytelling ontology

which was manually populated with activities and concepts that

are familiar to the target age group.

Validation made by linguists and storytellers on the resulting

stories have led to the conclusion that the design of the ontology

is insufficient to support the reasoning engine of the story

planner. It lacks certain attributes which can contribute to the

logical correlation of events to produce a coherent story.

Although the ontology design is inadequate, it can still be used to

causally connect events. Hence, in this study, the proposed

design of the ontology will still utilize the previous structure

adapted from ConceptNet [8] while appending other attributes,

specifically conditions and parameters needed for an action or

event to happen in the story world.

1.3 World State Picture Books 2 reasons through the ontology by retrieving the

causal chain of events between the initial state (e.g., the main

character arrives at the camp) and the target goal of the story

(e.g., the character learns to be brave). Due to solely relying on a

causal chain of events, the scoring function used to select an

event from a set of candidate events is random. This simplistic

approach in story planning combined with a set of binary

relations defining events in the ontology did not consistently

produce a chain of events that will lead to a coherent story to be

generated. Oftentimes, it is necessary to consider the world state

to constrain the possible events to be included in the story flow.

Tracking the world state as actions and events occur in the story

is currently not supported in Picture Books 2. A world state

contains knowledge about the state of the different characters and

objects in the story and are updated every time a character

executes an action that may change not only his/her state, but

that of the other characters and objects present in the story world.

In [9], this represents the first level of story representation - the

semantic networks for story world knowledge. The first level

relies on the general ontology of semantic network describing the

knowledge about the concepts and possible relations that exist

between them. World states or story world description [9] is an

instantiation of some concepts in the general ontology which is

concerned with the current properties and relationships of a

concept in a certain point of time.

Although the ontology of Picture Books 2 includes properties and

relationships between concepts, no world state has been utilized

and updated as the story progresses. Instead it assumed static

states, states that do not change over time. Because of this, the

scoring function purely relied on a random selection of

81 Proceedings of the 8th National Natural Language Processing Research Symposium, pages 81-84

De La Salle University, Manila, 24-25 November 2011

subsequent events that may not be applicable to the character's or

object's current state, leading to illogical stories.

A logical story is defined to be a story that exhibits consistency

and coherency. Both terms are adapted from Uijling’s [13]

definition of consistency and coherency in a story. In a consistent

story, the character’s actions are believable because the story

planner process of identifying the candidate actions that a

character may perform at a given point in the story is guided by

its knowledge of the current state of the story world. A coherent

story means that the plot has a logical flow of events.

Consider the story below generated by Picture Books 2 [3]. In

line #3, the object marshmallow has been introduced. However,

as the story progresses, the marshmallow was never mentioned

again; it played no part towards the development of the theme

where Danny the dog has to learn to become brave. Instead,

another object, the flashlight, introduced in line #8, was the key

object that led Danny to search for the cause of his fear and to

attain a change in his trait – that of becoming braver.

#1> It was a fine evening.

#2> Danny the dog was in the camp for a trip.

#3> He buys a packed marshmallow.

#4> The camp is very big.

…

#5> He sees a shadow.

#6> Danny the dog feels scared.

#7> He does not know what to do.

#8> He turned on a flashlight.

#9> Danny the dog searched the shadow.

…

#10> Since then, Danny the dog learned to be brave.

The main question that this research seeks to address is to

determine the types of knowledge to be added to the current

ontology that can be used by a reasoning engine to assist the

story planner in coming up with stories that are logical. The rest

of this paper is organized as follows. Section 2 presents the

proposed ontology structure, where the various types of

knowledge needed by a story planner are discussed. Section 3

presents the initial design of the system, as well as the

interaction among the character agent, plot agent and world

agent. The idea for these three agents was taken from the Virtual

Storyteller [12].

2. ONTOLOGY STRUCTURE The main source of knowledge for the generation of the fabula or

story plan in Picture Books 2 comes from three main categories.

These are the narratological knowledge, domain knowledge, and

character frame.

2.1 Narratological Knowledge The narratological knowledge represents concepts on narrative

structures as defined in [4]. It drives the flow of the story.

Themes and author goals are stored in this knowledge. Author

goals represent the premise for the story and include the conflict,

counter action for the conflict, and resolution for each theme.

2.2 Domain Knowledge The semantic description about concepts, objects, and their

relation are contained under the domain knowledge [4]. The

domain knowledge is the static storytelling knowledge

comprising mainly of the concepts and their relations that are

familiar to children, as well as fabula elements, links,

parameters, conditions, and primitives.

Concepts are connected via relations. Adapted from ConceptNet

[8], these relations include isA, receivesAction, usedFor,

hasProperty, and locatedAt relations.

Moreover, these relations aid the story planner generate the

fabula, which is defined as a chronological sequence of narrated

events for the story. The structure of the fabula is adapted from

[11] on the fabula model for story generation, which is comprised

of the fabula elements and links respectively.

The fabula elements define the story elements which comprise

the happenings in the story. These utilize the concepts,

parameters, primitives, and conditions, to define a single fabula

element. Table 1 shows a sample fabula element.

Table 1. Sample Fabula Element

Fabula Element Action

Concept go

Parameter agent, direction

Primitive PTRANS

[precond: agent ISA character &

direction ISA place &

!(agent LOCATEDAT direction)]

[postcond:

agent LOCATEDAT direction]

Pre-condition null

Post-condition null

On the other hand, the links causally connect these elements

together. These elements together with the links form an implicit

hierarchical structure, which form a story. Figure 1 shows a

sample connection of two fabula elements.

Figure 1. Sample Link

The parameters define the roles required for an action to be

executed, such as the agent and the receiver of the action. In

contrast, conditions define the pre-requisites for an event or

action to take place and its consequential results.

Lastly, the primitives consist of the most basic actions and their

corresponding conditions. These primitives are based from the

theory of Roger Schank on conceptual dependency [10]. Table 2

shows a sample set of primitives.

Table 2. Primitives as defined in Schank [10]

Primitive Description

Atrans Transfer of possession

Ptrans Transfer of physical location

Grasp Agent picks up an object/drops an object

Ingest An agent eats an object

82

2.3 Character Frame The character frame contains description of characters in the

domain. It is where all the information about a character is

stored. This includes the sub-characters, personality, desires, and

roles.

Sub-characters contain all possible sub-characters that may

appear in a story. Table 3 illustrates the sample information

stored for sub-characters. Table 3. Sample Sub-character

Name Gender Type

Peggy female pig

Personality, on the other hand, contains the positive and negative

traits of each character. Also as the name implies, desires define

the desires of characters. For example, children like sweets.

Lastly, roles specify the relationship of a character to another. It

also defines the initial emotion of a character to another. Thus,

this emotion is represented dynamically; as the story progresses,

the emotion felt by a character towards another may change. For

instance, Peggy is the sister of Peter and it is defined that Peter

loves Peggy and vice versa. Over the course of the story, Peter

may do something that may cause Peggy to hate Peter. As such,

Peggy’s emotion toward Peter will change from love to hate.

3. ARCHITECTURAL DESIGN From the multi-scene input picture specified through the Story

Editor of Picture Books 2, a corresponding abstract scene

representation [2] is fed to the story planner to generate the

fabula. The fabula is then forwarded to the Sentence Planner of

Picture Books 2 for lexicalization and aggregation of character

goals, which are then transformed into English sentences during

the realization phase to produce the actual story text.

The Story Planner, which is the main focus of this paper, is

responsible for generating the contents of the story (the fabula or

story plan). Figure 2 illustrates the interaction among the three

agents in the Story Planner.

Figure 2. Architectural Design for the Story Planner

3.1 Plot Agent The plot agent serves as the director of the story planner. It

supervises the selection of the story events. The plot agent also

selects the theme for the story and ensures the coherency of the

generated fabula, which must lead to the selected theme of the

story. In order to accomplish these purposes, it has both the

premise generator and the fabula generator.

The concept of a premise is adapted from Egri [5], which defines

it as a guiding force to ensure that the story will not lead to a

meaningless direction. Hence, Egri proposed a premise to be

comprised of three essential parts, which are the character,

conflict, and conclusion. The character is not only described as

the character itself but also the environment in which the

character belongs. The conflict, on the other hand, describes the

primary conflict which holds the action and counter-action that it

evokes; while the conclusion is the final state of the world. These

parts are represented as the setting or the initial state of the

world, the action and reaction pair that describes the primary

conflict, and the final state of the world, respectively.

The theme selector selects the story’s theme, which defines the

conflict and conclusion of the story. The selection is based on

the negative traits of the main character and the number of

matched conditions between the conflict and resolution to the

input scenes. The highest scoring theme will then be sent to the

setting formulator for it to formulate a setting according to the

selected theme. When the setting for the story is successfully

generated, the fabula or story plan for the selected theme and

setting will then be generated.

Since stories revolve around character actions, the fabula

generator would need the help of the character agent. The

character agent approach provides a means for each character in

the story world to decide on what it can do to accomplish its

goals. It generates a set of plan steps to achieve its current goal.

In order to verify the consistency of these plan steps with the

selected theme, the fabula generator will deliberate on these plan

steps.

The deliberation process considers which plan of action is to be

executed. The plot agent can also manipulate the generated

fabula by inserting events which would affect the character thus

influencing its next plan of action. These events are defined as

the unintentional change in the world state by a character.

Generating events that allow the system to force the story to go

with the given theme is handled by the events generator.

Together, they make up the fabula generator which works with

the character agent to produce a story that corresponds to the

selected theme and given input scenes by the user.

If the plot agent cannot produce a fabula for the selected theme,

it reiterates the process of formulating another setting using the

next candidate theme. The next candidate theme is the next

highest scoring theme. This is determined by comparing the

requirements of the theme with the given input scenes. The more

similarities the theme possesses with the given scenes, the more

likely that that theme is possible to happen.

3.2 Character Agent As mentioned earlier, the character agent decides which actions a

character can take to accomplish its goals. It represents the goals

and plans of a character in the story. Characters have goals that

serve as the basis for their actions. These goals are formed by

their existing goals, personality, needs, and desires.

Thus, the character agent renders the set of candidate goals of a

character given its existing goals, personality, desires, and

current world state. These attributes are accessible to the

character agent through the plot agent, which retrieves these

83

from the world agent. These also provide the character a goal to

pursue.

To achieve its goal, a character has to perform a set of actions.

These actions are realized by generating the next action steps

based on its current goal or previous event that happened. It

filters the possible actions to select the candidate set of actions

that are relevant to the accomplishment of its goal. It also verifies

whether the given input scenes fit the candidate actions. These

set of actions are not automatically executed but are passed onto

the fabula generator, which will decide if the plan of action is to

be executed.

3.3 World Agent Lastly, a world is described by the characters and objects present

and also their properties. Thus, a world state is defined as the

state of the world in a particular point in time. It contains

information about the characters and objects present in the

world. Also, these world states change by appending, deleting or

modifying properties of either the characters or objects.

The world agent holds this world state in each timestep, which is

defined as a point in time that a particular event occurs. It is also

responsible for updating the world state in each timestep. This is

done after the plot agent decides to execute an action or event.

After an action is performed, the plot agent utilizes this updated

world state to verify if the next action can be executed given the

current world state. This process repeats until the system

generates the whole fabula for the story.

4. ONGOING WORK Though Picture Books 2 is able to generate stories, it is

important to consider the quality of the output which may affect

the child using the system. The redesign of the ontology structure

and reasoning engine of the system aims to enhance the story

planner of the system to be able to select a more logical story

path.

The ontology structure plays an important role in story

generation. It is the source from which the system generates the

contents of the story. The bigger the amount of relations

available aids the system to generate more flexible stories.

Hence, the ontology structure should be designed efficiently so as

to reduce the needed information to be inputted. Furthermore,

the right blend of the character agent and plot agent must be

achieved to balance the believability of a character and also the

occurring events which guide the story to its theme.

Issues that must be addressed throughout the development of the

system include representing when a character perceives an event

or action of another character. What would be the basis of the

system to decide that a character must act on the changes in its

environment? Also, given a list of goals of a character, how will

the character agent decide on which goal to pursue first?

5. ACKNOWLEDGMENTS We would like to acknowledge the Department of Science and

Technology – Engineering Research and Development for

Technology for their support in the achievement of this project.

6. REFERENCES [1] Ang, K., Antonio, J., Sanchez, D., and Yu, S. 2010. Picture

Books 2. Undergraduate Thesis, De La Salle University,

Manila, Philippines.

[2] Ang, K., Antonio, J., Sanchez, D., Yu, S., and Ong, E. 2010.

Generating Stories for a Multi-Scene Input Picture. In

Proceedings of the 7th National Natural Language

Processing Research Symposium, 21-26, November 19-20

2010, De La Salle University, Manila, Philippines.

[3] Ang, K., Yu, S., and Ong, E. 2011. Theme-Based Cause-

Effect Planning for Multiple-Scene Story Generation. In

Proceedings of the International Conference on

Computational Creativity, 48-53, April 27-29 2011, Mexico

City, Mexico.

[4] Cua, J., Ong, E., and Pease, A. 2011. Using SUMO to

Represent Storytelling Knowledge. Philippine Computing

Journal, 5(2):37-43, December 2010, Computing Society of

the Philippines.

[5] Egri, L. 2007. The Art of Dramatic Writing. Wildside Press.

[6] Hillman, J. 2003. Discovering Children’s Literature, Third

Edition. Columbus, OH: Merrill Prentice Hall.

[7] Hong, A., Siy, J., Solis, C., and Tabirao, E. 2008. Picture

Books: An Automated Story Generator. Undergraduate

Thesis, De La Salle University, Manila, Philippines.

[8] Liu, H., and Singh, P. 2004. Conceptnet - A Practical

Commonsense Reasoning Tool-kit. BT Technology Journal,

22 (4), 211–226.

[9] Oinonen, K., Theune, M., Nijholt, A., and Uijlings, J. 2006.

Designing a Story Database for Use in Automatic Story

Generation. In Proceedings of the 5th International

Conference on Entertainment Computing, 298-301,

Cambridge, UK.

[10] Schank, R. C. 1972. Conceptual Dependency: {A} Theory of

Natural Language Understanding. Cognitive Psychology,

(3)4, 532-631.

[11] Swartjes, I., and Theune, M. 2006. A Fabula Model for

Emergent Narrative. In Proceedings of the Third

International Conference on Technologies for Interactive

Digital Storytelling and Entertainment, (Darmstadt,

Germany, December 04 - 06, 2006).

[12] Theune, M., Faas, S., Nijholt, A., and Heylen, D. 2002. The

Virtual Storyteller. IEEE, 23(2), 20–21.

[13] Uijlings, J.R.R. 2006. Designing the Virtual Environment

for Story Generation. Master’s Thesis, University of

Amsterdam, Amsterdam, Netherlands.

84