gathering background knowledge for story understanding through crowdsourcing

Gathering Background Knowledge for Story Understanding through Crowdsourcing

Christos T. Rodosthenous & Loizos Michael Computational Cognition Lab, Open University of Cyprus

Introduction General research problem

Acquisition of Background Knowledge to be used by an automated story understanding system

Approach to the problem Develop a method / system to facilitate knowledge acquisition using crowdsourcing techniques Source task to the crowd

Engine Architecture

Knowledge Representation Gather knowledge in a machine-readable form

High-level version of the Event Calculus (Michael, 2010)

A fluent F: is an object whose value can change through the course of time An action A: event that occurs at a specific time-point A literal L: fluent or an action, or their negation

Knowledge Representation Φ implies L: constraints that hold at each story time-point e.g., person(X) implies can(X,think)

Φ causes L: capture the conditions whose presence at some time-point is sufficient to change the state of L at the next time-point e.g., attack(X,Y) causes war(X,Y)

Crowdsourcing “A strategy that combines the effort of the public to solve a problem or produce a resource” (Wang et al., 2013). Games With A Purpose (GWAPs)

Genre of crowdsourcing ESP Game, Verbocity, Common Consensus

“Knowledge Coder” Game Output-agreement games template

Players required to agree on the same output they produce Game story

Planet Earth is captured by alien forces capable of intercepting human communications in natural language. Join the resistance forces and encode human knowledge in a form that is not readable by aliens.

Scoring and incentives Players are rewarded with points:

For each successful mission attempt When other players contribute and verify the former players’ mission results and vice versa

Awards issued after a certain score

Considerations Assumptions for human participants

Knowledgeable Honest Willing to participate

Game comprises multiple steps

Lower probability of user error Easier control of the outcomes of each step Facilitate the integration with knowledge understanding systems

Considerations Cheating: non-standard methods for creating an advantage beyond normal gameplay Anti-cheating mechanisms

Player anonymity Internet address recording/filtering Time-bounded missions

Acquiring Knowledge Acquire broad knowledge

Use in Multiple Stories Methodology comprises 6 steps casted as game missions:

Mission 1 - Sentence processing Mission 2 - Verb and noun identification Mission 3 - Predicate construction Mission 4 - Rule construction Mission 5 - Rule generalization Mission 6 - Rule evaluation

1. Sentence processing Sentence: A cat chased the mice.

After processing: {cat,chase,mouse}

2.Verb & noun identification Selected phrase: {cat,chase,mouse}

After separation: {cat,mouse} are nouns

{chase} is a verb

3.Predicate construction

Formal expression: chase(cat,mouse)

Selected words: {cat,mouse} nouns,{chase} verb

action

4.Rule construction

Rule 1: chase(cat,mouse) causes fear(mouse,cat)

Rule 2: chase(cat,mouse) implies can(cat,run)

Formal expression: chase(cat,mouse)

5. Rule generalization

Rule 2: cat(X) and chase(X,Y) implies can(X,run) Rule 1: chase(X,Y) implies can(X,run)

Rule: chase(cat,mouse) implies can(cat,run)

6. Rule evaluation

Applicability

Sentence: A policeman was chasing a burglar.

Validity

Rule: chase(X,Y) implies can(X,run)

Rule Applicability The conditions in the body of the rule are met in the context of the selected sentence.

Sentence: A policeman was chasing a burglar.

Rule: chase(X,Y) implies can(X,run)

Body

X Y

Rule Validity Decide whether the head of the rule follows from the sentence.

Rule: chase(X,Y) implies can(X,run)

Head

Sentence: A policeman was chasing a burglar.

X

5 participants Men and women 18+ >High School education

2 Stories (Aesop's fables) The Oxen and the Butchers The Doe and the Lion

Empirical Setting

Access to a test game deployment for 1 week Training on how to play

Empirical Results Number of generated rules: 93

Number of causality rules: 15 Number of implication rules: 78

Examples of generated rules horn(X) and assemble(X) and carry(purpose) and sharpen(X) and assemble(certain,X,carry(purpose)) implies have(ox,horns)

beast(X) and throw(Y,mouth,X) implies kill(X,Y)

Empirical Results Examples of generated rules

beast(X) and man(Y) and doe(Z) and exclaime(Z) and escape(Z,Y) and throw(Z,X) implies kill(X,Z)

Example of a “good” rule beast(X) and throw(Y,mouth,X) implies kill(X,Y)

Typos are common in GWAPs

Player Feedback Missions 1 and 2

Easy to play Informative instructions

Missions 3 and 4 Required time before players understand fully what they were expected to do Interesting

Mission 5 Not very challenging

Mission 6 Easy to play

Player Feedback Interesting game story Would advertise the game to their friends Proposed tablet and mobile version of the game Requested integration with social media

Conclusions & Future Work Encouraging results in terms of the feasibility of our methodology Conduct further experiments with more stories and players Acquisition of not highly applicable rules

Need for stronger incentives to simplify the rules Integrate “Knowledge Coder” with a reasoning engine

Framework based on psychologically-validated models of narrative comprehension (Diakidoy et al., 2014)

Compare crowdsourcing methods used in “Knowledge Coder” game with automated knowledge acquisition methods

More information…. • Christos T. Rodosthenous • Email: christos.rodosthenous@ouc.ac.cy • WWW: http://cognition.ouc.ac.cy

Game is accessible online at: http://cognition.ouc.ac.cy/narrative