Polling the Blogosphere: a Rule-Based Approach to Belief Classification

Jason Kessler

Indiana University, Bloomington

Belief Analysis of Blogs

Polling the blogosphere on a controversial proposition Literal search on a proposition (e.g., “Obama is electable”) Which blog entries contain assert it? Which deny it? Aggregate results

243 bloggers assert it 616 bloggers deny it

Motivating Example

Polling for “the Moon landings were staged” “The theory that the Moon landings were staged is

complete nonsense.” The writer denies “the Moon landings were staged.”

Motivating Example

If Obama is electable, the country is in good shape. Writer takes no stance toward “Obama is

electable”.

Problem

When a writer uses a declarative finite clause, does that writer assert, deny, or take no stance toward its truth value?

This is the problem of identifying a writer’s stance toward a proposition.

Veridicity or facticity of a proposition.

Example

Everybody is sad that the bar closed. The writer asserts “the bar closed.” Belief != Sentiment

Negative sentiment toward “the bar closed” Positive stance.

Outline

System Description Given a proposition, sentence Dependency Parse

Syntactic Representation Hand written patterns over semantic classes

Veridicality Elements Veridicality Transformations

Evaluation Proof of concept Promising results

The theory that the Moon landings were staged is complete nonsense.

Dependency Parse

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

The theory that the Moon landings were staged is complete nonsense.

Veridicality Elements (VEs)

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

The theory that the Moon landings were staged is complete nonsense.

Veridicality Transformations (VTs)

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

The theory that the Moon landings were staged is complete nonsense.

Veridicality Transformations (VTs)

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

System Structure: Veridicality Elements

Find expressions that have the potential of changing the truth-value of a proposition or referring to it

Different classes affect truth values differently Examples:

Assertion – Positive The assertion that the sky is blue

Nonsense – Negative The idea that the sky is orange is nonsense

If – Neutral Pretend – Counter-factive

Finding Veridicality Elements

Manually created seed sets Search web for patterns likely to contain VEs “I agree with the assertion that”

“I * with the assertion that” “I quibble with the assertion that” “I take issue with the assertion that”

Manually classify matches, form new queries “I take issue with the * that”

“I take issue with the argument that”

Similar to Brin (1998)

System Structure:Veridicality Transformations

Relate these expressions to propositions Some expressions won’t be related to propositions Why bag-of-Veridicality-Elements fails

Templates over dependency graphs Select for a VE class and a proposition

System Structure:Veridicality Transformations

Examples Expression is a main verb, proposition is its comp. clause

John pretended the monkey was harmless. Cleft construction, expression is an adjective

It is inconceivable that two plus two equals five.

Another Example

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

If Bob goes to school, he realizes the Earth is round.

Another Example

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

If Bob goes to school, he realizes the Earth is round.

Another Example

Pipeline Stages:

2. Dependency Parse

3. Tag Veridicality Elements

4. Apply Veridicality Transformations

If Bob goes to school, he realizes the Earth is round.

Evaluation

Primitive, proof-of-concept evaluation Can we poll the blogosphere? Google blog search for “abortion is murder”

Unseen data Run the system on the first 100 hits. See if it does better baseline.

Evaluation

Exclude a number of results: Spam blogs Long, unparsable sentences Trivial sentences (no VEs)

Abortion is murder! Questions

Evaluation

Corpus Statistics: 48 Sentences

27 positive 3 negative 18 neutral

39 classified correctly (81% accuracy) Majority class was positive, giving a baseline of 56%

accuracy

Related Work

Nairn et al. (2006) focused on main verbs Complex behavior under negation

Work on contextual polarity for sentiment analysis. Wilson et al. (2005)

Statistical approach Polanyi and Zaenen (2006)

Theoretical approach

Related Work

Somasundaran et al. (2007) Statistical techniques used to detect presence of

“arguing” in a sentence. Arguing = writer takes a non-neutral stance toward

some content

Future Work

Annotate corpus Further testing Statistical approaches

Augment VE/VTs Integrate Nairn et al. (2006) Take into account questions

Takeaways

Belief analysis is a young field Bag-of-words is not enough Shallow linguistic methods show promise

Questions?

Thank you. References: Brin, S. 1998. Extracting patterns and relations from the world wide web. In

WebDB Workshop at 6th International Conference on Extending Database Technology, EDBT’98.

Nairn, R.; Condoravdi, C.; and Karttunen, L. 2006. Computing relative polarity for textual inference. In ICoS-5.

Polanyi, L.; and Zaenen, A. 2005. Contextual valence shifters. In Shanahan, J. G.; Qu, Y.; and Wiebe J., eds,. Computing Attitude and Affect in Text.

Somasundaran, S.; Wilson, T.; Wiebe, J.; and Stoyanov, V. 2007. QA with attitude: Exploiting opinion type analysis for improving question answering in on-line discussions and the news. In ICWSM.

Wilson, T.; Wiebe, J.; and Hoffmann, P. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP.

Implementation

Veridicality Element Classes:

Veridicality Transformations