criticism mining: text mining experiments on book, movie and music reviews

Criticism Mining: Text Mining Experiments on Book, Movie and Music Reviews

Xiao Hu, J. Stephen Downie, M. Cameron Jones

The International Music Information Retrieval Systems Evaluation Lab (IMIRSEL)

University of Illinois at Urbana-Champaign

THE ANDREW W. MELLON FOUNDATION

Agenda

MotivationCustomer reviews in epinions.comExperimental SetupData setResultsConclusions & Future Work

MotivationCritical consumer-generated reviews of humanities

materials a rich resource of reviewers’ opinions, and

background / contextual informationself-organized: pave ways to automatic

processing

Text mining: mature and ready to useCriticism mining: provides a tool to assist

humanities scholarsLocatingOrganizingAnalyzing critical review content

Customer Reviews

Published on www.epinions.com Focused on the book, movie and music Each review associated with:

a genre label a numerical quality rating

numerical rating associated

used in our experiments

Music Genres

Jazz, Rock, Country, Classical, Blues, Gospel, Punk, .…

Renaissance, Medieval, Baroque, Romantic, …

28 Major Genre Categories

Experimental Setup

to build and evaluate a prototype criticism mining system that could automatically : predict the genre of the work being reviewed predict the quality rating assigned to the

reviewed item differentiate book reviews and movie reviews,

especially for items in the same genredifferentiate fiction and non-fiction book

reviews

Data set

Reviews on

Book Movie Music

#. Of reviews

1800 1650 1800

#. Of genres

9 11 12

Mean of review length

1,095 words 1,514 words 1,547 words

Std. Dev. of review length

446 words 672 words 784 words

Term list size

41,060 47,015 47,864

Genre Taxonomy

Book MovieAction / Thriller1 Action /Adventure1

Juvenile Fiction 2 Children 2

Humor 3 Comedies 3

Horror 4 Horror/Suspense 4

Music & Performing Arts 5

Musical & Performing Arts 5

Science Fiction & Fantasy 6

Science-Fiction / Fantasy 6

Biography & Autobiography

Documentary

Mystery & Crime Dramas

Romance Education/General Interest

Japanimation (Anime)

War

Genre Taxonomy : Book

Fiction Non-fictionAction / Thriller1 Humor 3

Juvenile Fiction 2 Music & Performing Arts 5

Horror 4 Biography & Autobiography


Mystery & Crime

Romance

Genre Taxonomy : Music

Blues Heavy Metal

Classical International

Country Jazz Instrument

Electronic Pop Vocal

Gospel R&B

Hardcore/Punk

Rock & Pop

The genre labels and the rating information provided the ground truth for experiments

Data Preprocessing

HTML tags were stripped out; Stop words were NOT stripped out;Punctuation was NOT stripped out;

They may contain stylistic informationTokens were stemmed

Categorization Model & Implementation

Naïve Bayesian (NB) ClassifierComputationally efficientEmpirically effective

Text-to-Knowledge (T2K) ToolkitA text mining frameworkReady-to-use modules and itinerariesNatural Language Processing tools

integratedSupporting fast prototyping of text mining

NB itinerary in T2K

Data Preprocessing

NB Classifier

Results & Discussions

Genre Classification

Reviews on Book Movie MusicNumber of genres 9 11 12Reviews in each genre 200 150 150Term list size (terms) 41,060 47,015 47,864Mean of review length (words)

1,095 1,514 1,547

Std Dev of review length (words)

446 672 784

Mean of precision 72.18%

67.70%

78.89%

Std Dev of precision 1.89% 3.51% 4.11%5 fold random cross validation for book and movie reviews3 fold random cross validation for music reviews

Confusion : Book Reviews

Classified As

Action

Bio. Hor.

Hum.

Juv. Mus.

Mys.

Rom.

Sci.

Action 0.61 0.01 0.06 0.01 0.02 0.03 0.20 0.05 0.02

Bio. 0.04 0.70

0.01 0.05 0.03 0.13 0.01 0.03 0

Horror 0.09 0 0.66

0 0.05 0 0.12 0.02 0.06

Humor 0.01 0.10 0 0.74 0.03 0.08 0.01 0.01 0.03

Juvenile 0.01 0.01 0 0.07 0.86

0.02 0 0.02 0

Music 0 0.09 0 0 0.01 0.89

0 0 0.01

Mystery 0.20 0 0.01 0 0.01 0 0.70

0.05 0.04

Romance 0.06 0.01 0.01 0 0.04 0 0.08 0.78 0.03

Science 0.03 0 0.02 0.01 0.11 0.03 0.01 0.13 0.66

Confusion : MovieClassified As

Act. Ani. Chi. Com.

Doc.

Dra.

Edu.

Hor.

Mus.

Sci. War

Action 0.77

0 0 0.01 0 0.01 0.02 0 0 0.10 0.09

Anime 0 0.89

0.03 0.03 0 0 0 0 0 0.05 0

Children

0.02 0.01 0.95

0 0.01 0.01 0.01 0 0 0 0

Comedy

0.09 0.01 0.06 0.52 0.03 0.17 0.06 0.01 0.03 0.01 0.02

Docu. 0.02 0 0 0.04 0.63

0.01 0.19 0 0.09 0 0.02

Drama 0.16 0 0 0.12 0.10 0.45

0.05 0.03 0.03 0.01 0.04

Edu. 0 0 0.02 0.02 0.31 0.03 0.57

0 0 0.01 0.03

Horror 0.15 0.02 0.02 0.02 0.03 0.02 0.05 0.69

0 0.10 0.02

Music 0 0 0 0.01 0.18 0 0 0 0.81

0 0

Science 0.04 0.01 0.02 0 0.06 0.01 0.02 0.03 0 0.76 0.05War 0.11 0 0.01 0.01 0.08 0.08 0.05 0.03 0.02 0.02 0.59

Confusion : MusicClassified

AsBlu. Cla. Cou Ele. Gos. Pun. Met. Int’l Jazz Pop. RB Roc.

Blues 0.61 0 0.10 0 0 0 0 0 0 0 0 0.29

Classical

0 0.94 0 0.03 0 0 0 0 0 0 0 0.03

Country 0 0 0.92 0 0.03 0 0 0 0 0 0 0.06

Electr. 0 0 0 0.92 0 0 0.06 0 0 0 0 0.03

Gospel 0 0 0.05 0 0.80 0 0 0 0 0 0.05 0.10

Punk 0 0 0 0.05 0 0.71 0.05 0 0 0 0 0.19

Metal 0 0 0 0 0 0 0.89 0 0 0 0 0.11

Int’l 0 0.04 0.00 0.04 0 0 0 0.81 0 0 0 0.04

Jazz 0 0 0 0.04 0 0 0 0 0.89 0.04 0 0.04

Pop Vo. 0 0 0.04 0.07 0 0 0 0.04 0.07 0.68 0 0.11

R&B 0 0 0 0 0 0 0 0 0 0.06 0.88 0.06

Rock 0.03 0 0.03 0 0 0 0.03 0 0 0.03 0 0.89

Rating Classification

Five-class classification1 star vs. 2 stars vs. 3 stars vs. 4 stars vs 5

starsBinary Group classification

1 star + 2 stars vs. 4 stars + 5 starsad extremis classification

1 star vs. 5 stars5 fold random cross validation for all experiments

Rating : Book Reviews

Experiments 5 classes

Binary Group

Ad extremis

Number of classes 5 2 2Reviews in each class 200 400 300Term list size (terms) 34,123 28,339 23,131Mean of review length (words)

1,240 1,228 1,079


549 557 612

Mean of precision 36.70% 80.13% 80.67%Std Dev of precision 1.15% 4.01% 2.16%

Rating : Movie Reviews


Binary Group

Ad extremis


1,640 1,645 1,409


788 770 724


82.27%

85.75%

Std Dev of precision 2.27% 2.02% 1.20%

Rating : Music Reviews


Binary Group

Ad extremis


1,875 2,032 1,842


913 912 956


79.75%

85.94%


Confusion : Book Reviews

Classified As

1 star

2 stars

3 stars

4 stars

5 stars

1 star 0.45 0.21 0.15 0.09 0.10

2 stars 0.24 0.36 0.19 0.12 0.09

3 stars 0.11 0.17 0.28 0.22 0.21

4 stars 0.05 0.06 0.17 0.41 0.31

5 stars 0.04 0.07 0.17 0.26 0.46

Confusion : Movie Reviews

Classified As

1 star

2 stars

3 stars

4 stars

5 stars

1 star 0.49 0.19 0.17 0.08 0.072 stars 0.15 0.45 0.23 0.11 0.063 stars 0.04 0.24 0.28 0.27 0.174 stars 0.05 0.13 0.13 0.41 0.275 stars 0.07 0.03 0.16 0.20 0.54

Confusion : Music Reviews

Classified As

1 star

2 stars

3 stars

4 stars

5 stars

1 star 0.61 0.24 0.07 0.05 0.022 stars 0.24 0.15 0.36 0.15 0.093 stars 0.11 0.13 0.41 0.20 0.154 stars 0.03 0.06 0.10 0.32 0.485 stars 0 0 0.09 0.11 0.80

Classification of Book and Movie Reviews 1Reviews on all available genres

Books : 9 genres; Movies : 11 genres

Reviews on individual, comparable genresBook Movie

Action / Thriller1 Action /Adventure1

Juvenile Fiction 2 Children 2

Humor 3 Comedies 3

Horror 4 Horror/Suspense 4

Music & Performing Arts 5

Musical & Performing Arts 5


Science-Fiction / Fantasy 6

Classification of Book and Movie Reviews 2

Eliminated words that can directly suggest the categories: "book", "movie", "fiction", "film", "novel", "actor",

"actress", "read", "watch", "scene" Frequently occurred in each category, but not bothTo make things harder / avoid oversimplifying

Results suggest stylistic difference in users’ criticisms on books and movies

5 fold random cross validation for all experiments

Book vs. Movie Reviews 1

Genre All Genres

Action Horror

Number of classes 2 2 2

Reviews in each class 800 400 400

Term list size (terms) 49,263 24,552 25,509

Mean of review length (words)

1,608 933 1,779


697 478 546


95.63%

98.12%



Genre Humor/Comedy

Juvenile /Children

Number of classes 2 2

Reviews in each class 400 400

Term list size (terms) 26,713 21,326


1,091 849


625 333


97.87%

Std Dev of precision 1.05% 0.71%


Genre Music & Performing Arts

ScienceFiction & Fantasy

Number of classes 2 2

Reviews in each class 400 400

Term list size (terms) 23,217 25,088


791 1,011


531 544

Mean of precision 97.02% 97.25%

Std Dev of precision 1.49% 1.91%

Classification of Fiction and Non-fiction Book Reviews 1

Fiction Non-fiction

Action / Thriller1 Humor 3

Juvenile Fiction 2 Music & Performing Arts 5

Horror 4 Biography & Autobiography


Mystery & Crime

Romance

Classification of Fiction and Non-fiction Book Reviews 2

Eliminated words that can directly suggest the categories: "fiction", "non", "novel", "character", "plot", and

"story" Frequently occurred in each category, but not bothTo make things harder / avoid oversimplifying

Results suggest stylistic difference in users’ criticisms on fiction books and non-fiction ones

5 fold random cross validation for all experiments

Fiction vs. Non-fiction Book Reviews

Experiment Fiction vs. Non-fiction

Number of classes 2

Reviews in each class 600

Term list size (terms) 35,210


1,220


493


Std Dev of precision 1.16%

Confusion : Fiction vs. Non-fiction Book Reviews

Classified As

Fiction

Non-fiction

Fiction 0.98 0.02

Non-Fiction

0.09 0.91

ConclusionsCustomer reviews are an excellent

resource for studying humanities materialsSuccessful experiments:

High classification precisions:

Genres; Ratings; Book vs. movie reviews

Fiction vs. non-fiction book reviewsReasonable confusions

Text mining techniques can help find important information about the materials being reviewed

Criticism Mining : make the ever-growing consumer-generated review resources useful to humanities scholars.

Future work

More text mining techniquesdecision trees, frequent pattern mining

Other critical textblogs, wikis, etc

Other facets of reviews “usage” in music reviews

Feature studies answer the “why” questions

References Argamon, S., and Levitan, S. (2005). Measuring the Usefulness of

Function Words for Authorship Attribution. Proceedings of the 17th Joined International Conference of ACH/ALLC.

Downie, J. S., Unsworth, J., Yu, B., Tcheng, D., Rockwell, G., and Ramsay, S. J. (2005). A Revolutionary Approach to Humanities Computing?: Tools Development and the D2K Data-Mining Framework. Proceedings of the 17th Joined International Conference of ACH/ALLC.

Hu, X., Downie, J. S., West, K., and Ehmann, A. (2005). Mining Music Reviews: Promising Preliminary Results. Proceedings of the Sixth International Conference on Music Information Retrieval (ISMIR).

Sebastiani, F. (2002). Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34, 1.

Stamatatos, E., Fakotakis, N., and Kokkinakis, G. (2000). Text Genre Detection Using Common Word Frequencies. Proceedings of 18th International Conference on Computational Linguistics.

Questions?

IMIRSEL

Thank you!

THE ANDREW W. MELLON FOUNDATION

criticism mining: text mining experiments on book, movie and music reviews

Documents