text modeling with r, python, and spark

Modeling Text Data

Small

Big

Cluster Topics

1 2

3

Technologies

Small

Big

Cluster Topics

Clustering the SOTU

Small

Big

Cluster Topics

1

Data Set• 70 years of the State of the Union address

• 1945 (Truman) - 2015 (Obama)

• Avg. Length: ~ 6,700 words

• longest: ~34,000 words

• shortest: ~ 2,000 words

• total: 467,000 words

• Raw Data: 2.4 MB

Pipeline

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoyed twenty-two months uninterrupted economic recovery recovery not enough prevail long run expand long-run strength economy

Config Wrangle Model Cluster Visualize

america enjoyed twenty-two months uninterrupted economic recovery recovery not enough prevail long run expand long-run strength economy

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Config Wrangle Model Cluster Visualize

Topic Modeling SOTU

Small

Big

Cluster Topics

2

Data Set• 70 years of the State of the Union address

• 1945 (Truman) - 2015 (Obama)

• Avg. Length: ~ 6,700 words

• longest: ~34,000 words

• shortest: ~ 2,000 words

• total: 467,000 words

• Raw Data: 2.4 MB

Pipeline

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoy twenty-two month uninterrupted economy recovery recovery not enough prevail long run expand long-run strength economy

Config Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoy twenty-two month uninterrupted economy recovery recovery not enough prevail long run expand long-run strength economy

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Topic Modeling Congress

Small

Big

Cluster Topics

3

Data Set (Congress loves to talk)

• 20 years of Congressional Hearings (1995 - 2015)

• 19,381 documents (about 1,000 a year)

• Avg. Length: ~ 32,000 words (5x SOTU)

• longest: ~ 900,000 words (length of all 7 HP books)

• shortest: ~ 50 words

• total: 613 million words (1,300x SOTU)

• Raw Data: 3.8 GB

Pipeline

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

America has enjoyed twenty-two months of uninterrupted economic recovery. But recovery is not enough. If we are to prevail in the long run, we must expand the long-run strength of our economy.

america enjoy twenty-two month uninterrupted economy recovery recovery not enough prevail long run expand long-run strength economy

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Config Wrangle Model Extract Visualize

Modeling Text Data

Small

Big

Cluster Topics

1 2

3

exaptive.com/blog

Frank D. Evans@frankdevans

@exaptive

slideshare.net/frankdevansgithub.com/frankdevans/odsc_meetup_text_processing