mallet presentation

Mallet Demo

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

“A JAVA-BASED PACKAGE FOR STATISTICAL NATURAL LANGUAGE

PROCESSING, DOCUMENT CLASSIFICATION, CLUSTERING, TOPIC

MODELING, INFORMATION EXTRACTION, AND OTHER MACHINE LEARNING

APPLICATIONS TO TEXT.”

What is Mallet?

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

WRITTEN BY ANDREW MCCALLUM ET AL. AT UMASS AMHERST

What is Mallet?

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

1. DOWNLOAD MALLET2. PREPARE TEXT FOR INPUT

3. RUN MALLET4. MODIFY HYPER-PARAMETERS

5. ANALYZE TOPICS

How do you use mallet?

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

What is The Command Line?

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

1) Download Mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Mallet says: “Now run this command:

bin/mallet import-dir --input sample-data/web/* --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

bin/mallet import-dir --input sample-data/web/* --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input sample-data/web/* --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input sample-data/web/* --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input ~/Desktop/Mallet-demo/Transcripts-bulstrode --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input ~/Desktop/Mallet-demo/Transcripts-bulstrode --output web.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input ~/Desktop/Mallet-demo/Transcripts-bulstrode --output topic-input_bulstrode.mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Modifying the command:

~/downloads/mallet-2.0.7/bin/mallet import-dir --input ~/Desktop/Mallet-demo/Transcripts-bulstrode --output topic-input_bulstrode.mallet --keep-sequence --remove-stopwords

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

2) Import Directory into Mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

3) Run Mallet

Mallet says, run the command:

bin/mallet train-topics --input topic-input.mallet --num-topics 100 --output-state topic-state.gz

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

3) Run Mallet

Mallet says, run the command:

~/downloads/mallet-2.0.7/bin/mallet train-topics --input topic-input_bulstrode.mallet --num-topics 100 --output-state topic-state.gz 2> ~/Desktop/Mallet-demo/bulstrode-100t.txt

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

3) Run Mallet

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

4) Modify Model

--optimize-interval #: hyperparameter optimization

--optimize-burn-in #: iterations before above--output-doc-topics doc.txt: outputs topic

composition of documents--output-topic-keys doc.txt: more complete

data--inferencer-filename doc.txt: inferencer tool

(?)--evaluator-filename doc.txt: how to get topic

proportions for new documentsHannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

Background: Bulstrode Papers

Sir Richard Bulstrode (1610-1711), English author, diplomat, soldier

Papers were “communications made to him from this country [England?] in the form of news-letters giving, almost daily, the most minute details of everything going on at court and in the country.”

Research Questions, Bulstrode Papers:

What important topics were discussed during Bulstrode’s life?

What attitude was taken towards those topics?

Can we identify patterns in topics or attitudes over time?

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

0 10 house day bill ordered made sr lords mr till morning read resolved committee debate brought comittee bee john morrow

1 10 duke lord earle french court king de yesterday morning towne france night monday arrived tuesday present good returne dated

2 10 ye yt wch king ld lds day order commons ym great agst st againe doe ma goe night wth

3 10 maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

4 10 majestye hath sr councell soe late ambassador hee letters king majesty bee lord great made french mr prince himselfe

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

0 10 house day bill ordered made sr lords mr till morning read resolved committee debate brought comittee bee john morrow

1 10 duke lord earle french court king de yesterday morning towne france night monday arrived tuesday present good returne dated

2 10 ye yt wch king ld lds day order commons ym great agst st againe doe ma goe night wth

3 10 maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

4 10 majestye hath sr councell soe late ambassador hee letters king majesty bee lord great made french mr prince himselfe

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

0 10 house day bill ordered made sr lords mr till morning read resolved committee debate brought comittee bee john morrow

1 10 duke lord earle french court king de yesterday morning towne france night monday arrived tuesday present good returne dated

2 10 ye yt wch king ld lds day order commons ym great agst st againe doe ma goe night wth

3 10 maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

4 10 majestye hath sr councell soe late ambassador hee letters king majesty bee lord great made french mr prince himselfe

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

0 10 house day bill ordered made sr lords mr till morning read resolved committee debate brought comittee bee john morrow

1 10 duke lord earle french court king de yesterday morning towne france night monday arrived tuesday present good returne dated

2 10 ye yt wch king ld lds day order commons ym great agst st againe doe ma goe night wth

3 10 maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

4 10 majestye hath sr councell soe late ambassador hee letters king majesty bee lord great made french mr prince himselfe

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

5) Interpret a topic model

0 10 house day bill ordered made sr lords mr till morning read resolved committee debate brought comittee bee john morrow

1 10 duke lord earle french court king de yesterday morning towne france night monday arrived tuesday present good returne dated

2 10 ye yt wch king ld lds day order commons ym great agst st againe doe ma goe night wth

3 10 maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

4 10 majestye hath sr councell soe late ambassador hee letters king majesty bee lord great made french mr prince himselfe

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

Topic Three: War at Sea?

maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

--output-doc-topicsHannah Alpert -Abrams, University of Texas at Austin, Fall 2014

Topic Three: War at Sea?

maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

File 1672-04-12' Thursday, 4th.--His Maty has given orders that letters be imediately sent to all the Collonyes and Plantacons abroad to give them notice of his warr with Holland, and therefore to comand them to seize and secure all the effects of the people in those places, and to be carefull in securing the countryes. His Maty has given orders for the makeing commissions for another regiment to be raysed, under the command of my Ld Power as Collonell.

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

Topic Three: Third Anglo-Dutch War

maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

Topic Three: Third Anglo-Dutch War

maty dutch holland matie men pleased great haveing persons fleet mr warr severall wee sr hear states highness sea

File 1672-04-12' On Satturday last the commissions for Prince Rupert's regiment were signed by his Majesty, being in all 12 troops of Dragoons; his Maty was also pleased to signe commissions for another regiment of Foot to be now raysed, under the comand of Coll. Fitzgerald as Collonell, Mr Henry Butler, Lt Colonell and Sr Edward Carleton, Major.' Monday, 1st.--Sr John Talbot's company in his Matie's Guards is given to Captaine Guilliames, and Captaine Downing's company to Captaine Henry Williams; and Capt Downing has Sr Charles Wheler's company, also in the Guards.' His Maty has been pleased to give two comissions to the Duke of Yorke, one for impowering him to issue out his warrants to the Court of Admiralty to give sentence against what prizes shall be taken, and the other for impowering him to issue his comissions to privateers to take and destroy the King's enemyes.

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014

Take-Aways

The difficulties and rewards of working on the command line.

How to take a collection of topics and conduct an analysis of those topics.

How much human intervention is necessary to “process” the text prior to modeling.

Hannah Alpert -Abrams, University of Texas at Austin, Fall 2014