text categorization moshe koppel lecture 3:authorship attribution
DESCRIPTION
Text Categorization Moshe Koppel Lecture 3:Authorship Attribution. Mostly my own stuff together with Jonathan Schler, Shlomo Argamon, Ido Dagan, Jamie Pennebaker, Jonathan Fine, Kfir Zigdon, Iris Zigdon, Navot Akiva, Dror Mughaz, Eran Messeri. The Authorship Attribution Problem. - PowerPoint PPT PresentationTRANSCRIPT
Text CategorizationMoshe Koppel
Lecture 3:Authorship Attribution
Mostly my own stuff together with
Jonathan Schler, Shlomo Argamon,Ido Dagan, Jamie Pennebaker, Jonathan Fine,Kfir Zigdon, Iris Zigdon, Navot Akiva, Dror Mughaz, Eran Messeri
The Authorship Attribution Problem
Given the known writings of a small number of authors A1,…,An , determine which of them wrote document X.
The vanilla version is actually not very difficult.
History of Authorship Attribution
• Researchers have been trying to solve this problem since the 19th century
• Early research focused on classical works: Bible, Shakespeare, etc.
• The most famous work is Mosteller & Wallace (1964) on the Federalist Papers
• The proliferation of online text offers many new research opportunities
Authorship – Realistic Versions
• No suspects – Can we profile the author of X?
• One suspect – Can we ascertain guilt or innocence?
• Thousands of suspects – Can we pick a needle from the haystack?
Text Categorization
Model for A vs. B
(x1,x2,...,xN)=A (x1,x2,...,xN)=A (x1,x2,...,xN)=A
: :
(x1,x2,...,xN)=B (x1,x2,...,xN)=B (x1,x2,...,xN)=B
Learning Algorithm
A
B
Text
Text
Cleaning+ Feature
Extraction
Features for Authorship Attribution
What features do we want?
• Used consistently by given author regardless of document type
Eliminates topic words
• Might be used differently by different authorsOur challenge: identify these
Olden Days
• Early researchers sought a single “magic” feature• Mainly, they looked at complexity features:
– Sentence or word length
– Type/token ratio
– Hapax legomena
• The idea was the magic feature had a distinct constant value for each author
• This approach isn’t taken seriously these days.
Features for Categorizing by Style
• Function words (and, of, the,..) [Mosteller-Wallace 64]
Features for Categorizing by Style
• Function words
• Syntax – POS n-grams
Features for Categorizing by Style
• Function words
• Syntax – POS n-grams
• SFL trees
SFL Trees
Conjunctions
ConjExtension and, or, but, yet, however,…
ConjElaboration for_example, indeed,…
ConjEnhancement
ConjSpatiotemporal then, beforehand, afterwards, while, during,…
ConjCausalConditional if, because, since, hence, thus, therefore,…
•Similar trees for prepositions, pronouns, modal verbs
•Covers all function words, some POS
•Doesn’t require POS tagging
Features for Categorizing by Style
• Function words
• Syntax
• SFL trees
• Morphology (grammatical prefixes and suffixes, e.g. –ing, re-)
Features for Categorizing by Style
• Function words
• Syntax
• SFL trees
• Morphology
• Complexity measures (e.g. avg. word length, avg. sentence length, various entropy measures)
Features for Categorizing by Style
• Function words• Syntax• SFL trees• Morphology • Complexity measures• “Unstable” words (features that might be
replaced in a rewrite, e.g. huge:large)
Measuring Instability
An interesting idea:
Translate many documents to another
language and translate back.
See what sticks.
We did just that on Reuters 21578 and BNC*.
* We used five different languages to ensure robustness.
Word Stability -- Examples
huge .0
ratio .0
device .0
onto .06
has .28
over .55
nine .98
September .99
Features for Categorizing by Style
• Function words
• Syntax
• SFL trees
• Morphology
• Complexity measures
• “Unstable” words
• Idiosyncrasies
Idiosyncrasies
Human attribution experts use idiosyncrasiese.g. spelling errors, neologisms, quirky syntax
• Use spelling/grammar checker to identify “errors”
• Categorize errors by typee.g. ie ei, nn n
Some problematic feature sets
• Many researchers use content words
• A really easy feature set is character n-grams
• These sets are problematic since they ought to fail when an author changes topics
• In practice, they work very well
Learning -- Historical
• From 1964 until 1990’s all work had flavor of Naïve Bayes
• Idea was to characterize all of each candidate author’s work and see which one X is most similar to
• Various proximity measures used (often after dimension reduction)
• Since 1990’s variety of learning methods applied
Learning Algorithms
Let’s try some old friends (and one new one):
• Naïve Bayes
• Decision Trees
• Winnow
• SVM
• Bayesian regression
Three testbed corpora
• Emails between two correspondents (date split)– Some as short as one word
• Two books by each of nine authors (book split)– Different books have different content
• Twenty bloggers (date split)– Lots of authors
Email corpus – 2 authors
features/learner nb j48 rmwinnow bmr smo FW 60.2% 58.7% 66.1% 68.2% 63.8% POS 61.0% 59.0% 66.1% 66.3% 67.1% FW+POS 65.9% 61.6% 68.0% 67.8% 71.7% SFL 57.2% 57.2% 65.6% 67.2% 62.7% CW 67.1% 66.9% 74.9% 78.4% 74.7% CNG 70.0% 62.0% 69.6% 72.1% 71.7%
Literature corpus – 9 authors
features/learner nb j48 rmwinnow bmr smo FW 51.4% 44.0% 63.0% 73.8% 77.8% POS 45.9% 50.3% 53.3% 69.6% 75.5% FW+POS 56.5% 46.2% 61.7% 75.0% 79.5% SFL 66.1% 45.7% 62.8% 76.6% 79.0% CW 68.9% 50.3% 57.0% 80.0% 84.7% CNG 68.7% 40.8% 47.9% 79.8% 82.7%
Blog corpus – 20 authors
features/learner nb j48 winnowmc bmr smo FW 38.2% 30.3% 51.8% 63.2% 63.2% POS 34.0% 30.3% 51.0% 63.2% 60.6% FW+POS 47.0% 34.3% 62.3% 70.3% 72.0% SFL 35.4% 36.3% 61.4% 69.2% 71.7% CW 56.4% 51.0% 62.9% 72.5% 70.5% CNG 63.2% 46.5% 65.2% 79.3% 79.3%
Some LessonsSome lessons:• SVM and Bayesian regression are the best learners.• FW and single POS are generally enough.• In fact, SFL trees alone are often enough.• Content words and character n-grams work surprisingly
wellSome other facts worth knowing:• Common unstable words are better than FW.• For unedited texts, idiosyncrasies are best.• For some languages, morphology is needed.