part-of-speech tagging for historical english · penn corpora of historical english modern briesh...

Part-of-SpeechTaggingforHistoricalEnglish

YiYangandJacobEisensteinGeorgiaTech

[MuralidharanandHearst,2011&2012]

‣ DigitalhumaniEesresearch

‣ HowdoestheportrayalofmenandwomendifferinShakespeare’splays?

‣ What’sthelanguageusepaMernsinNorthAmericanslavenarraEves?


‣ NLPcanhelp!





‣ NLPcanhelp!




‣ OnlyifNLPworksforhistoricaltexts…

EarlyModernEnglish

Heesaidnobodyhadsaidanythingagtmee.

[HenryOxinden,1660]

EarlyModernEnglish


‣SpellingvariaEon

He againstHe me

[HenryOxinden,1660]

StanfordPOSTagger


‣SpellingvariaEon

Stanford:

StanfordPOSTagger

Heesaidnobodyhadsaidanythingagtmee.X X X

‣SpellingvariaEon

Stanford:Gold:

TransferLossforPOSTagging

0

5

10

15

20

25

3.0

Errorrate

ModernEnglish

[Raysonetal.,2007]

TransferLossforPOSTagging

0

5

10

15

20

25

18.0

3.0

Errorrate

ModernEnglish

EarlyModernEnglish

[Raysonetal.,2007]

Approaches

‣ SpellingnormalizaEon }‣ Mapfromhistoricalspellingstocontemporaryforms.

Raysonetal.(2007)Scheibleetal.(2011)Bollmann(2011)

Approaches

‣ DomainadaptaEon(thiswork)

‣ SpellingnormalizaEon }‣ Mapfromhistoricalspellingstocontemporaryforms.

‣ BuildrobustNLPsystemswithrepresentaEonlearning.

Raysonetal.(2007)Scheibleetal.(2011)Bollmann(2011)

}Yang&Eisenstein(2014)Yang&Eisenstein(2015)

SpellingNormalizaEon

[VARD;BaronandRayson,2008]

Original:Heesaidnobodyhadsaidanythingagtmee.

Normalized:Heesaidnobodyhadsaidanythingagedme.


‣CorrectnormalizaEon




X






‣IncorrectnormalizaEon

X X

against






‣IncorrectnormalizaEon‣FalsenegaEve

X X

against

X

He




X XX

Stanford:Gold:




X XX

Stanford:Gold:

XX

RepresentaEonLearning


RepresentaEonLearning


Hee

saidwascametold…

} HeI

We…

saidwascametold…

}IVOOV Context Context

FeatureEmbeddings

[FEMA;YangandEisenstein,2015]


FeatureEmbeddings


Heesaidnobodyhadsaidanythingagtmee.CurrWord=heeNextWord=said

Prefix1=hSuffix1=e

…

}features

1

2

3

4

FeatureEmbeddings


CurrWord=heeNextWord=said

Prefix1=hSuffix1=e

…

}features

1

2

3

4

u2

Outputembeddings

Inputembeddings

v1

v3

v4

p(ft|f2) / exp

�u2

>vt

�

FeatureEmbeddings



Prefix1=hSuffix1=e

…

}features

1

2

3

4

u2

Outputembeddings

Inputembeddings

v1

v3

v4

p(ft|f2) / exp

�u2

>vt

�

` =TX

t 6=2

logp(ft|f2)

WordEmbeddings

[word2vec;Mikolovetal.,2013]


Prefix1=hSuffix1=e

…

}features

heesaidnobodyhad…

}words

1

2

3

4

1

2

3

4

‣ Wordembeddings

‣ Featureembeddings

WordEmbeddings



Prefix1=hSuffix1=e

…

}features

heesaidnobodyhad…

}words

1

2

3

4

1

2

3

4

‣ Wordembeddings


‣ GenericrepresentaEons

WordEmbeddings



Prefix1=hSuffix1=e

…

}features

heesaidnobodyhad…

}words

1

2

3

4

1

2

3

4

‣ Wordembeddings



‣ Task-specificrepresentaEons

WordEmbeddings



Prefix1=hSuffix1=e

…

}features

heesaidnobodyhad…

}words

1

2

3

4

1

2

3

4

‣ Wordembeddings




‣ Wordco-occurrences

WordEmbeddings



Prefix1=hSuffix1=e

…

}features

heesaidnobodyhad…

}words

1

2

3

4

1

2

3

4

‣ Wordembeddings




‣ Wordco-occurrences

‣ Featureco-occurrences

LearningfromMulEpleDomains


‣PreviousworkonunsuperviseddomainadaptaEoninvolvesintwodomains.

LearningfromMulEpleDomains


‣PreviousworkonunsuperviseddomainadaptaEoninvolvesintwodomains.‣UnsupervisedmulE-domainadaptaEon

MulEpleFeatureEmbeddings






DomainAMributes: Genre Epoch




leMers 1600+

DomainAMributes: Genre Epoch




leMers 1600+

DomainAMributes:


Prefix1=hSuffix1=e

…

}features

1

2

3

4

Genre Epoch




leMers 1600+

DomainAMributes:


Prefix1=hSuffix1=e

…

}features

1

2

3

4

= +(shared) (leMers)

+(1600+)

Genre Epoch




Prefix1=hSuffix1=e

…

}features

1

2

3

4

(shared) (leMers)= + +

(1600+)




Prefix1=hSuffix1=e

…

}features

1

2

3

4

(shared) (leMers)

= + +

(1600+)

u2 = h(shared)2 + h(letters)

2 + h(1600+)2

= + +




Prefix1=hSuffix1=e

…

}features

1

2

3

4

(shared) (leMers) (1600+)

u2 = h(shared)2 + h(letters)

2 + h(1600+)2

= + +

p(ft|f2) / exp

�u2

>vt

�

Experiments

PennCorporaofHistoricalEnglishModernBriEshEnglish(MBE)

1840-1914

1770-1839

1700-1769

0 110,000 220,000 330,000 440,000

343,024

427,424

322,255

#oftokens

EarlyModernEnglish(EME)

1640-1710

1570-1639

1500-1569

0 177,500 355,000 532,500 710,000

640,255

706,587

614,315

#oftokens

[KrochandTaylor,2000;Krochetal.,2004]

TagsetMappings

‣PennCorporaofHistoricalEnglish(PCHE)tagset:83tags‣PennTreebank(PTB)tagset:45tags

[MoonandBaldridge,2007]

TagsetMappings

‣PennCorporaofHistoricalEnglish(PCHE)tagset:83tags‣PennTreebank(PTB)tagset:45tags

[MoonandBaldridge,2007]

ADJ

PCHE PTB

JJADVALSO RB

VB VBVBI… …

Systems

‣Supportvectormachine(SVM)tagger‣SixteenbasicfeaturetemplatesbyRatnaparkhi(1996)

Systems

‣Supportvectormachine(SVM)tagger

‣RepresentaEonlearningmethods

‣SixteenbasicfeaturetemplatesbyRatnaparkhi(1996)

‣Structuralcorrespondencelearning(SCL)‣Brownclustering‣word2vecembeddings‣MulEplefeatureembeddings(FEMA)

[Blitzeretal.,2006;Brownetal.,1992;Mikolovetal.,2013]

TemporalAdaptaEonModernBriEshEnglish(MBE)

1840-1914

1770-1839

1700-1769

0 110,000 220,000 330,000 440,000

343,024

427,424

322,255

#oftokens

EarlyModernEnglish(EME)

1640-1710

1570-1639

1500-1569

0 177,500 355,000 532,500 710,000

640,255

706,587

614,315

#oftokens

Train Train

Test1 Test1

Test2 Test2

0

1.2

2.4

3.6

4.8

6

4.6

Averageerrorrate

Baseline SCL Brown word2vecFEMA

Results:ModernBriEshEnglish

0

1.2

2.4

3.6

4.8

6

4.44.24.34.6

Averageerrorrate



0

1.2

2.4

3.6

4.8

6

3.74.44.24.3

4.6

Averageerrorrate



(-0.9)

(Ourmethod)

0

2.2

4.4

6.6

8.8

11

9.4

BaselineSCL Brown word2vec

FEMA

Averageerrorrate

Results:EarlyModernEnglish

0

2.2

4.4

6.6

8.8

11

8.38.08.29.4


FEMA

Averageerrorrate


0

2.2

4.4

6.6

8.8

11

6.6

8.38.08.29.4


FEMA

Averageerrorrate


(-2.8)

(Ourmethod)

AdaptaEonfromPTB

PennTreebank

ModernBriEshEnglish

EarlyModernEnglish

0 500,000 1,000,000 1,500,000 2,000,000

1,961,157

1,092,703

969,905

#oftokens

Train

Test1

Test2

AdaptaEonfromPTB

StandardevaluaEonscenarioforEnglishPOStagging.

AdaptaEonfromPTB

StandardevaluaEonscenarioforEnglishPOStagging.

‣Lowresourcelanguages‣Specificgenres,styles,orepochs

InsufficientdataannotaEonforhistoricaltexts.

0

4.6

9.2

13.8

18.4

23

18.9

Errorrate

Baseline SCL Brown word2vec FEMA


0

4.6

9.2

13.8

18.4

23

18.318.418.418.9

Baseline SCL Brown word2vec FEMAErrorrate


0

4.6

9.2

13.8

18.4

23

17.518.318.418.418.9



(-1.4)

(Ourmethod)

0

6

12

18

24

30

25.9



0

6

12

18

24

30

24.224.024.125.9




0

6

12

18

24

30

22.124.224.024.1

25.9


(-3.8)(Ourmethod)

0

6

12

18

24

30

22.125.9

Baseline FEMA+VARD

FEMAErrorrate

NormalizaEonvs.RepresentaEonLearning

(-3.8) (-2.6)(-4.9)

RepresentaEonlearning

(-3.8)

FEMA

(-3.8)22.1

0

6

12

18

24

30

23.322.125.9

Baseline VARD FEMA+VARD

FEMAErrorrate


(-3.8) (-2.6)(-4.9)


SpellingnormalizaEon

0

6

12

18

24

30

21.023.322.1

25.9

Baseline VARD FEMA+VARD

FEMAErrorrate


(-3.8) (-2.6)(-4.9)

RepresentaEonlearning+

normalizaEon


SpellingnormalizaEon

token annotaEonsinPCHE annotaEonsinPTB

,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)

.(period) ,(comma;12.3%).(period;87.7%) .(period)

to TO(54.6%)IN(44.3%) TO

all/any/every JJ DT

ErrorAnalysis

‣AnnotaEoninconsistenciesandtagsetmismatches

ErrorAnalysis

token annotaEonsinPCHE annotaEonsinPTB

,(comma) ,(comma;83.4%).(period;16.6%) ,(comma)

.(period) ,(comma;12.3%).(period;87.7%) .(period)

to TO(54.6%)IN(44.3%) TO

all/any/every JJ(quanEfier) DT

‣AnnotaEoninconsistenciesandtagsetmismatches

Conclusions

Conclusions

‣ FeatureembeddingsoutperformwordembeddingsbyexploiEngtask-specificinformaEoninfeaturetemplates.

Conclusions

‣ RepresentaEonlearningandspellingnormalizaEonarecomplementaryforimprovingtaggingperformance.


Conclusions

‣ RepresentaEonlearningandspellingnormalizaEonarecomplementaryforimprovingtaggingperformance.

‣ TagsetmismatchesmakeithardtoevaluatemodernPOStaggersforhistoricalEnglish.


part-of-speech tagging for historical english · penn corpora of historical english modern briesh...

Documents