getting to know your corpus
DESCRIPTION
Getting to know your corpus. Adam Kilgarriff Lexical Computing Ltd. Bad Science. Ben Goldacre. Bad Science. Ben Goldacre Biases in samples A quarter of the people who tested positive had just been on holiday in Mexico But the research team didn ’ t notice. Bad linguistics. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/1.jpg)
Getting to know your corpusAdam KilgarriffLexical Computing Ltd
![Page 2: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/2.jpg)
Kilgarriff: Getting to Know ...
2
Bad ScienceBen Goldacre
Oct 2012
![Page 3: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/3.jpg)
Kilgarriff: Getting to Know ...
3
Bad ScienceBen GoldacreBiases in samples
A quarter of the people who tested positive had just been on holiday in Mexico
But the research team didn’t notice
Oct 2012
![Page 4: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/4.jpg)
Kilgarriff: Getting to Know ...
4
Bad linguisticsOur corpus study shows X
But what was in the corpus?
Oct 2012
![Page 5: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/5.jpg)
Kilgarriff: Getting to Know ...
5
Bad linguisticsOur corpus study shows X
But what was in the corpus?Moral: Get to know your corpus
Oct 2012
![Page 6: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/6.jpg)
Kilgarriff: Getting to Know ...
6
How?Read it?Too big to readNot designed to be read
Oct 2012
![Page 7: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/7.jpg)
Kilgarriff: Getting to Know ...
7
How?Compare it with other(s)Keyword lists
Oct 2012
![Page 8: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/8.jpg)
Kilgarriff: Getting to Know ...
8
UKWaC vs. enTenTen12
Oct 2012
![Page 9: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/9.jpg)
Kilgarriff: Getting to Know ...
9enTenTen vs. UKWaC accord actually amendment among bad because behavior believe bill blog ca center citizen color defense determine do dollar earth effort election even evil fact faculty favor favorite federal foreign forth guess guy he her him himself his honor human kid kill kind know labor law let liberal like man maybe me military movie my nation never nor not nothing official oh organization percent political post president pretty professor program realize recognize say shall she sin soul speak state suppose tell terrorist that thing think thou thy toward true truth unto upon violation vote voter war what while why woman yes
accommodation achieve advice aim area assessment available band behaviour building centre charity click client club colour consultation contact council delivery detail develop development disabled email enable enquiry ensure event excellent facility favourite full further garden guidance guide holiday improve information insurance join link local main manage management match mm nd offer opportunity organisation organise page partnership please pm poker pp programme project pub pupil quality range rd realise recognise road route scheme sector service shop site skill specialist st staff stage suitable telephone th top tour training transport uk undertake venue village visit visitor website welcome whilst wide workshop www
Oct 2012
![Page 10: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/10.jpg)
Kilgarriff: Getting to Know ...
10enTenTen vs. UKWaC
accord actually amendment among bad because behavior believe bill blog ca center citizen color defense determine do dollar earth effort election even evil fact faculty favor favorite federal foreign forth guess guy he her him himself his honor human kid kill kind know labor law let liberal like man maybe me military movie my nation never nor not nothing official oh organization percent political post president pretty professor program realize recognize say shall she sin soul speak state suppose tell terrorist that thing think thou thy toward true truth unto upon violation vote voter war what while why woman yes
accommodation achieve advice aim area assessment available band behaviour building centre charity click client club colour consultation contact council delivery detail develop development disabled email enable enquiry ensure event excellent facility favourite full further garden guidance guide holiday improve information insurance join link local main manage management match mm nd offer opportunity organisation organise page partnership please pm poker pp programme project pub pupil quality range rd realise recognise road route scheme sector service shop site skill specialist st staff stage suitable telephone th top tour training transport uk undertake venue village visit visitor website welcome whilst wide workshop www
Oct 2012
![Page 11: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/11.jpg)
Kilgarriff: Getting to Know ...
11enTenTen vs. UKWaC
accord actually amendment among bad because behavior believe bill blog ca center citizen color defense determine do dollar earth effort election even evil fact faculty favor favorite federal foreign forth guess guy he her him himself his honor human kid kill kind know labor law let liberal like man maybe me military movie my nation never nor not nothing official oh organization percent political post president pretty professor program realize recognize say shall she sin soul speak state suppose tell terrorist that thing think thou thy toward true truth unto upon violation vote voter war what while why woman yes
accommodation achieve advice aim area assessment available band behaviour building centre charity click client club colour consultation contact council delivery detail develop development disabled email enable enquiry ensure event excellent facility favourite full further garden guidance guide holiday improve information insurance join link local main manage management match mm nd offer opportunity organisation organise page partnership please pm poker pp programme project pub pupil quality range rd realise recognise road route scheme sector service shop site skill specialist st staff stage suitable telephone th top tour training transport uk undertake venue village visit visitor website welcome whilst wide workshop www
Oct 2012
![Page 12: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/12.jpg)
Kilgarriff: Getting to Know ...
12enTenTen vs. UKWaC
accord actually amendment among bad because behavior believe bill blog ca center citizen color defense determine do dollar earth effort election even evil fact faculty favor favorite federal foreign forth guess guy he her him himself his honor human kid kill kind know labor law let liberal like man maybe me military movie my nation never nor not nothing official oh organization percent political post president pretty professor program realize recognize say shall she sin soul speak state suppose tell terrorist that thing think thou thy toward true truth unto upon violation vote voter war what while why woman yes
accommodation achieve advice aim area assessment available band behaviour building centre charity click client club colour consultation contact council delivery detail develop development disabled email enable enquiry ensure event excellent facility favourite full further garden guidance guide holiday improve information insurance join link local main manage management match mm nd offer opportunity organisation organise page partnership please pm poker pp programme project pub pupil quality range rd realise recognise road route scheme sector service shop site skill specialist st staff stage suitable telephone th top tour training transport uk undertake venue village visit visitor website welcome whilst wide workshop www
Oct 2012
![Page 13: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/13.jpg)
Kilgarriff: Getting to Know ...
13enTenTen vs. UKWaC
Core verbs be determine do
guess know let say shall suppose tell think
Pronouns he her him his me my
she
Biber: more informal
Oct 2012
![Page 14: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/14.jpg)
Kilgarriff: Getting to Know ...
14
Two cultures Famous book in England: C.P. Snow Humanities vs. science Linguistics: languages vs computer science Lancaster, Leeds: both
Oct 2012
![Page 15: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/15.jpg)
Kilgarriff: Getting to Know ...
15
Judgements Not all or nothing
Both have (lots of) AmE and BrE Observing patterns
Not right or wrong Where does ‘believe’ belong?
Bible or core verbs? No right answer, could be both
The better you know the data, the better you understand why words are there
Oct 2012
![Page 16: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/16.jpg)
Kilgarriff: Getting to Know ...
16
The maths“this word is twice as common here as there” Simplest approach
Normalise frequencies Per thousand, or per million
Take ratio
For examples Assume two 1m-word corpora
Normalisation not needed Fc=focus corpus Rc= reference corpus
Oct 2012
![Page 17: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/17.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
17Problem 1: You can’t divide by
zero
Standard solution: add one
Problem solved
fc rc ratio buggle 10 0 ? stort 100 0 ? nammikin 1000 0 ?
fc rc ratio buggle 11 1 11 stort 101 1 101 nammikin 1001 1 1001
![Page 18: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/18.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
18
Problem 2: High ratios more common, less interesting for rarer words
fc rc ratio interesting?
spug 10 1 10 no
grod 1000 100 10 yes
• ratio is not enough: frequency matters tooAlso • some researchers: grammar, grammar words• some researchers: lexis, content wordsNo right answerSlider?
![Page 19: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/19.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
19Solution
Don’t just add 1, add n: n=1
n=100
word fc rc fc+n rc+n Ratio Rankobscurish 10 0 11 1 11.00 1middling 200 100 201 101 1.99 2common 12000 10000 12001 10001 1.20 3
word fc rc fc+n rc+n Ratio Rankobscurish 10 0 110 100 1.10 3middling 200 100 300 200 1.50 1common 12000 10000 12100 10100 1.20 2
![Page 20: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/20.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
20
n=1000
word fc rc fc+n rc+n Ratio Rankobscurish 10 0 1010 1000 1.01 3middling 200 100 1200 1100 1.09 2common 12000 10000 13000 11000 1.18 1
![Page 21: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/21.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
21
Summary
word fc rc n=1 n=100 n=1000obscurish 10 0 1st 2nd 3rdmiddling 200 100 2nd 1st 2ndcommon 12000 10000 3rd 3rd 1st
![Page 22: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/22.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
22
But what aboutMutual informationLog-likelihoodChi-squareFisher’s test…Don’t they use cleverer maths?
![Page 23: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/23.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
23
Yes butClever maths is for hypothesis testing
Can you defeat null hypothesis?
Language is not random, so … you always canNull hypothesis never trueHypothesis-testing not informativeClever maths irrelevant
Kilgarriff 2006, CLLT
![Page 24: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/24.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
24
Varying the parameterBAWE
British Academic Written English Nesi and Thompson 2008
Student essays Arts/Humanities, Social Sciences, Life Sciences,
Physical Sciences fc: ArtsHum, rc: SocSci With n=10 and n=1000
![Page 25: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/25.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
25
![Page 26: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/26.jpg)
Oct 2012 Kilgarriff: Getting to Know ...
26
![Page 27: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/27.jpg)
Kilgarriff: Getting to Know ...
27
Parameters for keyword listsLemmas
Could be word forms, word classes
Simplemaths (default: 100, for mix of lexical and grammar
words)
Only all-lowercase-letters Could allow uppercase, or any at all
Minimum 2/3/4 characters Helps get words, not abbreviations etc
Oct 2012
![Page 28: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/28.jpg)
Kilgarriff: Getting to Know ...
28enTenTen vs. UKWaC
Obama Clinton Hillary McCain
Centre Leeds Manchester Edinburgh
Oct 2012
With parameters:• Simplemaths: 10• Uppercase and lowercase• Minimum length =5 (to exclude acronyms)
![Page 29: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/29.jpg)
Kilgarriff: Getting to Know ...
29
Two interlocking questionsHow do two corpora differ
How do two text types differ
Oct 2012
![Page 30: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/30.jpg)
Kilgarriff: Getting to Know ...
30
Two interlocking questions How do two corpora differ
enTenTen vs. UKWaC Interpret as:
Differences of corpus compilation procedures- and/or - Differences of proportions of text types
How do two text types differ BAWE example
Arts/humanities essays vs. Social Sciences essays Any other corpus differences
Unwanted biases But we need to know about them
Oct 2012
![Page 31: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/31.jpg)
Kilgarriff: Getting to Know ...
31
Designed vs crawledTwo main ways to build a large corpusDesign
Start from design spec Select data to fit spec
Crawl Crawl the web, you get what you get
Hot topic Designed (BNC, CNC): more expensive Crawled (*WaC, *TenTen): can you trust it
Oct 2012
![Page 32: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/32.jpg)
Kilgarriff: Getting to Know ...
32
Oct 2012
![Page 33: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/33.jpg)
Kilgarriff: Getting to Know ...
33
enTenTen keywords
Pronouns: our your Encoding: don percent Web: com site email request server internet comments click website online posted web list access data search www files file blog address page
University: article campus faculty graduate information project projects read research science student students
American spelling : behavior center color defense favor favorite labor organizations program programs toward
Bible: believe evil faith forth sin soul thee thou thy unto upon
Politics: current federal global laws nation president security world
Creative industries: author content create digital film game images media movie review story technology
Informal: folks guess guy guys kidsLanguage change: issues Other: code efforts entire focus human include including human located mission persons prior provides
BNC keywords
Pronouns: he herself her Encoding Speech: transcription: cos cent erm gon per pound pounds
Numbers: eight fifty five forty four half hundred nine nineteen seven six ten thirty three twenty two
British spelling: behaviour centre colour defence favour labour programme round towards
British lexical variants: bloody pupils shop
Past tense verbs: got felt turned smiled sat looked stood was said been seemed had went were knew put thought
Particles: away back down off Local government: council firm hospital local industrial police social speaker
Household nouns: bed car door eyes face garden girl hair house kitchen mother room tea
Informal: alright mean quite perhaps sort yeah yes
Language change: chairman Other: although club considerable could head know main manager night there studio yesterday
Oct 2012
![Page 34: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/34.jpg)
Kilgarriff: Getting to Know ...
34
Contrasts in both Cz and EnCrawled 1st and 2nd person forms Web
Designed Time Numbers Past tenses
Oct 2012
![Page 35: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/35.jpg)
Kilgarriff: Getting to Know ...
35
EnTenTen vs. UKWaC and BNC (American)BiblePolitics/electionUniversity
Oct 2012
![Page 36: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/36.jpg)
Kilgarriff: Getting to Know ...
36
EnTenTen vs. UKWaC and BNC (American) Bible Politics/election University
Compare A and B is it a bias of A, or of B?Compare, A, B and C More vantage points, triangulation Get to know your corpus better
Oct 2012
![Page 37: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/37.jpg)
Kilgarriff: Getting to Know ...
37
Quantitative comparison- but isn’t enTenTen more similar to UKWaC
than BNCTalk so far: qualitativeFor this question: quantitative
Distance between two corpora
Kilgarriff 2001: Comparing Corpora Now implemented in Sketch Engine
Oct 2012
![Page 38: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/38.jpg)
Kilgarriff: Getting to Know ...
38
Oct 2012
![Page 39: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/39.jpg)
Kilgarriff: Getting to Know ...
39
SummaryDon’t do bad scienceGet to know your corpus
Compare with others Qualitatively: keyword lists Quantitatively: distances
No excuses The Sketch Engine does all the technical work for
you
The joy of research
Oct 2012
![Page 40: Getting to know your corpus](https://reader035.vdocuments.us/reader035/viewer/2022062520/56815c64550346895dca71e8/html5/thumbnails/40.jpg)
Kilgarriff: Getting to Know ...
40
Thank you
http://www.sketchengine.co.uk
Oct 2012