janusz s. bień - core – aggregating the world’s …nicholas carr, the big switch: rewiring the...
TRANSCRIPT
![Page 1: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/1.jpg)
Facilitating access to digitalized dictionaries
Facilitating access to digitalized dictionaries
Janusz S. Bień
Formal Linguistics DepartmentUniversity of Warsaw
Representing Semantics in Digital LexicographyWarsaw, Poland, June 29–July 1, 2009
![Page 2: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/2.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
(ScanRobot by TREVENTUS Mechatronics GmbH, 2500 pages/hour)
![Page 3: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/3.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
What for?We are not scanning all of those books to be read by people[in October 2005] the [Google] engineer told him [Freeman Dyson]We are scanning them to be read by an AI.
Nicholas Carr, The Big Switch: Rewiring the World, from Edisonto Google, 2008
![Page 4: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/4.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
What for?We are not scanning all of those books to be read by people[in October 2005] the [Google] engineer told him [Freeman Dyson]We are scanning them to be read by an AI.
Nicholas Carr, The Big Switch: Rewiring the World, from Edisonto Google, 2008
![Page 5: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/5.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
What for?We are not scanning all of those books to be read by people[in October 2005] the [Google] engineer told him [Freeman Dyson]We are scanning them to be read by an AI.
Nicholas Carr, The Big Switch: Rewiring the World, from Edisonto Google, 2008
![Page 6: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/6.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
What for?We are not scanning all of those books to be read by people[in October 2005] the [Google] engineer told him [Freeman Dyson]We are scanning them to be read by an AI.
Nicholas Carr, The Big Switch: Rewiring the World, from Edisonto Google, 2008
![Page 7: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/7.jpg)
Facilitating access to digitalized dictionariesIntroduction
Mass digitalization
What for?We are not scanning all of those books to be read by people[in October 2005] the [Google] engineer told him [Freeman Dyson]We are scanning them to be read by an AI.
Nicholas Carr, The Big Switch: Rewiring the World, from Edisonto Google, 2008
![Page 8: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/8.jpg)
Facilitating access to digitalized dictionariesIntroduction
Dirty OCR
Optical Character Recognitionfully automaticno trainingno proof-reading
![Page 9: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/9.jpg)
Facilitating access to digitalized dictionariesDjVu technology
DjVu and DjVuLibre
What is DjVu?Yann Le Cun, Léon Bottou, Patrick Haffner, and Paul G. Howard1996
an image compression technique, a document format,and a software platform for delivering documents imagesover the Internet
http://leon.bottou.org/papers/lecun-2001
DjVuLibreFree (GNU GPL) implementation of DjVu
![Page 10: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/10.jpg)
Facilitating access to digitalized dictionariesDjVu technology
DjVu and DjVuLibre
Some design principles
Action Real-word equivalent Acceptable delayZooming/Panning Moving the eyes Immediate
Next/Previous Page Turning a page < 1 secondRandom Page access Finding a page < 3 seconds
http://leon.bottou.org/papers/lecun-2001
DjVu document formatgraphic layers (foreground, background, stencil)hidden text
![Page 11: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/11.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
A dictionary in DjVu format
The Century Dictionarypublished 1889 to 1891, enhanced 1911digitalized in 2001 almost single-handedly by Jeffery A. Triggs
we created The Century Dictionary Online becauseit is free, it is big, and it is beautiful. I should addfinally, a couple of more reasons: married to DjVutechnology, it is innovative, . . . .
http://global-language.com/CENTURY/
![Page 12: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/12.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
The Century Dictionary home page (old version)
![Page 13: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/13.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
The Century Dictionary: search form
![Page 14: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/14.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
The Century Dictionary: search hit list
![Page 15: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/15.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
The Century Dictionary: a highlighted hit
![Page 16: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/16.jpg)
Facilitating access to digitalized dictionariesSample dictionaries
Polish dictionaries in DjVu format
Important Polish dictionaries with dirty OCRThe Geographical Dictionary of the Polish Kingdomand other Slavic Countries (1880-1902)http://mbc.malopolska.pl/publication/113
‘Warsaw dictionary’ of Polish (1900–1927)http://ebuw.uw.edu.pl/publication/255
The Dictionary of 16th century Polish (1966–2???)http://kpbc.umk.pl/dlibra/publication/17781
Dictionaries without OCRUse the search facility of[Polish] Federation of Digital Librarieshttp://fbc.pionier.net.pl
![Page 17: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/17.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp
Polyinterpretation Indexing Query and Retrieval ProcesorGNU General Public Licensehttp://poliqarp.sourceforge.net/used by
the IPI PAN Corpus (since 2006)National Corpus of Polish (in preparation, demo available since2008)
Sample dataA volume of the Dictionary of 16th century PolishDigitally born, not OCRed!Cf. http://bc.klf.uw.edu.pl/71/and references therein
![Page 18: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/18.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Fom DjVu to Poliqarp
DjVu to XMLdjvutoxml (DjVuLibre library)
From XML to XCESXML Corpus Encoding Standarda subset of Text Encoding Initiative recommendation
Conversion specification: Janusz S. Bień(http://bc.klf.uw.edu.pl/105/)Converter implementation: Piotr Sikora(http://subversion.assembla.com/svn/djvu-fgrep)
![Page 19: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/19.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp - a query
![Page 20: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/20.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp - a query
![Page 21: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/21.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp — the context
![Page 22: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/22.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp — the metadata
![Page 23: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/23.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
Poliqarp — the tags
![Page 24: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/24.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
From Poliqarp to DjVu
An ad hoc solutionSpecification: Janusz S. BieńImplementation: Jakub Wilk
Planned solutionIntegration of Poliqarp with a DjVu viewer
![Page 25: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/25.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
From Poliqarp to DjVu (an ad hoc solution)
![Page 26: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/26.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
From Poliqarp to DjVu (an ad hoc solution)
![Page 27: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/27.jpg)
Facilitating access to digitalized dictionariesDictionaries as corpora
From Poliqarp to DjVu (an ad hoc solution)
![Page 28: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/28.jpg)
Facilitating access to digitalized dictionariesConcluding remarks
Future work
Digitalization tools for philological researchGrant N N519 384036of the Ministry of Science and Higher Education
Since 13 May 2009 to 12 November 2011
Team:Janusz S. Bień (project leader)Jakub Wilk, Krzysztof Szafran, Joanna A. Bilińska
![Page 29: Janusz S. Bień - CORE – Aggregating the world’s …Nicholas Carr, The Big Switch: Rewiring the World, from Edison to Google, 2008 Facilitating access to digitalized dictionaries](https://reader035.vdocuments.us/reader035/viewer/2022070801/5f02968f7e708231d405029d/html5/thumbnails/29.jpg)
Facilitating access to digitalized dictionariesConcluding remarks
Additional informations
The present slideshttp://bc.klf.uw.edu.pl/102
Contacthttp://fleksem.klf.uw.edu.pl/~jsbien
Project site (under construction)http://wbs.klf.uw.edu.pl