1

Automated Digital Libraries

William Y. ArmsDepartment of Computer Science

Cornell University

2

Two Questions

3

Before Digital Libraries

Access to scientific, medical, legal information

In the United States:

-- excellent if you belonged to a rich organization (e.g, a major university)

-- very poor otherwise

In many countries of the world:

-- very poor for everybody

4

Question 1

Must access to scientific and professional information be expensive?

5

Research Libraries are Expensive

library materials

buildings & facilities

staff

6

The Potential of Digital Libraries

materials

open access

computers & networks

staff

?

7

Question 2

How effectively can computers be used for the skilled tasks of professional librarianship?

-- Time horizon: 5 to 20 years

-- All materials in digital form

8

Automated Library Services

9

Skilled Librarianship

People are skilled at judgment, understanding, discrimination, etc.:

-- selection

-- cataloguing, indexing

-- seeking for information

-- evaluating information

-- reference service

Can computers provide equivalent services?

10

Equivalent Services

Example: Cataloguing rules

-- Application of cataloguing rules to monographs is skilled

-- It is hard to imagine a computer system with these skills

but ...

-- Catalogs and cataloguing rules are the means not the end

11

Equivalent Services

Information discovery

Why are web search services the most widely used information discovery tools in universities today?

12

Conventional Criteria

Web search services have many weaknesses

-- selection is arbitrary

-- index records are crude

-- no authority control

-- duplicate detection is weak

-- search precision is deplorable

yet they clearly satisfy important requirements ...

13

Effectiveness of Web Search

Inspec v. Google

Google is usually superior for general computing and computer science questions

> Broader coverage

> Adequate indexing records

> Better ranking

14

Simple Algorithms

+

Immense Computing Power

15

History: Licklider

J. C. R. LickliderLibraries of the Future, 1965

-- envisaged digital libraries for scientists at their place of work

-- listed desiderata for a digital library

-- studied construction of fully automated digital libraries

-- put emphasis on artificial intelligence and natural language processing

16

History: Licklider

Licklider's predictions for digital libraries were remarkably good, but ...

-- over optimistic about progress in artificial intelligence

-- underestimated what can be done by brute force computing

17

Brute Force Computing

Few people can appreciate the power of Moore's Law

-- Computing power doubles every 18 months

-- Increases 100 times in 10 years

-- Increases 10,000 times in 20 years

Simple algorithms + immense computing powermay outperform human intelligence

18

Brute Force Computing

Example

Creators of the world champion chess program (Deep Thought later Deep Blue)

-- moderate chess players

-- simple tree-search algorithm

-- very, very fast computer hardware

19

An Anecdote

The question (Marvin Minsky)

-- How would you design as computer system that can answer questions such as, "Why was the space station a bad idea?"?

The answer (Danny Hillis)

-- Design much more powerful computers!

20

Examples of

Automated Digital Library Services

21

Web Search

Brute force indexing and retrieval

-- retrieve every page on the web

-- index every word

-- repeat every month

Getting better all the time

-- improved algorithms

-- faster computers and networks

-- analysis of users

22

Web Search

Ranking algorithms

Closeness of match

-- vector space and statistical methods

(Salton, et al., c. 1970)

Importance of digital object

-- Google ranks web pages by how many other pages link to them, gives greater weight to links from higher ranking pages.

(NSF/DARPA/NASA Digital Libraries Initiative)

23

Archiving and Preservation

Internet Archive

-- Monthly, web crawler gathers every open access web page with associated images

-- Web pages are preserved for future generations

-- Files are available for scholarly research

not perfect ...

-- HTML pages, images; no Java applets, style sheets

-- materials are dumped with no organization or indexing

-- access for scholars is rudimentary

24

Reference Linking

Web of Science (ISI)

-- input: combination of automatic means, skilled people

-- limited number of journals

-- very expensive

ResearchIndex (a.k.a. CiteSeer, a.k.a. ScienceIndex)(NEC)

-- fully automatic

-- all open access material in computer science

-- a free service

25

Beyond Text

Informedia (Carnegie Mellon)

Automatic processing of segments of video, e.g., television news. Algorithms for:

-- dividing raw video into discrete items-- generating short summaries-- indexing the sound track using speech recognition-- recognizing faces-- searching using natural language processing

(NSF/DARPA/NASA Digital Libraries Initiative)

26

Costs and Benefits

27

Costs of Catalogs and Indexes

Catalog, index and abstracting records are very expensive when created by skilled professionals

-- only available for certain categories of material (e.g., monographs, scientific journals)

-- contain limited fields of information (e.g., no contents page)

-- restricted to static information

High costs reduce effectiveness and access

28

Costs of Automated Digital Libraries

The Google company

-- 5.5 million searches daily

-- 85 people (half technical, 14 with Ph.D. in computing)

-- 2,500 PCs running Linux, with 80 terabytes of disk

The Internet Archive

-- 7 people with support from Alexa

(March 2000)

29

Overall

If you are rich ...

-- Research libraries, using commercial information services, provide excellent service at very high cost to a favored few

-- Automated digital libraries are a long way from providing the personal reference service available to a faculty member at a well-endowed university

but ...

30

The Model T Library

The Model T Ford, with mass production, brought car travel to the masses ...

-- Automated digital libraries, with open access materials, can already provide good service at low cost-- In the future automated digital libraries can bring scientific, scholarly, medical and legal information to everybody at negligible cost

31

A Footnote

32

Library Expertise

The future of scientific and professional information is tied to computing, but ...

-- automated digital libraries need small teams of highly skilled people

-- development of automated digital libraries is bypassing libraries (Google, ResearchIndex, Informedia, Internet Archive)

The level of computing expertise in U.S. research libraries is depressingly low

33

Further reading

William Y. Arms, "Automated digital libraries." To be submitted to D-Lib Magazine, July/August 2000.

William Y. Arms, "Economic models for open-access publishing." iMP, March 2000. http://www.cisp.org/imp/march_2000/03_00arms.htm

34

Automated Digital Libraries

William Y. ArmsDepartment of Computer Science

Cornell University