1Tefko Saracevic

Types & structures of information resources

What is out there for searching ? What’s under the hood?

-essential knowledge for searchers

tefkos@rutgers.edu; http://comminfo.rutgers.edu/~tefko/

Central ideasAs a searcher you start with knowing:

Information resources• What is out there available

for searching• And there is a LOT!• In this lecture & course we

will explore a sample only– to illustrate

• from which you can generalize

• and explore later more fully in other courses or professionally

Their organization

• How structured, prepared– indexed, classified, tagged,

labeled, abstracted, full text treated … … …

– stored– made accessible

• All in laying the ground for searching

• Knowing what is under the hood

Tefko Saracevic 2

Content Structure

ToC

1. Definitions & terminology2. Examples of vendors3. Structure of records in databases4. Indexes – as used in searching 5. Conclusion

Tefko Saracevic 3

A few concepts that we are familiar with, but still worth revisiting

1. Definitions & terminology

Tefko Saracevic 4

DefinitionsResource: source of help: somebody or

something that is a source of help or information

Generic: A broad range of sources of information in a variety of formats

The data and information assets of an organization , incl. a library

Databases, files, systems containing organized information records

– Dialog, Google are inf. resources

Database ( from Webopedia)

A collection of information organized in such a way that a computer program can quickly select desired pieces of data. You can think of a database as an electronic filing system.

Tefko Saracevic 5

Information resource:

Definitions (cont.)

From Webopedia again:

Traditional databases are organized by fields, records, and files.

• A field is a single piece of information

• A record is one complete set of fields

• And a file is a collection of records.

• E.g. a telephone book is analogous to a file. It contains a list of records, each of which consists of three fields: name, address, and telephone number

• A catalog is a file. It contains a list of records (catalog entries) describing books in a library . Each record has fields, such as author, title, publisher, date, subject headings ….

Tefko Saracevic6

On fields for searching • Records (documents , objects)

used in information resources are always organized in fields– but different resources may and

do use different set of fields– metadata provides information

ABOUT a record; used for instance in Web records; always organized in fields

• Indexes used in searching are organized, divided by fields

• Fields serve to guide, point out, or otherwise facilitate searching

• Searching is automatically always done by fields, even if one does not know that or has no idea of fields

• But more about fields later

Tefko Saracevic 7

8

Who provides inf. resources for searching?

• Terminology as to who & what can be confusing & not consisted - so beware & do your own translation– Provider: a producer of databases; there are great many

providers covering many fields• e.g. Dept. of Education produces ERIC – a database of abstracts &

indexes of educational materials (articles, reports)

– Vendors or aggregators: organizations or companies that get databases from providers or set of sources like journals from publishers & organize them for searching; there is a large number of vendors

– some providers are their own vendors:• e.g. Chemical Abstract runs STN (Scientific & Technical Network)

Tefko Saracevic

Illustrates the ever changing information industry

2. Examples of vendors

Tefko Saracevic 9

Example of a vendor:• Dialog is oldest on the market

– started in 1972• Acquires databases from

information providers– it has over 900 databases

• Organizes content according to uniform structures

• Describes the content– done in Bluesheets

• a most important search tool for you!

• Provides uniform & complex searching capabilities– geared toward professionals

• you have to master them for effective searching

• Creates some own files– e.g super indexes as Dialindex

• Access– mostly through libraries &

companies as subscribers – RUL does not have it, but in class

free access

Tefko Saracevic 10

Story of Dialogillustrative of turbulences in inf. industry

1964 Roger Summit started Information Sciences Laboratory at Lockheed Missile & Space Company

– in the 1960’s developed Recon – online system for NASA (government contract)

1972 Summit convinced Lockheed of online commercial potential & it went public as Dialog

– advent of online information industry

1981 became subsidiary of Lockheed– moved to Palo Alto, CA

1989, the company was sold to Knight-Ridder - had other inf. resources

– incorporated DataStar, a European online company with 350 mostly European oriented databases - still there

1997 Dialog was bought by the U.K.-based M.A.I.D. Corp.

– moved to Cary, N.C. – still there

2000 The Thomson Corporation (now ThompsonReuters) acquired Dialog

– in 1992 Thomson bought ISI with citation indexes that became Web of Knowledge incl. Web of Science

2008 Dialog was bought by ProQuest– ProQuest was Bell & Howell, also UMI,

also University Microfilm …– has many inf. products & services– among them CSA another online

vendor with over 100 databases

Still in business!

Tefko Saracevic 11

BTW – why do we still teach Dialog?• Dialog is a legacy database –

grandady – some call it a dinosaurs

• So why do we use Dialog for exercises?

• Several reasons:• oldest and largest surviving

vendor• by far has a most

comprehensive set of databases

• has a well developed instructional program

But most importantly:– serves as a good test bed to

develop searching skills that are generalizable• learning what is under the hood

of all databases – what you will systematically learn

from using Dialog can be translated to all searching• & you get an insight into

problems with searching

12Tefko Saracevic

Newest large database:• Scopus started in 2004 by

Elsevier – a HUGE publisher• Very different from Dialog

– integrates over 17,000 journals & other materials (has no separate databases, but could be searched by broad fields, type of materials, etc.)

• Indexes all (or takes existing indexing for some)

• Elsevier also has– Scirus – free science search engine

– ScienceDirect – journals’ full texts, available on RUL, Indexes and databases

• Provides intuitive searching – geared toward end user– also provides various other

capabilities e.g. citation tracking

• Most subscribers libraries & companies– but through them access to

end users• RUL was subscribed, but dropped• in class you have free access

• Major competition to Web of Science (RUL has it)

Tefko Saracevic 13

14

Types of information databases• Many types are available:

– Bibliographic– Numeric– Full text– Directory– Image

• still, film, video

– Sound• spoken word, music

– Multimedia– Real time

• Some that are in Dialog are also available elsewhere or on their own

• Some vendors have exclusive right to some databases

• Many you find in RUL

Tefko Saracevic

Other vendors/aggregatorssample from RUL 275 databases; links require RUL login

Various disciplines or areas AgricolaAmerica: History and LifeBusiness and Industry DatabaseDissertations and ThesesEducation Index/Abstracts/Full TextFactivaHispanic-American Periodicals IndexLexisNexis AcademicMedline Oceanic AbstractsPollution AbstractsWomen's Studies International

Particularly related to LISACM Digital LibraryASIST Digital LibraryComputing ReviewsIEEE XploreLibrary, Information Science &

Technology Abstracts (LISTA)Library and Information Science Abstracts

(LISA)Library Literature and Information

ScienceProfessional Development CollectionResources for College Libraries (RCL)

Tefko Saracevic 15

16

a BIG, BIG problem• In Dialog & some other vendors you can search a

number of databases at the same time– so called federated searching

• in Dialog using file 411, Dialindex (get it: 411 … )

• In Scopus you search the whole thing – if you wish• BUT in RUL & elsewhere there is no federated

searching– you have to search each database separately– at RUL through Searchlight you can search 8 databases

• others you have to search one at the time– someday there will be federated searching, but at present do

not hold your breath

Tefko Saracevic

17

as would imagine …

Tefko Saracevic

Describing & organizing nature of content

3. Structure of records in databases

© Tefko Saracevic 18

19

Now unto structures – getting under the hood

• Databases structure own records – documents, objects …– why? to describe various parts of content for computers to

recognize – these are fields, as mentioned• you can recognize that a section of a document is a title,

but a computer has to be told that a title is a title– so that it can (among others) search for terms in a title when you

request so

• Fields in records are labeled as to content or function– most fields in databases indicate the same content

• e.g. title, author, index terms, abstract, text parts, source, …

– but various databases do it in their own way• in whatever convoluted way they do it, it is not that hard to decipher

Tefko Saracevic

20

Labeling schemes

• Many structure schemes were developed that prescribed what to label & what to call the label – meta languages– by providers, vendors, organizations, authorities– in different subjects, domains– for different types of objects

• Meta tags are used on the web – to describe & index– semantic web is in development, to further enable description

of and searching for meaning• MARC is a form of meta language• To use these schemes for effective searching you have no

choice but to get familiar

Tefko Saracevic

21

Transparency of structures

• In some databases description of structure is readily available – even though it may look forbidding, complicated

• good example: Bluesheets in Dialog• search fields in Scopus

• In others, structure is there but has to be discovered by surmising– even in and particularly in

• But clever, appropriate use of structure in searching is key to effective searching

…

Tefko Saracevic

© Tefko Saracevic 22

Describes the content of the file

Describes the content of the file

Example: Dialog file 438 Bluesheet

23

file 438 record & fields- each field is searchablee.g. /TI=title; AU=author; SO=source; JN=Journal; …

Tefko Saracevic

Indicates field &

abbreviation

Indicates field &

abbreviation

Organization of indexes in Dialogit has two kinds of indexes

• Dialog has a Basic Index – searched by default

• Entering a command s (or select) digital and libraries – finds all documents that have

the term digital and the term libraries anywhere in the document

– s digital and libraries/TI finds documents that have these terms in the title

• Dialog has also Additional Indexes– these are for Authors (AU),

Sources (SO) , Publication Years (PY) … & many more

– searched as s (or select) digital and libraries and AU=Saracevic

Tefko Saracevic 24

All other databases have similar arrangements as to indexes, but are not that clearly visible as in Dialog, but are searchable in selections

25

file 438: searching in Basic Index - it is searched by default

Examples how to search in basic index by words & other fields

S means select command; W means with – terms next to each other in that order

Examples how to search in basic index by words & other fields

S means select command; W means with – terms next to each other in that order

Tefko Saracevic

26

file 438: fields in Additional Indexes

Additional index is searched by indicating the field to be searched –

examples how to search them

Additional index is searched by indicating the field to be searched –

examples how to search them

If you want to search the latest update

only, add to search UD=9999

Neat trick:

Tefko Saracevic

27

file 438: fields inLimit

Searches can be limited to cover documents with given attributes –

examples how to limit searches

Searches can be limited to cover documents with given attributes –

examples how to limit searches

Tefko Saracevic

S2 means set 2 as retrieved previously

S2 means set 2 as retrieved previously

28

file 438: additionaluses of structure

Results can be sorted or ranked by given fields –

examples how to sort or rank results

Results can be sorted or ranked by given fields –

examples how to sort or rank results

Tefko Saracevic

29

file 438: options in displaying of results

Results can be displayed & then printed in a number of ways – examples of available formats

Results can be displayed & then printed in a number of ways – examples of available formats

But watch out! In real life some formats are free other cost $$$$!

But watch out! In real life some formats are free other cost $$$$!

Tefko Saracevic

30

Economics – tail that wags the whole dog

• In class Dialog searching is free– & you can use it for class exercises & learning

• In real life Dialog (as every other vendor) has an elaborate economic structure – different files have different price tags for use– time of use is calculated in DialUnits

• a Byzantine structure of charges - it is beyond understanding– in different files different formats have different price

attached• full formats in some files are really hefty!

Tefko Saracevic

31

Where to find all about structure?

• In Dialog in BlueSheets (file 415)– consult often! and again! and again! and again!– files have similarities and differences in structure –

BlueSheets show that• For other vendors:

– some have similar description as BlueSheets– some indicate fields that can be searched

• it shows structure – in some revelation comes from checking what is

available in advanced searching or in tips for searching– in some structure has to be surmised

Tefko Saracevic

Tefko Saracevic32

Structure in search engines & databases

• Mostly not readily apparent– but all have capabilities to be used in searching

• Again: revelation comes from checking what is available in Advanced Search, Search Features, Search Tips, Help, & the like

• Most users do NOT take advantage of using available structures in searching– professional searchers do

• part of their tool kit & competencies

Example: structure from Advanced Search

© Tefko Saracevic 33

Records are

structured & can be searched by these fields &

topics

More fields available

Example of structure from Scopus (features)

© Tefko Saracevic 34

Records are structured &

can be searched by additional 10

or so pull

down fields

More fields available

Subjects areas choices

More choices

Example of structure fromLibrary Literature & Information Science Full Text (at RUL)

© Tefko Saracevic 35

Records are structured &

can be searched by additional 20

or so pull

down fields

More fields available

36

Similarities & differences • All vendors & search

engines have basic & advanced Boolean-type search capabilities– but how it is done & bells and

whistles differ– once you master concepts

you can then do an AHA! when you encounter a variation & then translate

• Many vendors & search engines have advanced search features– many above & beyond Boolean

Tefko Saracevic

• All vendors rank output results– but how it is done differs– by default most (Dialog, Scopus

& most others) use LIFO – Last in First Out

– but also allow for a number of other ways. e.g. by source

• Search engines use ranking by relevance, clustering, PageRank & other criteria• proprietary – they do not tell you

about it - not easy to discern

Similarities & differences …

• Most users– do not know or care about

structure– do not search beyond default

capabilities– do not look beyond one or

two pages of results– miss many potentially

relevant results– do not know what is under

the hood– can’t do advanced – more

sophisticated – searching

• Professional searchers– know that structure is very

much connected to searching– learn about & use available

structures– understand defaults & use

advanced capabilities as necessary

– know “tricks” for not missing stuff or not getting to much or to much junk

– explore in order to learn what is under the hood

Tefko Saracevic 37

As used in searching4. Indexes

Tefko Saracevic 38

We all know what an index isbut to refresh

An index is a list of words and associated pointers to where those words can be found in a document

Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval- example of automatic indexing

• Many kinds of indexes e.g.– back of the book index,

alphabetical , subject, classified, faceted, …

• As to creation:– manual, automatic, – today trend is toward

automatic creation of indexes• by means of computer

algorithms to select words or phrases to identify content

Tefko Saracevic 39

Here we deal with index structures & in next lecture we deal with indexing vocabularies

Inverted indexes

• All databases have some kind of inverted index– searching is done through

them

Inverted index: An index containing terms, as

keys, mapped to references to the documents they appear in. The index is sorted by its keys. “Inverted” means that the documents are found by matching on terms, rather than the other way around.

From Apple Glossary

• End of the book index is an inverted index

• First inverted indexes were made in 12th century– concordance of the Bible

• a concordance is an alphabetical list of the principal words used in a book or body of work, with their position indicated as immediate context

• In contrast, sequential index is a full index for each document – one by one

Tefko Saracevic 40

Making & searching of inverted indexes

• Inverted indexes can be made from regular sequential indexes for every document

• But also from regular texts– abstracts and full texts

• Automatic indexes are made from texts – now easily– following given algorithms– omitting “stop” words

• Dialog has 9: AN, FOR, THE, AND, FROM, TO, BY, OF, WITH

• Searching is then done on the inverted index– so it is useful to understand the

structure• for a document every word is

identified as where it appears in text

• search looks for appearance e.g. if “digital” is in position 8 in sentence

10 & “library” is in position 9 in sentence 10 , then in a search is for “digital library” the algorithm looks what positions of terms “digital” & “library” is next to each other in same sentence, finds them & retrieves them as hit

Tefko Saracevic 41

Inverted indexes

Useful to know how they function to understand search & retrieval. Steps:

1. Each document is indexed– every word in a document is taken as index term with

exception of stop words– position in text is noted

2. Indexes for all documents are merged• index terms are arranged alphabetically in the bowel

of the system• under each index term are document numbers in which it

appears & position in text for that document

Tefko Saracevic 42

Example on creating an inverted index (from Walker & Janes, 1999)

Four documents: 101, 102, 103, 104Fields: TI=Bold; AB=text; DE=descriptor

Tefko Saracevic 43

© Tefko Saracevic 44

Terms for each document – after stop words eliminated

© Tefko Saracevic 45

Inverted index – a few last terms after letter R are missing, no space on page

TermsTerms

Doc no.Doc no. FieldField PositionPosition

46

In conclusion

Searching is more art than science, but an art that needs a lot of knowledge what is behind it

Tefko Saracevic

47Tefko Saracevic

You can do it!

Tefko Saracevic 48

Try!

just start moving your mouse on the empty page serving as canvas