Stop Searching and Start Stop Searching and Start FINDING: Strategies for FINDING: Strategies for Effective Web ResearchEffective Web Research

a presentation bya presentation by

Ken Wiseman Ken Wiseman

& IMSA& IMSA

ken@wisemantech.comken@wisemantech.com

Our goals today ...Our goals today ...

Discover the biggest mistakes made by most Discover the biggest mistakes made by most Internet users: Internet users: Typing search terms in the wrong box.Typing search terms in the wrong box. Using the wrong tool at the wrong time.Using the wrong tool at the wrong time.

Talk about the differences between Talk about the differences between directories and search engines (and when to directories and search engines (and when to use each.)use each.)

Learn some advanced Google searching Learn some advanced Google searching techniques.techniques.

DO ALL OF THIS IN ENGLISH!DO ALL OF THIS IN ENGLISH!

To Start To Start

Put the right request in the right Put the right request in the right placeplace

Put Web addresses in the address box (this is the URL stuff that begins with http://)

Put search terms (the stuff you are looking for) in the search box

The Biggest MistakeThe Biggest Mistake

Thinking that search tools are Thinking that search tools are card catalogs of the webcard catalogs of the web

9 Billion pages reside on the Web 9 Billion pages reside on the Web (4/02)(4/02)

• No search tool indexes all of the web.• The largest -Google-indexes less than 30% of the total (3 billion).• Each engine indexes a different set of web pages.

Search Engine Size - Search Engine Size -

0

500

1000

1500

2000

2500

3000

3500

SE 3033 2106 1689 1453 1147 1018 1015 733 275

Google

AlltheWe

AltaVista

Wisenut

Hotbot

MSN Teoma

NL Gigablast

Search Engine Showdown - 12/02

The Second Biggest The Second Biggest MistakeMistake

Using the wrong tool at the wrong Using the wrong tool at the wrong timetime

Three questionsThree questions

Where would you find the telephone number or Where would you find the telephone number or address of the Woodfield theatre?address of the Woodfield theatre? A telephone bookA telephone book

Where you would find the definition of the word Where you would find the definition of the word “pestilence?”“pestilence?” A dictionary (or in your school yearbook)A dictionary (or in your school yearbook)

Where would you find the name of the war that Where would you find the name of the war that the Treaty of Westphalia ended?the Treaty of Westphalia ended? An encyclopediaAn encyclopedia

What would happen if What would happen if you tried to look up the you tried to look up the definition of the word definition of the word

“pestilence” in the “pestilence” in the telephone book?telephone book?

YAHOO ISN’T A YAHOO ISN’T A SEARCH SEARCH ENGINE!ENGINE!

... it is a directory.... it is a directory.

(but this maybe changing)(but this maybe changing)

DirectoriesDirectories

Usually human-Usually human-compiled guides to the compiled guides to the web, where sites are web, where sites are organized by categoryorganized by category

Major directories:Major directories: MSNMSN YahooYahoo Netscape ODPNetscape ODP

DirectoriesDirectories How Internet Directories WorkVisit Web sitesEvaluatesDirectory EmployeeAdds & CatalogsDirectory ServerDirectory’s searchable index

Directory’sSearchable

index

Directory’sBrowsableCategories

SearchesUser BrowsesUser seeking informationreceives infoInternet/Web

Pages

What directories are good for ...What directories are good for ...

““What is the Web page address for some What is the Web page address for some company, organization, or entity?” (or “who company, organization, or entity?” (or “who makes product X?”)makes product X?”)

““Where can I find a list of Web pages that Where can I find a list of Web pages that focus on a particular, ‘universal’ topic?”focus on a particular, ‘universal’ topic?”

In other words, directories are In other words, directories are GREATGREAT for for “telephone book” searches.“telephone book” searches.

What directories AREN’T good for ...What directories AREN’T good for ...

Directories are Directories are horriblehorrible for for “encyclopedia” or “dictionary” searches.“encyclopedia” or “dictionary” searches.

The only exception is if the topic is so The only exception is if the topic is so universal that the directories have no universal that the directories have no choice but to link to a page or two that choice but to link to a page or two that discuss that topic (and even then the discuss that topic (and even then the selection will be slim.)selection will be slim.)

Search Engines have three parts:Search Engines have three parts:

1.1. A spider (also called a A spider (also called a "crawler" or a "bot") that "crawler" or a "bot") that goes to every page or goes to every page or representative pages on representative pages on every Web site that wants every Web site that wants to be searchable and to be searchable and reads it, using hypertext reads it, using hypertext links on each page to links on each page to discover and read a site's discover and read a site's other pages.other pages.

Search Engines have three parts:Search Engines have three parts:

2.2. A program that creates a A program that creates a huge index (sometimes huge index (sometimes called a "catalog") from the called a "catalog") from the pages that have been read.pages that have been read.

A program that receives your search request, compares it to the entries in the index, and returns results to you.

Directories vs Search EnginesDirectories vs Search Engines

Directories are human-compiled and have a Directories are human-compiled and have a small number of pages in their databases small number of pages in their databases (usually in the low millions)(usually in the low millions)

Search engines are machine-compiled and Search engines are machine-compiled and have a have a HUGEHUGE number of pages in their number of pages in their databases (usually in the hundreds of databases (usually in the hundreds of millions or even the billions)millions or even the billions)

The Second Biggest The Second Biggest Mistake -- RestatedMistake -- Restated

Using a directory as if it was a Using a directory as if it was a search engine ... and then not search engine ... and then not

understanding why you can’t find understanding why you can’t find anything!anything!

Top search sites – January 2002Top search sites – January 2002

1.1. MSNMSN

2.2. YahooYahoo

3.3. GoogleGoogle

4.4. AOLAOL

5.5. Ask JeevesAsk Jeeves

6.6. LookSmartLookSmart

7.7. InfospaceInfospace

8.8. OvertureOverture

9.9. NetscapeNetscape

10.10. AltaVistaAltaVista

-- Courtesy Jupiter Media Metrix

Which ones are directories?Which ones are directories?

1.1. MSNMSN

2.2. YahooYahoo

3.3. GoogleGoogle

4.4. AOLAOL

5.5. Ask JeevesAsk Jeeves

6.6. LookSmartLookSmart

7.7. InfospaceInfospace

8.8. OvertureOverture

9.9. Netscape ODPNetscape ODP

10.10. AltaVistaAltaVista

Why Use a Search Engine?Why Use a Search Engine?6 Billion+

3

2

1

0

Millions

Directory Name

Open Directory LookSmart MSN Yahoo NBCi/Snap Britannica google

Web Pages Indexed by Directories

Secondary resultsSecondary results

Most directories use a Most directories use a search engine as a search engine as a backup (Yahoo and backup (Yahoo and Netscape use Google, Netscape use Google, almost everyone else almost everyone else uses Inktomi)uses Inktomi)

Why add the extra Why add the extra step?step?

How the sites stack upHow the sites stack up

Most directories (like Most directories (like MSN and AOL) link to MSN and AOL) link to 2 or 3 million pages.2 or 3 million pages.

Most search engines Most search engines (like AlltheWeb and (like AlltheWeb and Google) link to Google) link to BillionsBillions of pages. of pages.

-- Courtesy searchenginewatch.com

Why do people Why do people predominantly use predominantly use

directories when search directories when search engines have more engines have more

stuff?stuff? Because no one Because no one ever takes the time to ever takes the time to teach us how to use a teach us how to use a

search engine!search engine!

The Third Biggest The Third Biggest MistakeMistake

Not knowing how to use Not knowing how to use directories or search engines to directories or search engines to

actually actually FINDFIND stuff stuff

Search engine rule #1Search engine rule #1

Be specific ... because if Be specific ... because if you aren’t specific, you’ll you aren’t specific, you’ll end up with a bunch of end up with a bunch of

garbage!garbage!

Preparing to SearchPreparing to Search Formulation of the research questionFormulation of the research question Identification of important concepts within the Identification of important concepts within the

question question Identification of search terms to describe those Identification of search terms to describe those

concepts concepts Consideration of synonyms and variations of Consideration of synonyms and variations of

those terms those terms Take a look at Take a look at Vivisimo Vivisimo

clustered results for helpclustered results for help Preparation of the search logicPreparation of the search logic

Search engine rule #2Search engine rule #2

Use quotes to search for Use quotes to search for phrases.phrases.

““ken wiseman”ken wiseman”

Use quotes for phrasesUse quotes for phrases To search for phrases, just put your phrase in To search for phrases, just put your phrase in

quotes.quotes. For example, For example, disney fantasyland disney fantasyland “pirates of the caribbean”“pirates of the caribbean” This would show you all the pages in Google’s This would show you all the pages in Google’s

index that contain the word index that contain the word disneydisney AND the word AND the word fantasylandfantasyland AND the phrase AND the phrase pirates of pirates of the caribbeanthe caribbean

By the way, while this search is By the way, while this search is technicallytechnically OK, my OK, my choice of keywords contains a (deliberate) factual choice of keywords contains a (deliberate) factual mistake. mistake.

Can you spot it?Can you spot it?

Arr, She Blows!Arr, She Blows!

Pirates of the Caribbean isn’t Pirates of the Caribbean isn’t in Fantasyland, it’s in in Fantasyland, it’s in Adventureland in Orlando Adventureland in Orlando and New Orleans Square in and New Orleans Square in Anaheim.Anaheim.

So searching for So searching for disney disney AND AND fantasyland fantasyland AND AND “pirates of the “pirates of the caribbean” caribbean” probably isn’t probably isn’t a good idea. a good idea.

Search engine rule #3Search engine rule #3

Use the + sign to Use the + sign to require.require.

Apple+computerApple+computer

Search engine math:Search engine math:+ & And+ & And

Limits your search

Apple & Computer

Only returns pages with both of these terms on them

Search engine rule #4Search engine rule #4

Use the - sign to Use the - sign to exclude.exclude.

apple -computerapple -computer

Search Engine Math:Search Engine Math:- & not- & not

Limits your search

Women not History

Only returns pages that contain one but not the other term on them

Boolean ORBoolean OR

Sometimes the default Sometimes the default ANDAND gets in the way. gets in the way. That’s where That’s where OROR comes in. comes in.

The Boolean operator The Boolean operator OROR is is alwaysalways in caps and in caps and goes between keywords.goes between keywords.

For example, an improvement over our earlier For example, an improvement over our earlier search would be search would be disney fantasyland OR disney fantasyland OR “pirates of the caribbean”“pirates of the caribbean” This would show you all the pages in Google’s index This would show you all the pages in Google’s index

that contain the word that contain the word disneydisney AND the word AND the word fantasylandfantasyland OR the phrase OR the phrase pirates of the pirates of the caribbeancaribbean (without the quotes) (without the quotes)

Three Ways to Three Ways to OROR at Google at Google

Just type Just type OROR between keywords between keywordsdisney fantasyland OR “pirates of the disney fantasyland OR “pirates of the caribbean”caribbean”

Put your Put your OROR statement in parentheses statement in parenthesesdisney (fantasyland OR “pirates of the disney (fantasyland OR “pirates of the caribbean”)caribbean”)

Use the Use the || (“pipe”) character in place of the word (“pipe”) character in place of the word ORORdisney (fantasyland | “pirates of the disney (fantasyland | “pirates of the caribbean”)caribbean”)

All three methods yield the All three methods yield the exactexact same results. same results.

Search engine math:Search engine math:OROR

Women or History

Returns every page with either of these terms on them

Broadens your search

OR, She Blows!OR, She Blows!

Just remember, Google’s Just remember, Google’s Boolean default is Boolean default is ANDAND

Sometimes the default Sometimes the default ANDAND gets in the way. gets in the way. That’s where That’s where OROR comes comes in.in.

How Insensitive!How Insensitive!

Google is Google is notnot case sensitive. case sensitive. So, the following searches all yield exactly So, the following searches all yield exactly

the same results: the same results: disney fantasyland piratesdisney fantasyland pirates

Disney Fantasyland PiratesDisney Fantasyland Pirates

DISNEY FANTASYLAND PIRATESDISNEY FANTASYLAND PIRATES

DiSnEy FaNtAsYlAnD pIrAtEsDiSnEy FaNtAsYlAnD pIrAtEs

Search engine rule #5Search engine rule #5

Combine symbols as often as Combine symbols as often as possible possible

(see rule #1).(see rule #1).

+”Martha Washington” –george +revolution+”Martha Washington” –george +revolution

The five rulesThe five rules

1.1. Be specific ... because if you aren’t specific, Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!you’ll end up with a bunch of garbage!

2.2. Use quotes to search for phrases.Use quotes to search for phrases.

3.3. Use the + sign to require.Use the + sign to require.

4.4. Use the - sign to exclude.Use the - sign to exclude.

5.5. Combine symbols as often as possible Combine symbols as often as possible (see rule #1).(see rule #1).

6.6. Don’t forget ORDon’t forget OR

Did You Know…Did You Know…

that large chunks of the Web are invisible to that large chunks of the Web are invisible to most search engines.most search engines.

That no one has a good handle on the That no one has a good handle on the magnitude of the invisible web*magnitude of the invisible web*

That much of the invisible web is of great That much of the invisible web is of great value to educators & studentsvalue to educators & students

So What?So What?

Would you intentionally exclude large Would you intentionally exclude large chunks of the Library of Congress’ 12 million chunks of the Library of Congress’ 12 million documents from your searches?documents from your searches?

How about the US Census Bureau?How about the US Census Bureau? How about health and medical databases?How about health and medical databases? Many newspapers?Many newspapers?

What Today’s Search Tools What Today’s Search Tools Can and Cannot findCan and Cannot find

Not search tool specific Not search tool specific Search tools were created to handle flat Search tools were created to handle flat

HTML pages. HTML pages. When confronted with a search box the When confronted with a search box the

search tool is stopped unless it has specific search tool is stopped unless it has specific instructions on how to handle that input box.instructions on how to handle that input box.

Dynamically created web pages have Dynamically created web pages have unusual URLsunusual URLs

What Today’s Search Tools What Today’s Search Tools Can and Cannot findCan and Cannot find

The LII page for Automobile (http://lii.org/search/file/automobiles) is in Google;

The LII page for Motorcycles (http://lii.org/search?title=Motorcycles; query=Motorcycles; searchtype=subject) is not.

Do you see why NOT??

Simple ExamplesSimple Examples

Ken Wiseman - 602 Hits

None contain my contact info

But…

Typical Search PagesTypical Search Pages

Finding Specialized databases in Google…Finding Specialized databases in Google…

Adding “searchable database” reduced the hits from 14,000 to 140 and added additional resources not found in the standard search!

Index of Specialized DatabasesIndex of Specialized Databases

BeaucoupsBeaucoups Listing of specialized search toolsListing of specialized search tools Most of the information is invisible to general Most of the information is invisible to general

search enginessearch engines Over 2500 search tools listed.Over 2500 search tools listed.

Other Invisible Web Recovery Other Invisible Web Recovery sitessites

Don’t Forget Don’t Forget Portals…Portals…

http://www.skewlsites.com/

http://www.homeworkspot.com/

http://www.awesomelibrary.org/

http://www.about.com/