cs523 information retrieval course introduction yÜcel saygin sabanci university

CS523 INFORMATION RETRIEVAL

COURSE INTRODUCTION•

• YÜCEL SAYGIN

• SABANCI UNIVERSITY

Contact Info

[email protected]

http://people.sabanciuniv.edu/~ysaygin

Tel : 9576No Specific office hours. You can drop by anytime you like. Email or call me to make sure I am at the office.

Course Info

Reference Book: Introduction to Information Retrieval,

Authors: Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze

Publisher:Cambridge University Press. 2008.

Course Info

Grading: Homework : 10% Project : 40% Paper presentation : 20% Term Paper : 20% Attendance during paper

presentations: 10%

Topics that will be covered

Document Retrieval TechniquesInformation Retrieval on the WebData Mining for Information Retrieval

Aim of the course

Knowledge: To introduce information retrieval

techniques

Skills: paper reading and presentation research and/or project work

A Rough Schedule

October, November: Lectures on various information

retrieval techniques

Remaining weeks: Paper and research project presentations

What I will do

Give the basics on information retrievalProject supervisionGive directions and advise on the projects Coordination of the presentations

What I expect you to do

Understand the basic concepts of Information Retrieval

Choose a specific area and two related papers on the same topic for presentation in class

Attendance is required for paper presentations and you will loose 2% of your overall grade for each presentation you missed.

Write a term paper on the two papers presented.

Do a project and a final report describing what you learned or achieved in the scope of the project.

Sources

TREC Conference http://trec.nist.gov/SIGIR Conference http://www.sigir.org/WWW Conference http://www2004.org/ACM TOIS JournalSIGMOD, VLDB, ICDE Conferences (database perspective)SIGKDD, ICDM Conferences (data mining perspective)

Tools

SMART IR (Cornell Univ.) http://www.cs.cornell.edu/Info/Projects/NLP/

Glimpse from Univ. Arizona http://webglimpse.net/

GoogleAltavistaYahoo

Information Retrieval

Refers to the retrieval of any type of information such as

Structured data (e.g. relational database) Text (We will focus on this) Video Image, sound DNA

Document Retrieval

User Query

Static Document Collection

Ranked Result

•Document Collection is previously indexed•User query is ad hoc•Results are ranked wrt their similarity to the user query

Document Routing

User profiles are set in advance

Incoming documents are directed to relevant usersUseful for redirecting corporate emails to relevant departments (sales, marketing, support etc)

Performance Metrics for IRPrecisionRecallNot practical to have good precision and recall

Whole Document Space

Relevant Documents

Retrieved Documents

Relevant and Retrieved Documents

First Reading for Tomorrow

The Anatomy of a Large-Scale Hypertextual Web Search Engine (WWW Conference 1998)

paper by Sergey Brin and Lawrence Page www-db.stanford.edu/~backrub/google.html

Web Information Retrieval

Two possible ways: Use the web structure starting from a

location like yahoo where things are categorized

Use search engines


Challenges Scale:

Hundreds of millions of queries per day Web grows, continuous crawling is needed Obstacles due to OS, and disk seek time

Google handles large data sets by indexing and compressionSearch quality is important

Completeness of the index is important But ranking is also of utmost importance due to the

size of the Web


Ranking (of google) The idea is to give importance to pages that

have a lot of back links Similar to the notion of citations in

academia A link graph of the web was formed and

maintained (518 million links in 1998 for the prototype)

Web Mining

(focused) Crawling and IndexingTopic DirectoriesClustering and ClassificationHyperlink AnalysisPersonalization (profiles, preferences)