Efficient Search in Large Textual Collections with Redundancy

Jiangong Zhang and Torsten Suel

Review by

Newton Alex

993940942

Problem

Searching over collections of data that include many different crawls and versions of each page

– E.g. Searching the Internet archive, email archives etc.

Not feasible to provide full text search due to high cost of processing a query

– E.g. Current indexing and query processing techniques when applied to say 10 successive crawls of the same URL will result in index sizes and query processing costs roughly 10 times that of single crawl

Proposed Solution

A new and general framework that results in significant savings in the size of the inverted index and the performance of query processing for webpage collections with redundancies.

Features– Content-dependent partitioning techniques, in particular

Winnowing.– Non redundant indexing. Two policies with respect to

indexing local sharing global sharing

– Modification of Document-at-a-time query processing algorithm to take advantage of the fragment based indexes

Critique

The paper does not described the data structures used or the hardware setup in detail.

The framework supports deleting old unused fragments. Why is a delete required when we are interested in versioned systems?

Since no duplicate fragments are maintained, deleting a fragment might result in removing fragments corresponding to other pages in the archive.

Relation to Course

This paper is similar to the Google News paper. However, this paper doesn’t describe the data structures or the environment setup in detail

Related to the concepts that were used in the Search engine project like inverted indexes, query matching etc.

Proposes methods for creating efficient indexes for redundant data.