This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Web of Science (1) Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Xi, W. Articles by Lim, E.-P. Search for Related Content Social Bookmarking                         What's this?

Incorporating window-based passage-level evidence in document retrieval

Nanyang Technological University, Singapore

Richard Xu-Rong

Nanyang Technological University, Singapore

Christopher S.G. Khoo

Nanyang Technological University, Singapore, assgkhoo{at}ntu.edu.sg

Ee-Peng Lim

Nanyang Technological University, Singapore

This study investigated whether document retrieval can be improved if documents are divided into smaller sub-documents or passages and the retrieval score for these passages are incorporated in the final retrieval score for the whole document. The documents were segmented by sliding a window of a certain size across the document and extracting the words displayed each time the window stopped. A retrieval score was calculated for each of the passages extracted and the highest score obtained by a passage of that size was taken as the document’s passage-level score for that window size. A range of window sizes was tried.

The experimental results indicated that using a fixed window size of 50 words gave better results than other window sizes for the TREC-5 and TREC-6 test collections. This window size yielded a significant retrieval improvement of 24% compared to using the whole-document retrieval score (using the traditional tf^*idf weighting scheme with cosine normalisation). However, combining this window score and the whole-document retrieval score did not yield a retrieval improvement.

Using a variable window size (ranging from 50 to 400 words) yielded a retrieval improvement of about 5% over using a fixed window size of 50. Different window sizes were found to work best for different queries. If the best window size to use for each query could be predicted accurately, a maximum retrieval improvement of 42% could be obtained.

Subsequent work suggests that the usefulness of passage-level evidence in document retrieval depends on the weighting scheme and type of normalisation used in the retrieval method.

Journal of Information Science, Vol. 27, No. 2, 73-80 (2001)
DOI: 10.1177/016555150102700202


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?