This Article Full Text (PDF) All Versions of this Article: 0165551508091308v1 34/6/861    most recent References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Wang, Y. D. Articles by Forgionne, G. Search for Related Content Social Bookmarking                         What's this?

Testing a decision-theoretic approach to the evaluation of information retrieval systems

Department of Applied Information Technology, George Mason University, USA, ywangm{at}gmu.edu

Guisseppi Forgionne

Information Systems Department, University of Maryland Baltimore County, USA

With information overload a real problem, especially on the Internet, there has been much interest in developing effective and efficient information retrieval (IR) systems. The various information retrieval approaches will require accurate evaluation to justify the requisite substantial development and implementation investment. Recently, a comprehensive and integrated evaluation model has been proposed and illustrated. By analyzing the evaluation measures using the analytic hierarchy process (AHP), the model transforms IR evaluation into a multi-criteria decision making (MCDM) problem, which assesses both the IR outcome and the interactive IR process. This paper extends that research by refining the evaluation model and by testing the research question through mathematical testing and simulation. The tests confirm the need to include both process and outcome criteria in any IR evaluation and prove the superiority of the proposed decision-theoretic approach over the traditional evaluation methodologies that focus on the IR outcome alone.

Key Words: information retrieval system • evaluation • information search process • decision making • multi-criteria model • analytic hierarchy process

This version was published on December 1, 2008

Journal of Information Science, Vol. 34, No. 6, 861-876 (2008)
DOI: 10.1177/0165551508091308


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?