This Article Full Text (OnlineFirst PDF) All Versions of this Article: 0165551507087237v1 34/5/726    most recent Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Google Scholar Articles by Huntington, P. Articles by Jamali, H. R. Social Bookmarking                   What's this?

Web robot detection in the scholarly information environment

CIBER:¹ School of Library, Archive and Information Studies, University College London

^* To whom correspondence should be addressed.

	Abstract

An increasing number of robots harvest information on the world wide web for a wide variety of purposes. Protocols developed at the inception of the web laid out voluntary procedures in order to identify robot behaviour, and exclude it if necessary. Few robots now follow this protocol and it is now increasingly difficult to filter for this activity in reports of on-site activity. This paper seeks to demonstrate the issues involved in identifying robots and assessing their impact on usage in regard to a project which sought to establish the relative usage patterns of open access and non-open access articles in the Oxford University Press published journal Glycobiology, which offers in a single issue articles in both forms. A number of methods for identifying robots are compared and together these methods found that 40% of the raw logs of this journal could be attributed to robots.

Key Words: electronic journals; robot detection; web crawlers; web log analysis

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?