In studying actual Web searching by the public at large, we analyzed over one million Web queries by users of the Excite search engine. We found that most people use few search terms, few modified queries, view few Web pages, and rarely use advanced search features. A small number of search terms are used with high frequency, and a great many terms are unique; the language of Web queries is distinctive. Queries about recreation and entertainment rank highest. Findings are compared to data from two other large studies of Web queries. This study provides an insight into the public practices and choices in Web searching.

/pdf/searching-the-web-the-public-and-their-queries-1ypq21msuf.pdf

Searching the Web: the public and their queries

Previous work on understanding user web search behavior has focused on how people search and what they are searching for, but not why they are searching. In this paper, we describe a framework for understanding the underlying goals of user searches, and our experience in using the framework to manually classify queries from a web search engine. Our analysis suggests that so-called navigational" searches are less prevalent than generally believed while a previously unexplored "resource-seeking" goal may account for a large fraction of web searches. We also illustrate how this knowledge of user search goals might be used to improve future web search engines.

/pdf/understanding-user-goals-in-web-search-sc3y1kcrr3.pdf

Understanding user goals in web search

This paper serves as a companion or extension to the "Inside PageRank" paper by Bianchini et al. [Bianchini et al. 03]. It is a comprehensive survey of all issues associated with PageRank, covering the basic PageRank model, available and recommended solution methods, storage issues, existence, uniqueness, and convergence properties, possible alterations to the basic model, suggested alternatives to the traditional solution methods, sensitivity and conditioning, and finally the updating problem. We introduce a few new results, provide an extensive reference list, and speculate about exciting areas of future research.

/pdf/deeper-inside-pagerank-2snurfc515.pdf

Deeper Inside PageRank

The Web and especially major Web search engines are essential tools in the quest to locate online information for many people. This paper reports results from research that examines characteristics and changes in Web searching from nine studies of five Web search engines based in the US and Europe. We compare interactions occurring between users and Web search engines from the perspectives of session length, query length, query complexity, and content viewed among the Web search engines. The results of our research shows (1) users are viewing fewer result pages, (2) searchers on US-based Web search engines use more query operators than searchers on European-based search engines, (3) there are statistically significant differences in the use of Boolean operators and result pages viewed, and (4) one cannot necessary apply results from studies of one particular Web search engine to another Web search engine. The wide spread use of Web search engines, employment of simple queries, and decreased viewing of result pages may have resulted from algorithmic enhancements by Web search engine companies. We discuss the implications of the findings for the development of Web search engines and design of online content.

/pdf/how-are-we-searching-the-world-wide-web-a-comparison-of-nine-1vg6xqg009.pdf

How are we searching the world wide web?: a comparison of nine search engine transaction logs

The Network Society is now more than ever the essential guide to the past, consequences and future of digital communication. Fully revised, this Third Edition covers crucial new issues and updates. This book remains an accessible, comprehensive, must-read introduction to how new media function in contemporary society.

The network society

In this paper we present an analysis of an AltaVista Search Engine query log consisting of approximately 1 billion entries for search requests over a period of six weeks. This represents almost 285 million user sessions, each an attempt to fill a single information need. We present an analysis of individual queries, query duplication, and query sessions. We also present results of a correlation analysis of the log entries, studying the interaction of terms within queries. Our data supports the conjecture that web users differ significantly from the user assumed in the standard information retrieval literature. Specifically, we show that web users type in short queries, mostly look at the first 10 results only, and seldom modify the query. This suggests that traditional information retrieval techniques may not work well for answering web search requests. The correlation analysis showed that the most highly correlated items are constituents of phrases. This result indicates it may be useful for search engines to consider search terms as parts of phrases even if the user did not explicitly specify them as such.

/pdf/analysis-of-a-very-large-web-search-engine-query-log-4rysjnhagp.pdf

Analysis of a very large web search engine query log

This article presents a high-level discussion of some problems in information retrieval that are unique to web search engines. The goal is to raise awareness and stimulate research in these areas.

/pdf/challenges-in-web-search-engines-3vamssdwpc.pdf

Challenges in web search engines

This article presents a high-level discussion of some problems that are unique to web search engines. The goal is to raise awareness and stimulate research in these areas.

/pdf/challenges-in-web-search-engines-ow1oof3s28.pdf

Techniques for crawling hyperlinked documents are provided. Hyperlinked documents to be crawled are grouped by host and the host to be crawled next is selected according to a stall time of the host. The stall time can indicate the earliest time that the host should be crawled and the stall times can be a predetermined amount of time, vary by host and be adjusted according to actual retrieval times from the host.

Distributed crawling of hyperlinked documents

A system modifies documents to aid users in determining which of the entries in the documents to choose. The system identifies a document that includes one or more entries. The system determines a score for each of the entries and modifies the identified document, or entries in the identified document, based on the determined scores. The system then provides the modified document to a user.

Craig Silverstein

Papers

Analysis of a very large web search engine query log

Challenges in web search engines

Challenges in web search engines

Distributed crawling of hyperlinked documents

Scoring links in a document