KEY BENEFIT: Written by a leader in the field of information retrieval, this text provides the background and tools needed to evaluate, compare and modify search engines. KEY TOPICS: Coverage of the underlying IR and mathematical models reinforce key concepts. Numerous programming exercises make extensive use of Galago, a Java-based open source search engine. MARKET: A valuable tool for search engine and information retrieval professionals.

Search Engines: Information Retrieval in Practice

This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work.

https://www.microsoft.com/en-us/research/wp-content/uploads/2010/01/1500000017.pdf

Web Crawling

Hybrid peer-to-peer architectures use special nodes to provide directory services for regions of the network ("regional directory services"). Hybrid peer-to-peer architectures are a potentially powerful model for developing large-scale networks of complex digital libraries, but peer-to-peer networks have so far tended to use very simple methods of resource selection and document retrieval. In this paper, we study the application of content-based resource selection and document retrieval to hybrid peer-to-peer networks. The directory nodes that provide regional directory services construct and use the content models of neighboring nodes to determine how to route query messages through the network. The leaf nodes that provide information use content-based retrieval to decide which documents to retrieve for queries. The experimental results demonstrate that using content-based retrieval in hybrid peer-to-peer networks is both more accurate and more efficient for some digital library environments than more common alternatives such as Gnutella 0.6.

/pdf/content-based-retrieval-in-hybrid-peer-to-peer-networks-406h9ttrv4.pdf

Content-based retrieval in hybrid peer-to-peer networks

Federated search (federated information retrieval or distributed information retrieval) is a technique for searching multiple text collections simultaneously. Queries are submitted to a subset of collections that are most likely to return relevant answers. The results returned by selected collections are integrated and merged into a single list. Federated search is preferred over centralized search alternatives in many environments. For example, commercial search engines such as Google cannot easily index uncrawlable hidden web collections while federated search systems can search the contents of hidden web collections without crawling. In enterprise environments, where each organization maintains an independent search engine, federated search techniques can provide parallel search over multiple collections.

There are three major challenges in federated search. For each query, a subset of collections that are most likely to return relevant documents are selected. This creates the collection selection problem. To be able to select suitable collections, federated search systems need to acquire some knowledge about the contents of each collection, creating the collection representation problem. The results returned from the selected collections are merged before the final presentation to the user. This final step is the result merging problem.

The goal of this work, is to provide a comprehensive summary of the previous research on the federated search challenges described above.

https://www.microsoft.com/en-us/research/wp-content/uploads/2011/01/now.pdf

Federated Search

Peer-to-Peer (P2P) networks enable users to directly share digital content (such as audio, video, and text files) as well as real-time data (such as telephony traffic) with other users without depending on a central server. Although originally popularized by unlicensed online music services such as Napster, P2P networking has recently emerged as a viable multimillion dollar business model for the distribution of academic and clinical information, telecommunications, and social networking. Written at an accessible level for any reader familiar with fundamental Internet protocols, Peer-to-Peer Networking and Applications explains the conceptual operations and architecture underlying basic P2P systems using well-known commercial systems as models. The book also delineates the latest research directions, thereby providing not only a sophisticated understanding of current systems, but also the means to improve upon these systems with innovations that will better performance, security, and flexibility. Peer-to-Peer Networking and Applications is thus both a valuable starting point and an important reference to those practioners employed by any of the 200 companies with approximately $400 million invested in this new and lucrative technology.



Uses well-known commercial P2P systems as models, thus demonstrating real-world applicability.

Discusses how current research trends in wireless networking, high-def content, DRM, etc. will intersect with P2P, allowing readers to account for future developments in their designs.

Provides online access to the Overlay Weaver P2P emulator, an open-source tool that supports a number of peer-to-peer applications with which readers can practice.

P2P Networking and Applications

Peer-to-peer architectures are a potentially powerful model for developing large-scale networks of text-based digital libraries, but peer-to-peer networks have so far provided very limited support for text-based federated search of digital libraries using relevance-based ranking. This paper addresses the problems of resource representation, resource ranking and selection, and result merging for federated search of text-based digital libraries in hierarchical peer-to-peer networks. Existing approaches to text-based federated search are adapted and new methods are developed for resource representation and resource selection according to the unique characteristics of hierarchical peer-to-peer networks. Experimental results demonstrate that the proposed approaches offer a better combination of accuracy and efficiency than more common alternatives for federated search in peer-to-peer networks.

Federated Search of Text-Based Digital Libraries in Hierarchical Peer-to-Peer Networks.

/pdf/federated-search-of-text-based-digital-libraries-in-36lxdzs03f.pdf

Federated search of text-based digital libraries in hierarchical peer-to-peer networks

User modeling for information retrieval has mostly been studied to improve the effectiveness of information access in centralized repositories. In this paper we explore user modeling in the context of full-text federated search in peer-to-peer networks. Our approach models a user's persistent, long-term interests based on past queries, and uses the model to improve search efficiency for future queries that represent interests similar to past queries. Our approach also enables queries representing a user's transient, ad-hoc interests to be automatically recognized so that search for these queries can rely on a relatively large search radius to avoid sacrificing effectiveness for efficiency. Experimental results demonstrate that our approach can significantly improve the efficiency of full-text federated search without degrading its accuracy. Furthermore, the proposed approach does not require a large amount of training data, and is robust to a range of parameter values.

/pdf/user-modeling-for-full-text-federated-search-in-peer-to-peer-26jhrx42rk.pdf

Jie Lu

Papers

Content-based retrieval in hybrid peer-to-peer networks

Federated Search of Text-Based Digital Libraries in Hierarchical Peer-to-Peer Networks.

Federated search of text-based digital libraries in hierarchical peer-to-peer networks

User modeling for full-text federated search in peer-to-peer networks

Merging retrieval results in hierarchical peer-to-peer networks