Recent advancements in deep neural networks for graph-structured data have led to state-of-the-art performance on recommender system benchmarks. However, making these methods practical and scalable to web-scale recommendation tasks with billions of items and hundreds of millions of users remains an unsolved challenge. Here we describe a large-scale deep recommendation engine that we developed and deployed at Pinterest. We develop a data-efficient Graph Convolutional Network (GCN) algorithm, which combines efficient random walks and graph convolutions to generate embeddings of nodes (i.e., items) that incorporate both graph structure as well as node feature information. Compared to prior GCN approaches, we develop a novel method based on highly efficient random walks to structure the convolutions and design a novel training strategy that relies on harder-and-harder training examples to improve robustness and convergence of the model. We also develop an efficient MapReduce model inference algorithm to generate embeddings using a trained model. Overall, we can train on and embed graphs that are four orders of magnitude larger than typical GCN implementations. We show how GCN embeddings can be used to make high-quality recommendations in various settings at Pinterest, which has a massive underlying graph with 3 billion nodes representing pins and boards, and 17 billion edges. According to offline metrics, user studies, as well as A/B tests, our approach generates higher-quality recommendations than comparable deep learning based systems. To our knowledge, this is by far the largest application of deep graph embeddings to date and paves the way for a new generation of web-scale recommender systems based on graph convolutional architectures.

Graph Convolutional Neural Networks for Web-Scale Recommender Systems

The present invention is directed to an integrated implementation framework and resulting medium for knowledge retrieval, management, delivery and presentation. The system includes a first server component that is responsible for adding and maintaining domain-specific semantic information and a second server component that hosts semantic and other knowledge for use by the first server component that work together to provide context and time-sensitive semantic information retrieval services to clients operating a presentation platform via a communication medium. Within the system, all objects or events in a given hierarchy are active Agents semantically related to each other and representing queries (comprised of underlying action code) that return data objects for presentation to the client according to a predetermined and customizable theme. This system provides various means for the client to customize and “blend” Agents and the underlying related queries to optimize the presentation of the resulting information.

System and method for knowledge retrieval, management, delivery and presentation

Often, in the real world, entities have two or more representations in databases. Duplicate records do not share a common key and/or they contain errors that make duplicate matching a difficult task. Errors are introduced as the result of transcription errors, incomplete information, lack of standard formats, or any combination of these factors. In this paper, we present a thorough analysis of the literature on duplicate record detection. We cover similarity metrics that are commonly used to detect similar field entries, and we present an extensive set of duplicate detection algorithms that can detect approximately duplicate records in a database. We also cover multiple techniques for improving the efficiency and scalability of approximate duplicate detection algorithms. We conclude with coverage of existing tools and with a brief discussion of the big open problems in the area

/pdf/duplicate-record-detection-a-survey-4crkywvv1d.pdf

Duplicate Record Detection: A Survey

This is a survey on graph visualization and navigation techniques, as used in information visualization. Graphs appear in numerous applications such as Web browsing, state-transition diagrams, and data structures. The ability to visualize and to navigate in these potentially large, abstract graphs is often a crucial part of an application. Information visualization has specific requirements, which means that this survey approaches the results of traditional graph drawing from a different perspective.

/pdf/graph-visualization-and-navigation-in-information-24tisrjfna.pdf

Graph visualization and navigation in information visualization: A survey

/pdf/duplicate-record-detection-a-survey-2rm1fz3gbd.pdf

We present an efficient query evaluation method based on a two level approach: at the first level, our method iterates in parallel over query term postings and identifies candidate documents using an approximate evaluation taking into account only partial information on term occurrences and no query independent factors; at the second level, promising candidates are fully evaluated and their exact scores are computed. The efficiency of the evaluation process can be improved significantly using dynamic pruning techniques with very little cost in effectiveness. The amount of pruning can be controlled by the user as a function of time allocated for query evaluation. Experimentally, using the TREC Web Track data, we have determined that our algorithm significantly reduces the total number of full evaluations by more than 90%, almost without any loss in precision or recall. At the heart of our approach there is an efficient implementation of a new Boolean construct called WAND or Weak AND that might be of independent interest.

Efficient query evaluation using a two-level retrieval process

We introduce static index pruning methods that significantly reduce the index size in information retrieval systems.We investigate uniform and term-based methods that each remove selected entries from the index and yet have only a minor effect on retrieval results. In uniform pruning, there is a fixed cutoff threshold, and all index entries whose contribution to relevance scores is bounded above by a given threshold are removed from the index. In term-based pruning, the cutoff threshold is determined for each term, and thus may vary from term to term. We give experimental evidence that for each level of compression, term-based pruning outperforms uniform pruning, under various measures of precision. We present theoretical and experimental evidence that under our term-based pruning scheme, it is possible to prune the index greatly and still get retrieval results that are almost as good as those based on the full index.

https://researcher.watson.ibm.com/researcher/files/us-fagin/sigir01.pdf

Static index pruning for information retrieval systems

A method for searching a corpus of documents, such as the World Wide Web, includes defining a knowledge domain and identifying a set of reference documents in the corpus pertinent to the domain. Upon inputting a query, the corpus is searched using the set of reference documents to find one or more of the documents in the corpus that contain information in the domain relevant to the query. The set of reference documents is updated with the found documents that are most relevant to the domain. The updated set is used in searching the corpus for information in the domain relevant to subsequent queries.

Information search using knowledge agents

A query is received and parsed to generate a set of query terms. Statistical information is identified regarding each of the query terms and different permutations of the query terms. Additionally, lexical affinities associated with the permutations of query terms are identified. Next, the query is classified into a query category and a set of ranking parameters and routing information (associated with the query category) are identified. The query is then issued to a search engine by applying the identified ranking parameters and routing information, whereupon the search engine executes the query and forwards search results that can be accessed by an application using an API (e.g., the results can be viewed via a browser).

Automatic query routing and rank configuration for search queries in an information retrieval system

A method includes describing the thread configurations of a volume of well-ordered electronic message transmissions (EMT) and utilizing the thread configuration data to conduct selective searches of the EMT volume. An apparatus includes a thread processor and a query manager. The thread processor analyzes the EMT threads and records the thread configuration data. The query manager utilizes the thread configuration data to conduct selective searches of the EMT volume.

Michael Herscovici

Papers

Efficient query evaluation using a two-level retrieval process

Static index pruning for information retrieval systems

Information search using knowledge agents

Automatic query routing and rank configuration for search queries in an information retrieval system

Indexing and searching of electronic message transmission thread sets