We show the existence of a Locality-Sensitive Hashing (LSH) family for the angular distance that yields an approximate Near Neighbor Search algorithm with the asymptotically optimal running time exponent. Unlike earlier algorithms with this property (e.g., Spherical LSH [Andoni, Indyk, Nguyen, Razenshteyn 2014], [Andoni, Razenshteyn 2015]), our algorithm is also practical, improving upon the well-studied hyperplane LSH [Charikar, 2002] in practice. We also introduce a multiprobe version of this algorithm, and conduct experimental evaluation on real and synthetic data sets. 
We complement the above positive results with a fine-grained lower bound for the quality of any LSH family for angular distance. Our lower bound implies that the above LSH family exhibits a trade-off between evaluation time and quality that is close to optimal for a natural class of LSH functions.

Practical and Optimal LSH for Angular Distance

We show the existence of a Locality-Sensitive Hashing (LSH) family for the angular distance that yields an approximate Near Neighbor Search algorithm with the asymptotically optimal running time exponent. Unlike earlier algorithms with this property (e.g., Spherical LSH [1, 2]), our algorithm is also practical, improving upon the well-studied hyperplane LSH [3] in practice. We also introduce a multiprobe version of this algorithm and conduct an experimental evaluation on real and synthetic data sets.

We complement the above positive results with a fine-grained lower bound for the quality of any LSH family for angular distance. Our lower bound implies that the above LSH family exhibits a trade-off between evaluation time and quality that is close to optimal for a natural class of LSH functions.

/pdf/practical-and-optimal-lsh-for-angular-distance-shuxhm1sos.pdf

Practical and optimal LSH for angular distance

Nearest neighbor searches in high-dimensional space have many important applications in domains such as data mining, and multimedia databases. The problem is challenging due to the phenomenon called "curse of dimensionality". An alternative solution is to consider algorithms that returns a c-approximate nearest neighbor (c-ANN) with guaranteed probabilities. Locality Sensitive Hashing (LSH) is among the most widely adopted method, and it achieves high efficiency both in theory and practice. However, it is known to require an extremely high amount of space for indexing, hence limiting its scalability.In this paper, we propose several surprisingly simple methods to answer c-ANN queries with theoretical guarantees requiring only a single tiny index. Our methods are highly flexible and support a variety of functionalities, such as finding the exact nearest neighbor with any given probability. In the experiment, our methods demonstrate superior performance against the state-of-the-art LSH-based methods, and scale up well to 1 billion high-dimensional points on a single commodity PC.

/pdf/srs-solving-c-approximate-nearest-neighbor-queries-in-high-25fk473pza.pdf

SRS: solving c-approximate nearest neighbor queries in high dimensional euclidean space with a tiny index

Machine learning, graph analytics and sparse linear algebra-based applications are dominated by irregular memory accesses resulting from following edges in a graph or non-zero elements in a sparse matrix. These accesses have little temporal or spatial locality, and thus incur long memory stalls and large bandwidth requirements. A traditional streaming or striding prefetcher cannot capture these irregular access patterns. A majority of these irregular accesses come from indirect patterns of the form A[B[j]]. We propose an efficient hardware indirect memory prefetcher (IMP) to capture this access pattern and hide latency. We also propose a partial cacheline accessing mechanism for these prefetches to reduce the network and DRAM bandwidth pressure from the lack of spatial locality. Evaluated on 7 applications, IMP shows 56% speedup on average (up to 2.3×) compared to a baseline 64 core system with streaming prefetchers. This is within 23% of an idealized system. With partial cacheline accessing, we see another 9.4% speedup on average (up to 46.6%).

/pdf/imp-indirect-memory-prefetcher-1y7z38a0pm.pdf

IMP: indirect memory prefetcher

Many modern methods for prediction leverage nearest neighborsearch to find past training examples most similar toa test example, an idea that dates back in text to at leastthe 11th century and has stood the test of time. This monographaims to explain the success of these methods, both intheory, for which we cover foundational nonasymptotic statisticalguarantees on nearest-neighbor-based regression andclassification, and in practice, for which we gather prominentmethods for approximate nearest neighbor search thathave been essential to scaling prediction systems reliant onnearest neighbor analysis to handle massive datasets. Furthermore,we discuss connections to learning distances foruse with nearest neighbor methods, including how randomdecision trees and ensemble methods learn nearest neighborstructure, as well as recent developments in crowdsourcingand graphons.In terms of theory, our focus is on nonasymptotic statisticalguarantees, which we state in the form of how many trainingdata and what algorithm parameters ensure that a nearestneighbor prediction method achieves a user-specified errortolerance. We begin with the most general of such resultsfor nearest neighbor and related kernel regression and classificationin general metric spaces. In such settings in whichwe assume very little structure, what enables successful predictionis smoothness in the function being estimated forregression, and a low probability of landing near the decisionboundary for classification. In practice, these conditionscould be difficult to verify empirically for a real dataset. Wethen cover recent theoretical guarantees on nearest neighborprediction in the three case studies of time series forecasting,recommending products to people over time, and delineatinghuman organs in medical images by looking at imagepatches. In these case studies, clustering structure, whichis easier to verify in data and more readily interpretable bypractitioners, enables successful prediction.

/pdf/explaining-the-success-of-nearest-neighbor-methods-in-3stth1n2ml.pdf

Explaining the Success of Nearest Neighbor Methods in Prediction

Finding nearest neighbors has become an important operation on databases, with applications to text search, multimedia indexing, and many other areas. One popular algorithm for similarity search, especially for high dimensional data (where spatial indexes like kd-trees do not perform well) is Locality Sensitive Hashing (LSH), an approximation algorithm for finding similar objects.In this paper, we describe a new variant of LSH, called Parallel LSH (PLSH) designed to be extremely efficient, capable of scaling out on multiple nodes and multiple cores, and which supports high-throughput streaming of new data. Our approach employs several novel ideas, including: cache-conscious hash table layout, using a 2-level merge algorithm for hash table construction; an efficient algorithm for duplicate elimination during hash-table querying; an insert-optimized hash table structure and efficient data expiration algorithm for streaming data; and a performance model that accurately estimates performance of the algorithm and can be used to optimize parameter settings. We show that on a workload where we perform similarity search on a dataset of > 1 Billion tweets, with hundreds of millions of new tweets per day, we can achieve query times of 1-2.5 ms. We show that this is an order of magnitude faster than existing indexing schemes, such as inverted indexes. To the best of our knowledge, this is the fastest implementation of LSH, with table construction times up to 3.7× faster and query times that are 8.3× faster than a basic implementation.

/pdf/streaming-similarity-search-over-one-billion-tweets-using-4btv1hplix.pdf

Streaming similarity search over one billion tweets using parallel locality-sensitive hashing

Streaming Similarity Search over One Billion Tweets Using Parallel Locality-Sensitive Hashing

Digital twin networks (DTNs) are real-time replicas of physical networks. They are emerging as a powerful technology for design, diagnosis, simulation, what-if-analysis, and artificial intelligence (AI)/machine learning (ML) driven real-time optimization and control of the sixth-generation (6G) wireless networks. Despite the great potential of what digital twins can offer for 6G, realizing the desired capabilities of 6G DTNs requires tackling many design aspects including data, models, and interfaces. In this article, we provide an overview of 6G DTNs by presenting prominent use cases and their service requirements, describing a reference architecture, and discussing fundamental design aspects. We also present a real-world example to illustrate how DTNs can be built upon and operated in a real-time reference development platform - Omniverse.

Todd Mostak

Papers

Streaming similarity search over one billion tweets using parallel locality-sensitive hashing

Streaming Similarity Search over One Billion Tweets Using Parallel Locality-Sensitive Hashing

6G Digital Twin Networks: From Theory to Practice