Proceedings ArticleDOI
On the resemblance and containment of documents
TLDR
The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that could be done independently for each document.Abstract:
Given two documents A and B we define two mathematical notions: their resemblance r(A, B) and their containment c(A, B) that seem to capture well the informal notions of "roughly the same" and "roughly contained." The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that can be done independently for each document. Furthermore, the resemblance can be evaluated using a fixed size sample for each document. This paper discusses the mathematical properties of these measures and the efficient implementation of the sampling process using Rabin (1981) fingerprints.read more
Citations
More filters
Proceedings ArticleDOI
CanDID: Can-Do Decentralized Identity with Legacy Compatibility, Sybil-Resistance, and Accountability
Deepak Maram,Harjasleen Malvai,Fan Zhang,Nerla Jean-Louis,Alexander Frolov,Tyler Kell,Tyrone Lobban,Christine Moy,Ari Juels,Andrew Miller +9 more
TL;DR: CanDID as mentioned in this paper is a platform for practical, user-friendly realization of decentralized identity, the idea of empowering end users with management of their own credentials by issuing credentials in a userfriendly way that draws securely and privately on data from existing, unmodified web service providers.
Journal ArticleDOI
Interactive navigation of open data linkages
TL;DR: The Toronto Open Data Search system offers users a highly interactive experience making unrelated (and unlinked) dynamic collections of datasets appear as a richly connected cloud of data that can be navigated and combined easily in real time.
Proceedings ArticleDOI
A Minimal Variance Estimator for the Cardinality of Big Data Set Intersection
TL;DR: In this paper, a new algorithm based on the Maximum Likelihood (ML) method is proposed to estimate the intersection size of two big data streams, and the algorithm achieves the optimal variance.
Journal ArticleDOI
A distributed near-optimal LSH-based framework for privacy-preserving record linkage
TL;DR: A framework which relies on the Map/Reduce paradigm in order to distribute computations among underutilized commodity hardware resources uniformly, without imposing an extra overhead on the existing infrastructure is presented.
Patent
Representative document selection for sets of duplicate documents in a web crawler system
TL;DR: In this article, a set of documents sharing the same content as the newly crawled document is identified and a single representative document for the new set is identified in accordance with the set of predefined conditions.
References
More filters
Book
The Probabilistic Method
TL;DR: A particular set of problems - all dealing with “good” colorings of an underlying set of points relative to a given family of sets - is explored.
Journal ArticleDOI
Syntactic clustering of the Web
TL;DR: An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.
Journal ArticleDOI
Min-Wise Independent Permutations
TL;DR: This research was motivated by the fact that such a family of permutations is essential to the algorithm used in practice by the AltaVista web index software to detect and filter near-duplicate documents.
Proceedings Article
Finding similar files in a large file system
TL;DR: Application of sif can be found in file management, information collecting, program reuse, file synchronization, data compression, and maybe even plagiarism detection.
Proceedings ArticleDOI
Copy detection mechanisms for digital documents
TL;DR: This paper proposes a system for registering documents and then detecting copies, either complete copies or partial copies, and describes algorithms for such detection, and metrics required for evaluating detection mechanisms (covering accuracy, efficiency, and security).