Proceedings ArticleDOI
On the resemblance and containment of documents
TLDR
The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that could be done independently for each document.Abstract:
Given two documents A and B we define two mathematical notions: their resemblance r(A, B) and their containment c(A, B) that seem to capture well the informal notions of "roughly the same" and "roughly contained." The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that can be done independently for each document. Furthermore, the resemblance can be evaluated using a fixed size sample for each document. This paper discusses the mathematical properties of these measures and the efficient implementation of the sampling process using Rabin (1981) fingerprints.read more
Citations
More filters
Dissertation
Sketching and streaming high-dimensional vectors
TL;DR: This thesis describes several new contributions in the area of sketching and streaming algorithms, including the first space-optimal streaming algorithm for the distinct elements problem, a general reduction from empirical entropy estimation in turnstile streams to moment estimation, and a proof of the Johnson-Lindenstrauss lemma.
Journal ArticleDOI
An Approach of Semantic Similarity Measure between Documents Based on Big Data
TL;DR: This paper investigates MapReduce programming model as a specific framework for managing distributed processing in a large of amount documents and proposes the approach of semantic similarity measures using WordNet as an external network semantic resource.
Proceedings ArticleDOI
Is this a good title
TL;DR: It is shown that titles decay slowly but are far more stable than the pages' content, and a predictive model for the quality of any given web page title in terms of its discovery performance is provided.
Proceedings ArticleDOI
On permutations with limited independence
TL;DR: The novel notion of "krankwise independence" is introduced, the family size of k-rankwise independent permutations and related permutation families are investigated, and the known lower bound is introduced.
Journal ArticleDOI
Web service QoS prediction: when collaborative filtering meets data fluctuating in big-range
TL;DR: A simple yet effective similarity model which considers the influence of big-range among users’ QoS data for accurately characterizing the similarity between users, and which significantly outperforms state-of-the-art CF methods to predict Web service QoS ratings accurately.
References
More filters
Book
The Probabilistic Method
TL;DR: A particular set of problems - all dealing with “good” colorings of an underlying set of points relative to a given family of sets - is explored.
Journal ArticleDOI
Syntactic clustering of the Web
TL;DR: An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.
Journal ArticleDOI
Min-Wise Independent Permutations
TL;DR: This research was motivated by the fact that such a family of permutations is essential to the algorithm used in practice by the AltaVista web index software to detect and filter near-duplicate documents.
Proceedings Article
Finding similar files in a large file system
TL;DR: Application of sif can be found in file management, information collecting, program reuse, file synchronization, data compression, and maybe even plagiarism detection.
Proceedings ArticleDOI
Copy detection mechanisms for digital documents
TL;DR: This paper proposes a system for registering documents and then detecting copies, either complete copies or partial copies, and describes algorithms for such detection, and metrics required for evaluating detection mechanisms (covering accuracy, efficiency, and security).