Proceedings ArticleDOI
On the resemblance and containment of documents
TLDR
The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that could be done independently for each document.Abstract:
Given two documents A and B we define two mathematical notions: their resemblance r(A, B) and their containment c(A, B) that seem to capture well the informal notions of "roughly the same" and "roughly contained." The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that can be done independently for each document. Furthermore, the resemblance can be evaluated using a fixed size sample for each document. This paper discusses the mathematical properties of these measures and the efficient implementation of the sampling process using Rabin (1981) fingerprints.read more
Citations
More filters
Old and new challenges in automatic plagiarism detection
Paul Clough,Regent Court +1 more
TL;DR: The nature of the plagiarism problem is explored, and the approaches used so far for its detection are summarized, and a number of methods used to measure text reuse are discussed.
Journal ArticleDOI
Theory and applications of b-bit minwise hashing
Ping Li,Arnd Christian König +1 more
TL;DR: B-bit minwise hashing is described, which can provide an order of magnitude improvements in storage requirements and computational overhead over the original scheme in practice in practice and is compared to other important alternative techniques proposed for estimating set similarities.
Proceedings ArticleDOI
Learning Hybrid Representations to Retrieve Semantically Equivalent Questions
TL;DR: Retrieving similar questions in online QA (2) BOW-CNN is more robust than the pure CNN for long texts.
Journal ArticleDOI
Discovering high quality answers in community question answering archives using a hierarchy of classifiers
TL;DR: This work proposes a hybrid hierarchy-of-classifiers framework to model the QA pairs and proposes a number of novel features that are effective in distinguishing the quality of answers.
Proceedings ArticleDOI
Design tradeoffs for data deduplication performance in backup workloads
TL;DR: A general-purpose framework to evaluate various deduplication solutions in a large N-dimensional parameter space to find some reasonable solutions that have sustained backup performance and perform a suitable tradeoff between dedUplication ratio, memory footprints, and restore performance.
References
More filters
Book
The Probabilistic Method
TL;DR: A particular set of problems - all dealing with “good” colorings of an underlying set of points relative to a given family of sets - is explored.
Journal ArticleDOI
Syntactic clustering of the Web
TL;DR: An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.
Journal ArticleDOI
Min-Wise Independent Permutations
TL;DR: This research was motivated by the fact that such a family of permutations is essential to the algorithm used in practice by the AltaVista web index software to detect and filter near-duplicate documents.
Proceedings Article
Finding similar files in a large file system
TL;DR: Application of sif can be found in file management, information collecting, program reuse, file synchronization, data compression, and maybe even plagiarism detection.
Proceedings ArticleDOI
Copy detection mechanisms for digital documents
TL;DR: This paper proposes a system for registering documents and then detecting copies, either complete copies or partial copies, and describes algorithms for such detection, and metrics required for evaluating detection mechanisms (covering accuracy, efficiency, and security).