Proceedings ArticleDOI
On the resemblance and containment of documents
TLDR
The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that could be done independently for each document.Abstract:
Given two documents A and B we define two mathematical notions: their resemblance r(A, B) and their containment c(A, B) that seem to capture well the informal notions of "roughly the same" and "roughly contained." The basic idea is to reduce these issues to set intersection problems that can be easily evaluated by a process of random sampling that can be done independently for each document. Furthermore, the resemblance can be evaluated using a fixed size sample for each document. This paper discusses the mathematical properties of these measures and the efficient implementation of the sampling process using Rabin (1981) fingerprints.read more
Citations
More filters
Book ChapterDOI
Decentralized and Adaptive K-Means Clustering for Non-IID Data Using HyperLogLog Counters
Amira Soliman,Sarunas Girdzijauskas,Mohamed-Rafik Bouguelia,Sepideh Pashami,Sławomir Nowaczyk +4 more
TL;DR: This paper presents a decentralized and adaptive k-means algorithm that clusters data from multiple sources organized in peer-to-peer networks and addresses the challenge of decentralized clustering with skewed non-IID data and asynchronous computations by integrating HyperLogLog counters with k- means algorithm.
Routine Learning: from Reactive to Proactive Environments
TL;DR: The routine learning paradigm developed here can utilize already recognized contexts despite their meaning in the real world, and offers a step along the long road towards functional and calm intelligent environments.
A Sketch-based Sampling Algorithm on Sparse Data
TL;DR: This work proposes a sketch-based sampling algorithm, which effectively exploits the data sparsity and combines the advantages of both conventional random sampling and more modern randomized algorithms such as local sensitive hashing (LSH).
Journal ArticleDOI
A Simulation Analysis of Redundancy and Reliability in Primary Storage Deduplication
TL;DR: Analysis of storage system reliability using public file system snapshots from two research groups shows that deduplication consistently reduces the damage of sector errors due to intra-file redundancy elimination, but potentially increases the damages of whole-disk failures if the highly referenced chunks are not carefully placed on disk.
Proceedings ArticleDOI
Exploring Deep Learning in Semantic Question Matching
TL;DR: In a nutshell, this research predicts the semantic coincidence between the question pairs extracting highly dominant features and hence, determine the probability of question being duplicate.
References
More filters
Book
The Probabilistic Method
TL;DR: A particular set of problems - all dealing with “good” colorings of an underlying set of points relative to a given family of sets - is explored.
Journal ArticleDOI
Syntactic clustering of the Web
TL;DR: An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.
Journal ArticleDOI
Min-Wise Independent Permutations
TL;DR: This research was motivated by the fact that such a family of permutations is essential to the algorithm used in practice by the AltaVista web index software to detect and filter near-duplicate documents.
Proceedings Article
Finding similar files in a large file system
TL;DR: Application of sif can be found in file management, information collecting, program reuse, file synchronization, data compression, and maybe even plagiarism detection.
Proceedings ArticleDOI
Copy detection mechanisms for digital documents
TL;DR: This paper proposes a system for registering documents and then detecting copies, either complete copies or partial copies, and describes algorithms for such detection, and metrics required for evaluating detection mechanisms (covering accuracy, efficiency, and security).