This paper establishes the theoretical framework of b-bit minwise hashing. The original minwise hashing method has become a standard technique for estimating set similarity (e.g., resemblance) with applications in information retrieval, data management, computational advertising, etc.By only storing b bits of each hashed value (e.g., b=1 or 2), we gain substantial advantages in terms of storage space. We prove the basic theoretical results and provide an unbiased estimator of the resemblance for any b. We demonstrate that, even in the least favorable scenario, using b=1 may reduce the storage space at least by a factor of 21.3 (or 10.7) compared to b=64 (or b=32), if one is interested in resemblance >0.5.

/pdf/b-bit-minwise-hashing-49k1ozly6u.pdf

b-Bit minwise hashing

Minwise hashing is a standard technique in the context of search for efficiently computing set similarities. The recent development of b-bit minwise hashing provides a substantial improvement by storing only the lowest b bits of each hashed value. In this paper, we demonstrate that b-bit minwise hashing can be naturally integrated with linear learning algorithms such as linear SVM and logistic regression, to solve large-scale and high-dimensional statistical learning tasks, especially when the data do not fit in memory. We compare b-bit minwise hashing with the Count-Min (CM) and Vowpal Wabbit (VW) algorithms, which have essentially the same variances as random projections. Our theoretical and empirical comparisons illustrate that b-bit minwise hashing is significantly more accurate (at the same storage cost) than VW (and random projections) for binary data.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/NIPS2011_Hashing.pdf

Hashing Algorithms for Large-Scale Learning

This article introduces and discusses the concept of academic search engine optimization (ASEO). Based on three recently conducted studies, guidelines are provided on how to optimize scholarly literature for academic search engines in general, and for Google Scholar in particular. In addition, we briefly discuss the risk of researchers' illegitimately ‘over-optimizing’ their articles.

/pdf/academic-search-engine-optimization-aseo-optimizing-29x44t7tux.pdf

Academic Search Engine Optimization ( ASEO ): Optimizing Scholarly Literature for Google Scholar & Co.

b-Bit Minwise Hashing in

The web is one of the most popular vectors to spread malware. Attackers lure victims to visit compromised web pages or entice them to click on malicious links. These victims are redirected to sites that exploit their browsers or trick them into installing malicious software using social engineering.In this paper, we tackle the problem of detecting malicious web pages from a novel angle. Instead of looking at particular features of a (malicious) web page, we analyze how a large and diverse set of web browsers reach these pages. That is, we use the browsers of a collection of web users to record their interactions with websites, as well as the redirections they go through to reach their final destinations. We then aggregate the different redirection chains that lead to a specific web page and analyze the characteristics of the resulting redirection graph. As we will show, these characteristics can be used to detect malicious pages.We argue that our approach is less prone to evasion than previous systems, allows us to also detect scam pages that rely on social engineering rather than only those that exploit browser vulnerabilities, and can be implemented efficiently. We developed a system, called SpiderWeb, which implements our proposed approach. We show that this system works well in detecting web pages that deliver malware.

/pdf/shady-paths-leveraging-surfing-crowds-to-detect-malicious-4cbfy659le.pdf

Shady paths: leveraging surfing crowds to detect malicious web pages

Automatically generated content is ubiquitous in the web: dynamic sites built using the three-tier paradigm are good examples (e.g., commercial sites, blogs and other sites edited using web authoring software), as well as less legitimate spamdexing attempts (e.g., link farms, faked directories).Those pages built using the same generating method (template or script) share a common “look and feel” that is not easily detected by common text classification methods, but is more related to stylometry.In this work we study and compare several HTML style similarity measures based on both textual and extra-textual features in HTML source code. We also propose a flexible algorithm to cluster a large collection of documents according to these measures. Since the proposed algorithm is based on locality sensitive hashing (LSH), we first review this technique.We then describe how to use the HTML style similarity clusters to pinpoint dubious pages and enhance the quality of spam classifiers. We present an evaluation of our algorithm on the WEBSPAM-UK2006 dataset.

Tracking Web spam with HTML style similarities

In a fast changing world, with more and more demanding clients, quality needs increase. To satisfy it, a lot of organizations producing software chose the Software Process Improvement approach. At France Telecom R&D, we develop this approach using the ISO/IEC TR 15504(SPICE) process model. The improvement programme we implemented was combined with risk management. To optimise the risk management in the programme, we chose to analyse statistically the risks of the programme. In this communication, we present how we set up the programme, the risk management implementation and the risk data analysis methods and results.

Analysis of Risks in a Software Improvement Programme

In our article of Airweb 2006 workshop [4], we combined the approaches of html noise preprocessing (removing content), minhash fingerprinting and similarity clustering to spot dubious sets of web pages. For this challenge the idea is the same but we study more preprocessing and clustering strategies that we use to smooth the predictions of a classifier. We test two learning methodologies and sumbit two predictions.

/pdf/spamchallenge-2007-france-telecom-r-d-submissions-1e83x3wjqd.pdf

Emmanuel Chauveau

Papers

Tracking Web spam with HTML style similarities

Analysis of Risks in a Software Improvement Programme

SpamChallenge 2007 : France Telecom R&D Submissions

Analysis of risks in a Software Improvement programme