There is, I think, something ethereal about i —the square root of minus one. I remember first hearing about it at school. It seemed an odd beast at that time—an intruder hovering on the edge of reality.

Usually familiarity dulls this sense of the bizarre, but in the case of i it was the reverse: over the years the sense of its surreal nature intensified. It seemed that it was impossible to write mathematics that described the real world in …

I and i

Data Mining - Concepts and Techniques.

Power-law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man-made phenomena. Unfortunately, the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution—the part of the distribution representing large but rare events—and by the difficulty of identifying the range over which power-law behavior holds. Commonly used methods for analyzing power-law data, such as least-squares fitting, can produce substantially inaccurate estimates of parameters for power-law distributions, and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all. Here we present a principled statistical framework for discerning and quantifying power-law behavior in empirical data. Our approach combines maximum-likelihood fitting methods with goodness-of-fit tests based on the Kolmogorov-Smirnov (KS) statistic and likelihood ratios. We evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches. We also apply the proposed methods to twenty-four real-world data sets from a range of different disciplines, each of which has been conjectured to follow a power-law distribution. In some cases we find these conjectures to be consistent with the data, while in others the power law is ruled out.

/pdf/power-law-distributions-in-empirical-data-4dybihkltl.pdf

Power-Law Distributions in Empirical Data

Fun and exciting textbook on the mathematics underpinning the most dynamic areas of modern science and engineering.

/pdf/information-theory-inference-and-learning-algorithms-gbv6i2zu22.pdf

Information Theory, Inference and Learning Algorithms

We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

Cuckoo hashing holds great potential as a high-performance hashing scheme for real applications. Up to this point, the greatest drawback of cuckoo hashing appears to be that there is a polynomially small but practically significant probability that a failure occurs during the insertion of an item, requiring an expensive rehashing of all items in the table. In this paper, we show that this failure probability can be dramatically reduced by the addition of a very small constant-sized stash. We demonstrate both analytically and through simulations that stashes of size equivalent to only three or four items yield tremendous improvements, enhancing cuckoo hashing's practical viability in both hardware and software. Our analysis naturally extends previous analyses of multiple cuckoo hashing variants, and the approach may prove useful in further related schemes.

More Robust Hashing: Cuckoo Hashing with a Stash

Graption: A graph-based P2P traffic classification framework for the internet backbone

An encoded loss resilient message, includes a first number of first data items, a second number of second data items, and a third number of third data items. Respective portions of the first data items correspond to different numbers of associated second data items in a first distribution. Respective portions of the second data items correspond to different numbers of associated first data items in a second distribution. Respective portions of the second data items correspond to different numbers of associated third data items in a third distribution which is proportional to the first distribution with the different numbers of associated third data items equaling the different numbers of associated second data items multiplied by the first number divided by the second number. Respective portions of the third data items correspond to other different numbers of associated second data items in a fourth distribution which is proportional to the second distribution with the other different numbers of associated second data items equaling the different numbers of associated first data items multiplied by the second number divided by the third number.

Loss resilient code with double heavy tailed series of redundant layers

Daily deal sites have become the latest Internet sensation, providing discounted offers to customers for restaurants, ticketed events, services, and other items. We begin by undertaking a study of the economics of daily deals on the web, based on a dataset we compiled by monitoring Groupon and LivingSocial sales in 20 large cities over several months. We use this dataset to characterize deal purchases; glean insights about operational strategies of these firms; and evaluate customers' sensitivity to factors such as price, deal scheduling, and limited inventory. We then marry our daily deals dataset with additional datasets we compiled from Facebook and Yelp users to study the interplay between social networks and daily deal sites. First, by studying user activity on Facebook while a deal is running, we provide evidence that daily deal sites benefit from significant word-of-mouth effects during sales events, consistent with results predicted by cascade models. Second, we consider the effects of daily deals on the longer-term reputation of merchants, based on their Yelp reviews before and after they run a daily deal. Our analysis shows that while the number of reviews increases significantly due to daily deals, average rating scores from reviewers who mention daily deals are 10% lower than scores of their peers on average.

/pdf/daily-deals-prediction-social-diffusion-and-reputational-4ohffrrx2c.pdf

Daily deals: prediction, social diffusion, and reputational ramifications

Network traffic can be represented by a Traffic Dispersion Graph (TDG) that contains an edge between two nodes that send a particular type of traffic (e.g., DNS) to one another. TDGs have recently been proposed as an alternative way to interpret and visualize network traffic. Previous studies have focused on static properties of TDGs using graph snapshots in isolation. In this work, we represent network traffic with a series of related graph instances that change over time. This representation facilitates the analysis of the dynamic nature of network traffic, providing additional descriptive power. For example, DNS and P2P graph instances can appear similar when compared in isolation, but the way the DNS and P2P TDGs change over time differs significantly. To quantify the changes over time, we introduce a series of novel metrics that capture changes both in the graph structure (e.g., the average degree) and the participants (i.e., IP addresses) of a TDG. We apply our new methodologies to improve graph-based traffic classification and to detect changes in the profile of legacy applications (e.g., e-mail).

/pdf/exploiting-dynamicity-in-graph-based-traffic-analysis-9bdfap7wof.pdf

Michael Mitzenmacher

Papers

More Robust Hashing: Cuckoo Hashing with a Stash

Graption: A graph-based P2P traffic classification framework for the internet backbone

Loss resilient code with double heavy tailed series of redundant layers

Daily deals: prediction, social diffusion, and reputational ramifications

Exploiting dynamicity in graph-based traffic analysis: techniques and applications