Measuring empirical computational complexity

TLDR: The tool, the Trend Profiler (trend-prof), is described, for constructing models of empirical computational complexity that predict how many times each basic block in a program runs as a linear or a powerlaw function of user-specified features of the program's workloads.

Abstract: The standard language for describing the asymptotic behavior of algorithms is theoretical computational complexity. We propose a method for describing the asymptotic behavior of programs in practice by measuring their empirical computational complexity. Our method involves running a program on workloads spanning several orders of magnitude in size, measuring their performance, and fitting these observations to a model that predicts performance as a function of workload size. Comparing these models to the programmer's expectations or to theoretical asymptotic bounds can reveal performance bugs or confirm that a program's performance scales as expected. Grouping and ranking program locations based on these models focuses attention on scalability-critical code. We describe our tool, the Trend Profiler (trend-prof), for constructing models of empirical computational complexity that predict how many times each basic block in a program runs as a linear (y = a + bx) or a powerlaw (y = axb) function of user-specified features of the program's workloads. We ran trend-prof on several large programs and report cases where a program scaled as expected, beat its worst-case theoretical complexity bound, or had a performance bug.

Figures

Figure 3.56: Top several functions in the lpsolve benchmark, ranked by maximum totalself-cost. We use r for feature rows, c for feature columns, and v1 through v9 for loop count variables.

Figure 3.29: Best fit plots for the Fibonacci heap’s deleteMin operation during Dijkstra’s algorithm. transitive-cost :≈ 32, SE = 9 (top), call-count := n (middle), total-transitive-cost :≈ 36n− 2900, SE = 4370 (bottom).

Figure 3.20: The self-cost :≈ 2, SE = 0 (top), total-self-cost :≈ 2.25n− 38, SE = 19 (middle), and total-self-cost residuals (bottom) of looking up in a chaining hash table each of the n items spread among its 2000 buckets by a uniform random hash function.

Figure 3.43: Best-fit plot (top) and residuals plot (middle) for simple dot’s gvLayoutJobs total-transitive-cost :≈ 0.24n2.49 + 30n1.73 + 3319n− 22912, SE = 6.34× 105. The bottom plot is the best-fit plot for gvLayoutJobs’s callee mincross step’s total-transitive-cost :≈ 0.24n2.49, SE = 5.89× 105.

Figure 3.24: Best-fit plot (top) and residuals plot (bottom) for CF-TrendProf’s model for isort :≈ 0.18n2 + 30n+ 1, SE = 2.58× 108 on sorted, nearly sorted, random, and reverse-sorted inputs.

Figure 3.4: Data records for the bsort example for several workloads. The notation 1r23 is run-length encoding for 1 repeated 23 times.

Full text

Measuring Empirical Computational Complexity
by
Simon Fredrick Goldsmith
B.S. (Carnegie Mellon University) 2001
A dissertation submitted in partial satisfaction of the
requirements for the degree of
Doctor of Philosophy
in
Computer Science
in the
GRADUATE DIVISION
of the
UNIVERSITY OF CALIFORNIA, BERKELEY
Committee in charge:
Professor Alex Aiken, Co-chair
Professor Koushik Sen, Co-chair
Professor Rastislav Bodik
Professor Dor Abrahamson
Fall 2008

The dissertation of Simon Fredrick Goldsmith is approved:
Co-chair Date
Co-chair Date
Date
Date
University of California, Berkeley
Fall 2008

Measuring Empirical Computational Complexity
Copyright 2008
by
Simon Fredrick Goldsmith

1
Abstract
Measuring Empirical Computational Complexity
by
Simon Fredrick Goldsmith
Doctor of Philosophy in Computer Science
University of California, Berkeley
Professor Alex Aiken, Co-chair
Professor Koushik Sen, Co-chair
Scalability is a fundamental problem in computer science. Computer scientists
often describ e the scalability of algorithms in the language of theoretical computational
complexity, bounding the number of operations an algorithm performs as a function of the
size of its input. The main contribution of this dissertation is to provide an analogous
description of the scalability of actual software implementations run on realistic workloads.
We propose a metho d for describing the asymptotic behavior of programs in prac-
tice by measuring their empirical computational complexity. Our method involves running a
program on workloads spanning several orders of magnitude in size, measuring their perfor-
mance, and fitting these observations to a model that predicts performance as a function of
workload size. Comparing these models to the programmer’s expectations or to theoretical
asymptotic bounds can reveal performance bugs or confirm that a program’s performance

2
scales as expected.
We develop our methodology for constructing these models of empirical complexity
as we describe and evaluate two techniques. Our first technique, BB-TrendProf, con-
structs models that predict how many times each basic block runs as a linear (y = a + bx)
or a powerlaw (y = ax
b
) function of user-specified features of the program’s workloads. To
present output succinctly and focus attention on scalability-critical code, BB-TrendProf
groups and ranks program locations based on these models. We demonstrate the power of
BB-TrendProf compared to existing tools by running it on several large programs and
reporting cases where its models show (1) an implementation of a complex algorithm scal-
ing as ex pected, (2) two complex algorithms beating their worst-case theoretical complexity
bounds when run on realistic inputs, and (3) a performance bug.
Our second technique, CF-TrendProf, models performance of loops and func-
tions both p er-function-invocation and per-workload. It improves upon the precision of
BB-TrendProf’s models by using control flow to generate candidates from a richer fam-
ily of models and a novel model selection criteria to select among these candidates. We
show that CF-TrendProf’s improvements to model generation and selection allow it to
correctly characterize or closely approximate the empirical scalability of several well-known
algorithms and data structures and to diagnose several synthetic, but realistic, scalabil-
ity problems without observing an e gregiously expensive workload. We also show that
CF-TrendProf deals with multiple workload features better than BB-TrendProf. We
qualitatively compare the output of BB-TrendProf and CF-TrendProf and discuss
their relative strengths and weaknesses.

Frequently asked questions

Q1. What is the effect of clustering on performance?

Clustering dramatically reduces the number of degrees of freedom of the overall performance model; that is, clustering simplifies their presentation of program performance by dramatically reducing the number of program components whose costs the authors model. 

Q2. What are the functions that can be used to allocate performance cost to different callers?

Contexts are useful for apportioning performance cost of a library to different callers or even costs of data structure operations to different instances of the data structure. 

Q3. What is the way to mark function invocations?

CF-TrendProf allows the user to mark function invocations with contexts basedon the call graph or on arbitrary runtime data values. 

Q4. What is the way to understand the relationship between performance and workload features?

the log-log scatter plot would seem to be more appropriate for understanding the relationship between performance and workload features since, unlike a linear-linear scatter plot, a constant relative error corresponds to constant distance. 

Q5. What is the danger of allowing a model to grow gratuitously complex?

there is a danger that if the authors allow their models to grow gratuitously complex, that they will overfit the training data and not generalize to other data. 

Q6. What is the mechanism to verify bounds?

The mechanism is to annotate each function with types that describe how many steps the function takes to compute its result and use a dependent type system to verify these bounds. 

Q7. How would a user realize an ideal situation in a larger algorithm?

In order to realize such an ideal situation in the context of a larger algorithm, a user would likely have to provide CF-TrendProf with suitable context annotations and invocation features. 

Q8. What is the effect of the CF-TrendProf on the performance of the benchmark?

This dependence of performance on such subtle properties means that the apparent scalability of an algorithm that CF-TrendProf measures is as much a consequence of the code being measured, as it is of the empirical distribution of workloads. 

Q9. What is the purpose of reducing the number of unknown entities?

In a setting where performance need not be a clean function of workload features, reducing the number of unknown entities is useful.