Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.

/pdf/xgboost-a-scalable-tree-boosting-system-48ocu6x4c7.pdf

XGBoost: A Scalable Tree Boosting System

Introduction to linear and nonlinear programming

Matrix Factorization Techniques for Recommender Systems

This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications

https://dl.acm.org/doi/pdf/10.1145/2934664

Apache Spark: a unified engine for big data processing

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLLIB provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLLIB supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLLIB has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

/pdf/mllib-machine-learning-in-apache-spark-27ddi10m3m.pdf

MLlib: machine learning in apache spark

Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. declarative queries and optimized storage), and lets SQL users call complex analytics libraries in Spark (e.g. machine learning). Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g. schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL-on-Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.

http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf

Spark SQL: Relational Data Processing in Spark

Low-distortion embeddings are critical building blocks for developing random sampling and random projection algorithms for common linear algebra problems. We show that, given a matrix A ∈ Rn x d with n >> d and a p ∈ [1, 2), with a constant probability, we can construct a low-distortion embedding matrix Π ∈ RO(poly(d)) x n that embeds Ap, the lp subspace spanned by A's columns, into (RO(poly(d)), |~cdot~|p); the distortion of our embeddings is only O(poly(d)), and we can compute Π A in O(nnz(A)) time, i.e., input-sparsity time. Our result generalizes the input-sparsity time l2 subspace embedding by Clarkson and Woodruff [STOC'13]; and for completeness, we present a simpler and improved analysis of their construction for l2. These input-sparsity time lp embeddings are optimal, up to constants, in terms of their running time; and the improved running time propagates to applications such as (1 pm e)-distortion lp subspace embedding and relative-error lp regression. For l2, we show that a (1+e)-approximate solution to the l2 regression problem specified by the matrix A and a vector b ∈ Rn can be computed in O(nnz(A) + d3 log(d/e) /e^2) time; and for lp, via a subspace-preserving sampling procedure, we show that a (1 pm e)-distortion embedding of Ap into RO(poly(d)) can be computed in O(nnz(A) ⋅ log n) time, and we also show that a (1+e)-approximate solution to the lp regression problem minx ∈ Rd |A x - b|p can be computed in O(nnz(A) ⋅ log n + poly(d) log(1/e)/e2) time. Moreover, we can also improve the embedding dimension or equivalently the sample size to O(d3+p/2 log(1/e) / e2) without increasing the complexity.

/pdf/low-distortion-subspace-embeddings-in-input-sparsity-time-1tkzav8hf5.pdf

Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression

We describe a parallel iterative least squares solver named \textttLSRN that is based on random normal projection. \textttLSRN computes the min-length solution to $\min_{x \in \mathbb{R}^n} \|A x - b\|_2$, where $A \in \mathbb{R}^{m \times n}$ with $m \gg n$ or $m \ll n$, and where $A$ may be rank-deficient. Tikhonov regularization may also be included. Since $A$ is involved only in matrix-matrix and matrix-vector multiplications, it can be a dense or sparse matrix or a linear operator, and LSRN automatically speeds up when $A$ is sparse or a fast linear operator. The preconditioning phase consists of a random normal projection, which is embarrassingly parallel, and a singular value decomposition of size $\lceil \gamma \min(m,n) \rceil \times \min(m,n)$, where $\gamma$ is moderately larger than $1$, e.g., $\gamma = 2$. We prove that the preconditioned system is well-conditioned, with a strong concentration result on the extreme singular values, and hence that the number of iterations is fully predictable ...

Xiangrui Meng

Papers

Apache Spark: a unified engine for big data processing

MLlib: machine learning in apache spark

Spark SQL: Relational Data Processing in Spark

Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression

Lsrn: a parallel iterative solver for strongly over- or underdetermined systems.