XGBoost: A Scalable Tree Boosting System
Tianqi Chen,Carlos Guestrin +1 more
TLDR
This paper proposes a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning and provides insights on cache access patterns, data compression and sharding to build a scalable tree boosting system called XGBoost.Abstract:
Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.read more
Citations
More filters
Proceedings Article
LightGBM: a highly efficient gradient boosting decision tree
TL;DR: It is proved that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size, and is called LightGBM.
Posted Content
CatBoost: unbiased boosting with categorical features
Liudmila Ostroumova Prokhorenkova,Gleb Gusev,Aleksandr Vorobev,Anna Veronika Dorogush,Andrey Gulin +4 more
TL;DR: CatBoost as discussed by the authors is a new gradient boosting toolkit that uses ordered boosting, a permutation-driven alternative to the classic algorithm, and an innovative algorithm for processing categorical features.
Journal ArticleDOI
The rise of deep learning in drug discovery.
TL;DR: The first wave of applications of deep learning in pharmaceutical research has emerged in recent years, and its utility has gone beyond bioactivity predictions and has shown promise in addressing diverse problems in drug discovery.
Journal ArticleDOI
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
Li Yang,Abdallah Shami +1 more
TL;DR: This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively and introducing several state-of-the-art optimization techniques.
Posted Content
MoleculeNet: A Benchmark for Molecular Machine Learning
Zhenqin Wu,Bharath Ramsundar,Evan N. Feinberg,Joseph Gomes,Caleb Geniesse,Aneesh S. Pappu,Karl Leswing,Vijay S. Pande +7 more
TL;DR: MoleculeNet benchmarks demonstrate that learnable representations are powerful tools for molecular machine learning and broadly offer the best performance, however, this result comes with caveats.
References
More filters
Journal ArticleDOI
Random Forests
TL;DR: Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Journal Article
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +15 more
TL;DR: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems, focusing on bringing machine learning to non-specialists using a general-purpose high-level language.
Journal ArticleDOI
Greedy function approximation: A gradient boosting machine.
TL;DR: A general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion, and specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification.
Journal ArticleDOI
Additive Logistic Regression : A Statistical View of Boosting
TL;DR: This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
Journal ArticleDOI
Stochastic gradient boosting
TL;DR: It is shown that both the approximation accuracy and execution speed of gradient boosting can be substantially improved by incorporating randomization into the procedure.