Open AccessJournal Article
MLlib: machine learning in apache spark
Xiangrui Meng,Joseph K. Bradley,Burak Yavuz,Evan R. Sparks,Shivaram Venkataraman,Davies Liu,Jeremy Freeman,DB Tsai,Manish Amde,Sean Owen,Doris Xin,Reynold Xin,Michael J. Franklin,Reza Bosagh Zadeh,Matei Zaharia,Ameet Talwalkar +15 more
Reads0
Chats0
TLDR
MLlib as mentioned in this paper is an open-source distributed machine learning library for Apache Spark that provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives.Abstract:
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLLIB provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLLIB supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLLIB has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.read more
Citations
More filters
Proceedings ArticleDOI
dislib: Large Scale High Performance Machine Learning in Python
TL;DR: This paper presents and evaluates dislib, a distributed machine learning library on top of PyCOMPSs programming model that addresses the issues of other existing libraries and shows that dislib can be up to 9 times faster, and can process data sets up to 16 times larger than other popular distributed machineLearning libraries, such as MLlib.
Journal ArticleDOI
CrossRec: Cross-Domain Recommendations Based on Social Big Data and Cognitive Computing
TL;DR: This work proposes a cross-domain recommender system, including three approaches, based on multi-source social big data, and shows that the accuracies of the three proposed approaches are significantly improved compared with the conventional recommender approaches, such as collaborative filtering and matrix factorization.
Proceedings ArticleDOI
One-Pass Logistic Regression for Label-Drift and Large-Scale Classification on Distributed Systems
TL;DR: A novel variant of LR, namely one-pass logistic regression (OLR) is introduced to offer a principled treatment for label-drift and large-scale classifications and is extended to a distributed setting for parallelization, termed sparkling OLR (Spark-OLR).
Journal ArticleDOI
Seagull: an infrastructure for load prediction and optimized resource allocation
Olga Poppe,Tayo Amuneke,Dalitso Banda,Aritra De,Ari Green,Manon Knoertzer,Ehi Nosakhare,Karthik Rajendran,Deepak Shankargouda,Meina Wang,Alan Au,Carlo Curino,Qun Guo,Alekh Jindal,Ajay Kalhan,Morgan Oslake,Sonia Parchani,Vijay Ramani,Raj Sellappan,Saikat Sen,Sheetal Shrotri,Soundararajan Srinivasan,Ping Xia,Shize Xu,Alicia Yang,Yiwen Zhu +25 more
TL;DR: The Seagull infrastructure is built that processes per-server telemetry, validates the data, trains and deploys ML models, used to predict customer load per server, and optimize service operations, and minimizes interference with user-induced load and improves customer experience.
Book ChapterDOI
Cognitive Computing: Where Big Data Is Driving Us.
TL;DR: The concepts and challenges to design Cognitive Systems are discussed, which will address the questions for Cognitive Systems: What are the needs?
References
More filters
Journal Article
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +15 more
TL;DR: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems, focusing on bringing machine learning to non-specialists using a general-purpose high-level language.
Journal ArticleDOI
Latent dirichlet allocation
TL;DR: This work proposes a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams, and Hofmann's aspect model.
Posted Content
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Andreas Müller,Joel Nothman,Gilles Louppe,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +18 more
TL;DR: Scikit-learn as mentioned in this paper is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems.
Proceedings Article
Latent Dirichlet Allocation
TL;DR: This paper proposed a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams, and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI).
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.