Open AccessJournal Article
MLlib: machine learning in apache spark
Xiangrui Meng,Joseph K. Bradley,Burak Yavuz,Evan R. Sparks,Shivaram Venkataraman,Davies Liu,Jeremy Freeman,DB Tsai,Manish Amde,Sean Owen,Doris Xin,Reynold Xin,Michael J. Franklin,Reza Bosagh Zadeh,Matei Zaharia,Ameet Talwalkar +15 more
Reads0
Chats0
TLDR
MLlib as mentioned in this paper is an open-source distributed machine learning library for Apache Spark that provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives.Abstract:
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLLIB provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLLIB supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLLIB has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.read more
Citations
More filters
Journal ArticleDOI
Applications of Neural Networks in Biomedical Data Analysis
TL;DR: This review discusses the latest networks and how they work, with a focus on the analysis of biomedical data, particularly biomarkers in bioimage data, and presents a data analysis of publications about neural networks to provide a quantitative insight into the use of network types and the number of journals per year.
System-Aware Optimization for Machine Learning at Scale
TL;DR: A general optimization framework for machine learning, CoCoA, is proposed that gives careful consideration to systems parameters, often incorporating them directly into the method and theory, and can achieve orders-of-magnitude speedups for solving modern machine learning problems at scale.
Sharing without Showing: Building Secure Collaborative Systems
TL;DR: This dissertation presents four systems that utilize hardware enclaves as well as advanced cryptographic techniques for secure computation on workloads that range from SQL analytics to machine learning that are orders of magnitude faster compared to prior work or the more straightforward ways of integrating cryptography into systems.
Book ChapterDOI
Challenges in Storing and Processing Big Data Using Hadoop and Spark
TL;DR: This chapter discusses the need of new analytical platforms, in particular different big data frameworks like Hadoop and Spark, along with MapReduce programming concepts, and describes steps to develop programs in Spark Streaming, Spark SQL and GRAPHX.
Tool support for architectural decision making in large software intensive projects
TL;DR: A tool (ADeX) is presented to manage ADDs and support the decision-making process and can answer questions about what decisions have been made and which elements and quality attributes are affected by a decision.
References
More filters
Journal Article
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +15 more
TL;DR: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems, focusing on bringing machine learning to non-specialists using a general-purpose high-level language.
Journal ArticleDOI
Latent dirichlet allocation
TL;DR: This work proposes a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams, and Hofmann's aspect model.
Posted Content
Scikit-learn: Machine Learning in Python
Fabian Pedregosa,Gaël Varoquaux,Alexandre Gramfort,Vincent Michel,Bertrand Thirion,Olivier Grisel,Mathieu Blondel,Andreas Müller,Joel Nothman,Gilles Louppe,Peter Prettenhofer,Ron Weiss,Vincent Dubourg,Jake Vanderplas,Alexandre Passos,David Cournapeau,Matthieu Brucher,Matthieu Perrot,Edouard Duchesnay +18 more
TL;DR: Scikit-learn as mentioned in this paper is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems.
Proceedings Article
Latent Dirichlet Allocation
TL;DR: This paper proposed a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams, and Hof-mann's aspect model, also known as probabilistic latent semantic indexing (pLSI).
Journal ArticleDOI
MapReduce: simplified data processing on large clusters
Jeffrey Dean,Sanjay Ghemawat +1 more
TL;DR: This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable.