A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment
TLDR
In this paper, a Parallel Random Forest (PRF) algorithm for big data on the Apache Spark platform is presented. And the PRF algorithm is optimized based on a hybrid approach combining dataparallel and task-parallel optimization, and a dual parallel approach is carried out in the training process of RF and a task Directed Acyclic Graph (DAG) is created according to the parallel training process.Abstract:
With the emergence of the big data age, the issue of how to obtain valuable knowledge from a dataset efficiently and accurately has attracted increasingly attention from both academia and industry. This paper presents a Parallel Random Forest (PRF) algorithm for big data on the Apache Spark platform. The PRF algorithm is optimized based on a hybrid approach combining data-parallel and task-parallel optimization. From the perspective of data-parallel optimization, a vertical data-partitioning method is performed to reduce the data communication cost effectively, and a data-multiplexing method is performed is performed to allow the training dataset to be reused and diminish the volume of data. From the perspective of task-parallel optimization, a dual parallel approach is carried out in the training process of RF, and a task Directed Acyclic Graph (DAG) is created according to the parallel training process of PRF and the dependence of the Resilient Distributed Datasets (RDD) objects. Then, different task schedulers are invoked for the tasks in the DAG. Moreover, to improve the algorithm's accuracy for large, high-dimensional, and noisy data, we perform a dimension-reduction approach in the training process and a weighted voting approach in the prediction process prior to parallelization. Extensive experimental results indicate the superiority and notable advantages of the PRF algorithm over the relevant algorithms implemented by Spark MLlib and other studies in terms of the classification accuracy, performance, and scalability. With the expansion of the scale of the random forest model and the Spark cluster, the advantage of the PRF algorithm is more obvious.read more
Citations
More filters
Journal ArticleDOI
Integrated Blockchain and Edge Computing Systems: A Survey, Some Research Issues and Challenges
TL;DR: This survey investigates some of the work that has been done to enable the integrated blockchain and edge computing system and discusses the research challenges, identifying several vital aspects of the integration of blockchain andEdge computing: motivations, frameworks, enabling functionalities, and challenges.
Journal ArticleDOI
Comparative analysis of surface water quality prediction performance and identification of key water parameters using different machine learning models based on big data.
Kangyang Chen,Hexia Chen,Chuanlong Zhou,Yichao Huang,Xiangyang Qi,Ruqin Shen,Fengrui Liu,Min Zuo,Xinyi Zou,Jinfeng Wang,Yan Zhang,Da Chen,Xingguo Chen,Xingguo Chen,Yongfeng Deng,Yongfeng Deng,Hongqiang Ren +16 more
TL;DR: Compared to other 7 models, decision tree (DT), random forest (RF) and deep cascade forest (DCF) trained by data sets of pH, DO, CODMn, and NH3-N had significantly better performance in prediction of all 6 Levels of water quality recommended by Chinese government.
Journal ArticleDOI
Stochastic Online Learning for Mobile Edge Computing: Learning from Changes
TL;DR: Numerical studies show that the network throughput can increase by eight times through adopting stochastic online learning as compared to existing offline implementations of MapReduce, the widely adopted big data analytic framework.
Journal ArticleDOI
A collaborative architecture of the industrial internet platform for manufacturing systems
TL;DR: A collaborative architecture for industrial Internet platform (IIP) called industrial operation system (Ind-OS), which contains the industrial driver, digital thread and micro-services to provide a better cooperative enterprise information system (EIS) environment for manufacturing systems.
Journal ArticleDOI
Multiple convolutional neural networks for multivariate time series prediction
TL;DR: Tests on two large real-world datasets, show that the proposed model, called Multiple CNNs, has a strong advantage over other time series prediction methods.
References
More filters
Journal ArticleDOI
Random Forests
TL;DR: Internal estimates monitor error, strength, and correlation and these are used to show the response to increasing the number of features used in the forest, and are also applicable to regression.
Journal ArticleDOI
Conditional variable importance for random forests
TL;DR: A new, conditional permutation scheme is developed for the computation of the variable importance measure that reflects the true impact of each predictor variable more reliably than the original marginal approach.
Journal ArticleDOI
Data mining with big data
TL;DR: A HACE theorem is presented that characterizes the features of the Big Data revolution, and a Big Data processing model is proposed, from the data mining perspective, which involves demand-driven aggregation of information sources, mining and analysis, user interest modeling, and security and privacy considerations.
Journal Article
Analysis of a random forests model
TL;DR: An in-depth analysis of a random forests model suggested by Breiman (2004), which is very close to the original algorithm, and shows in particular that the procedure is consistent and adapts to sparsity, in the sense that its rate of convergence depends only on the number of strong features and not on how many noise variables are present.