The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare. 

https://link.springer.com/content/pdf/10.1007/s11063-022-11055-6.pdf

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

Non-point source (NPS) pollution is an important factor affecting the quality of water environment. In recent years, a large number of online water quality monitoring stations have been used to obtain continuous time series water quality monitoring data. These data provide the necessary basis for the application of deep learning methods in water quality prediction. However, the prediction accuracy of traditional deep learning methods is low, especially in predicting the water quality with NPS pollution. Aiming to address this limitation, a novel deep learning model named SOD-VGG-LSTM with the simulation-observation difference (SOD) modular based on physical process, the visual geometry (VGG) modular reflecting spatial characteristics, and the long short-term memory (LSTM) modular based on deep learning method was developed to improve the accuracy of the water quality prediction with NPS pollution. The established model can overcome the problem that mechanism models can not predict the changes of water quality on the hourly or minute time scale. The model was applied in Lijiang River watershed. Experimental results indicated that the proposed model had the highest accuracy in the extreme value prediction compared with the mechanism model and LSTM model. The maximum relative errors between the predicted and observed results for DO, CODMn, NH3-N, and TP were 8.47%, 19.76%, 24.1%, and 35.4%, respectively. The model evaluation demonstrated that the established SOD-VGG-LSTM model achieved superior computational performance compared to Auto Regression Integreate Moving Average model (ARIMA), Support Vector Regression model (SVR), and Recurrent Neural Network model (RNN). The evaluation results showed that SOD-VGG-LSTM achieved 3.2–39.3% higher R2 than ARIMA, SVR and RNN. The proposed model can provide a new method for water quality prediction with NPS pollution. 

A novel model for water quality prediction caused by non-point sources pollution based on deep learning and feature extraction methods

Precipitation forecasting is essential for the assessment of several hydrological processes. This study shows that based on a machine learning approach, reliable models for precipitation prediction can be developed. The tropical monsoon-climate northern region of Bangladesh, including the Rangpur and Sylhet division, was chosen as the case study. Two machine learning algorithms were used: M5P and support vector regression. Moreover, a novel hybrid model based on the two algorithms was developed. The performance of prediction models was assessed by means of evaluation metrics and graphical representations. A sensitivity analysis was also carried out to assess the prediction accuracy as the number of exogenous inputs reduces and lag times increases. Overall, the hybrid model M5P-SVR led to the best predictions among used models in this study, with R2 values up to 0.87 and 0.92 for the stations of Rangpur and Sylhet, respectively.

/pdf/precipitation-forecasting-in-northern-bangladesh-using-a-26z4rr2h.pdf

Precipitation Forecasting in Northern Bangladesh Using a Hybrid Machine Learning Model

Characteristics and Classification of Microseismic Signals in Heading Face of Coal Mine: Implication for Coal and Gas Outburst Warning

Runoff Forecasting using Convolutional Neural Networks and optimized Bi-directional Long Short-term Memory

Flood forecasting is an essential non-engineering measure for flood prevention and disaster reduction. Many models have been developed to study the complex and highly random rainfall-runoff process. In recent years, artificial intelligence methods, such as the artificial neural network (ANN), have attempted to construct rainfall-runoff models. The more advanced deep learning methods of long short-term memory (LSTM) network have been proved to better predict hydrological time series. However, the selection of LSTM hyperparameters in the past mostly relied on the experience of the staff, which often led to failure to achieve the best performance. The aim of this study is to develop a method to improve flood forecast accuracy and lead time. A deep learning neural network model based on LSTM networks and particle swarm optimization (PSO) is proposed in this paper. The PSO algorithm was used to optimize the LSTM hyperparameter to improve the ability to learn data sequence features. The model focuses on the Jingle Watershed in the Fenhe River and the Lushi Watershed in the Luohe River and was used to predict flood processes using rainfall and runoff observation data from stations in the watersheds. We evaluated the performance of the model with the Nash Sutcliffe efficiency coefficient, root mean square error, and bias. The results show that the PSO-LSTM model outperforms the M-EIES, ANN, PSO-ANN, and LSTM at all stations in the watersheds. The PSO-LSTM model improves the flood forecasting accuracy at different lead times, especially for those exceeding 6 h, and has higher prediction accuracy and stability. The PSO-LSTM model could be used to improve accuracy in short-term flood forecast applications. 

Research on Particle Swarm Optimization in LSTM Neural Networks for Rainfall-Runoff Simulation

As a world-renowned entertainment company, Walt Disney Company has exerted an important influence on the development of the world's entertainment industry. This paper introduces the development status of Walt Disney Company and analyzes the development strategy of Walt Disney Company. This paper argues that although Walt Disney Company faces fierce competition, it has a good foundation for development and comprehensive entertainment products. Finally, this paper puts forward some suggestions, hoping to contribute to the sustainable development of Walt Disney Company.

/pdf/introduction-and-development-strategy-of-walt-disney-company-2fg0wrvd.pdf

Introduction and Development Strategy of Walt Disney Company

Facing a great deal of materials and data in the process of design and manufacturing of clutch product, Quick and accurate knowledge extraction from the literature resources and web pages is a key to the design of vehicle clutch The motheds of automatic knowledge acquisition based on literature and XML pages were proposed as well as specific algorithms and processes of knowledge extraction from documents and web pages were constructedTechniques of knowledge element extraction, agent and ontology were applied to the intelligent type selection systemPractice tells that this intelligent type selection system can better assist intelligent type selection and design of vehicle clutch and knowledge management

Yuan Hao Xu

Papers

Research on Particle Swarm Optimization in LSTM Neural Networks for Rainfall-Runoff Simulation

Introduction and Development Strategy of Walt Disney Company

Study on Method of Knowledge Acquisition and its Application in the Design of Vehicle Clutch