The learning process and hyper-parameter optimization of artificial neural networks (ANNs) and deep learning (DL) architectures is considered one of the most challenging machine learning problems. Several past studies have used gradient-based back propagation methods to train DL architectures. However, gradient-based methods have major drawbacks such as stucking at local minimums in multi-objective cost functions, expensive execution time due to calculating gradient information with thousands of iterations and needing the cost functions to be continuous. Since training the ANNs and DLs is an NP-hard optimization problem, their structure and parameters optimization using the meta-heuristic (MH) algorithms has been considerably raised. MH algorithms can accurately formulate the optimal estimation of DL components (such as hyper-parameter, weights, number of layers, number of neurons, learning rate, etc.). This paper provides a comprehensive review of the optimization of ANNs and DLs using MH algorithms. In this paper, we have reviewed the latest developments in the use of MH algorithms in the DL and ANN methods, presented their disadvantages and advantages, and pointed out some research directions to fill the gaps between MHs and DL methods. Moreover, it has been explained that the evolutionary hybrid architecture still has limited applicability in the literature. Also, this paper classifies the latest MH algorithms in the literature to demonstrate their effectiveness in DL and ANN training for various applications. Most researchers tend to extend novel hybrid algorithms by combining MHs to optimize the hyper-parameters of DLs and ANNs. The development of hybrid MHs helps improving algorithms performance and capable of solving complex optimization problems. In general, the optimal performance of the MHs should be able to achieve a suitable trade-off between exploration and exploitation features. Hence, this paper tries to summarize various MH algorithms in terms of the convergence trend, exploration, exploitation, and the ability to avoid local minima. The integration of MH with DLs is expected to accelerate the training process in the coming few years. However, relevant publications in this way are still rare. 

https://link.springer.com/content/pdf/10.1007/s11063-022-11055-6.pdf

Application of Meta-Heuristic Algorithms for Training Neural Networks and Deep Learning Architectures: A Comprehensive Review

Non-point source (NPS) pollution is an important factor affecting the quality of water environment. In recent years, a large number of online water quality monitoring stations have been used to obtain continuous time series water quality monitoring data. These data provide the necessary basis for the application of deep learning methods in water quality prediction. However, the prediction accuracy of traditional deep learning methods is low, especially in predicting the water quality with NPS pollution. Aiming to address this limitation, a novel deep learning model named SOD-VGG-LSTM with the simulation-observation difference (SOD) modular based on physical process, the visual geometry (VGG) modular reflecting spatial characteristics, and the long short-term memory (LSTM) modular based on deep learning method was developed to improve the accuracy of the water quality prediction with NPS pollution. The established model can overcome the problem that mechanism models can not predict the changes of water quality on the hourly or minute time scale. The model was applied in Lijiang River watershed. Experimental results indicated that the proposed model had the highest accuracy in the extreme value prediction compared with the mechanism model and LSTM model. The maximum relative errors between the predicted and observed results for DO, CODMn, NH3-N, and TP were 8.47%, 19.76%, 24.1%, and 35.4%, respectively. The model evaluation demonstrated that the established SOD-VGG-LSTM model achieved superior computational performance compared to Auto Regression Integreate Moving Average model (ARIMA), Support Vector Regression model (SVR), and Recurrent Neural Network model (RNN). The evaluation results showed that SOD-VGG-LSTM achieved 3.2–39.3% higher R2 than ARIMA, SVR and RNN. The proposed model can provide a new method for water quality prediction with NPS pollution. 

A novel model for water quality prediction caused by non-point sources pollution based on deep learning and feature extraction methods

Precipitation forecasting is essential for the assessment of several hydrological processes. This study shows that based on a machine learning approach, reliable models for precipitation prediction can be developed. The tropical monsoon-climate northern region of Bangladesh, including the Rangpur and Sylhet division, was chosen as the case study. Two machine learning algorithms were used: M5P and support vector regression. Moreover, a novel hybrid model based on the two algorithms was developed. The performance of prediction models was assessed by means of evaluation metrics and graphical representations. A sensitivity analysis was also carried out to assess the prediction accuracy as the number of exogenous inputs reduces and lag times increases. Overall, the hybrid model M5P-SVR led to the best predictions among used models in this study, with R2 values up to 0.87 and 0.92 for the stations of Rangpur and Sylhet, respectively.

/pdf/precipitation-forecasting-in-northern-bangladesh-using-a-26z4rr2h.pdf

Precipitation Forecasting in Northern Bangladesh Using a Hybrid Machine Learning Model

Characteristics and Classification of Microseismic Signals in Heading Face of Coal Mine: Implication for Coal and Gas Outburst Warning

Runoff Forecasting using Convolutional Neural Networks and optimized Bi-directional Long Short-term Memory

Flood forecasting is an essential non-engineering measure for flood prevention and disaster reduction. Many models have been developed to study the complex and highly random rainfall-runoff process. In recent years, artificial intelligence methods, such as the artificial neural network (ANN), have attempted to construct rainfall-runoff models. The more advanced deep learning methods of long short-term memory (LSTM) network have been proved to better predict hydrological time series. However, the selection of LSTM hyperparameters in the past mostly relied on the experience of the staff, which often led to failure to achieve the best performance. The aim of this study is to develop a method to improve flood forecast accuracy and lead time. A deep learning neural network model based on LSTM networks and particle swarm optimization (PSO) is proposed in this paper. The PSO algorithm was used to optimize the LSTM hyperparameter to improve the ability to learn data sequence features. The model focuses on the Jingle Watershed in the Fenhe River and the Lushi Watershed in the Luohe River and was used to predict flood processes using rainfall and runoff observation data from stations in the watersheds. We evaluated the performance of the model with the Nash Sutcliffe efficiency coefficient, root mean square error, and bias. The results show that the PSO-LSTM model outperforms the M-EIES, ANN, PSO-ANN, and LSTM at all stations in the watersheds. The PSO-LSTM model improves the flood forecasting accuracy at different lead times, especially for those exceeding 6 h, and has higher prediction accuracy and stability. The PSO-LSTM model could be used to improve accuracy in short-term flood forecast applications. 

Research on Particle Swarm Optimization in LSTM Neural Networks for Rainfall-Runoff Simulation

Relative chlorophyll content (SPAD) is an important index for characterizing the nitrogen nutrient status of plants. Continuous, rapid, nondestructive, and accurate estimation of SPAD values in wheat after heading stage can positively impact subsequent nitrogen fertilization management strategies, which regulate grain filling and yield quality formation. In this study, the estimation of SPAD of leaf relative chlorophyll content in spring wheat was conducted at the experimental base in Wuyuan County, Inner Mongolia in 2021. Multispectral images of different nitrogen application levels at 7, 14, 21, and 28 days after the wheat heading stage were acquired by DJI P4M UAV. A total of 26 multispectral vegetation indices were constructed, and the measured SPAD values of wheat on the ground were obtained simultaneously using a handheld chlorophyll meter. Four machine learning algorithms, including deep neural networks (DNN), partial least squares (PLS), random forest (RF), and Adaptive Boosting (Ada) were used to construct SPAD value estimation models at different time from heading growth stages. The model’s progress was evaluated by the coefficient of determination (R2), root mean square error (RMSE), and mean absolute error (MAPE). The results showed that the optimal SPAD value estimation models for different periods of independent reproductive growth stages of wheat were different, with PLS as the optimal estimation model at 7 and 14 days after heading, RF as the optimal estimation model at 21 days after heading, and Ada as the optimal estimation model at 28 d after heading. The highest accuracy was achieved using the PLS model for estimating SPAD values at 14 d after heading (training set R2 = 0.767, RMSE = 3.205, MAPE = 0.060, and R2 = 0.878, RMSE = 2.405, MAPE = 0.045 for the test set). The combined analysis concluded that selecting multiple vegetation indices as input variables of the model at 14 d after heading stage and using the PLS model can significantly improve the accuracy of SPAD value estimation, provides a new technical support for rapid and accurate monitoring of SPAD values in spring wheat.

/pdf/estimation-of-relative-chlorophyll-content-in-spring-wheat-tocv5yos.pdf

Estimation of Relative Chlorophyll Content in Spring Wheat Based on Multi-Temporal UAV Remote Sensing

Climate change and rapid urbanization have increased pressure on drainage systems, posing new challenges to preventing and controlling urban waterlogging. In 2013, China proposed the Sponge City, a strategic measure for urban waterlogging control. This study quantifies the effects of stormwater management measures in runoff reduction for different levels of rainfall and conducts a one‐dimensional visual analysis of urban waterlogging risks. At the same time, the best cost‐effective scheme is determined based on life‐cycle cost, analytic hierarchy process, and regret decision theory. The results showed stormwater management measures could realize the function of runoff control and waterlogging prevention, especially under low precipitation. However, these measures were still not enough to eliminate waterlogging risk. Combined measures have stronger runoff control capabilities than single measures. Considering economic, environmental, and operational impacts comprehensively, the combined measures of bio‐retention (BR), permeable pavement (PP), and green roof (GR) were determined as the best cost‐effective scheme because of the lowest regret value. The proposed method is helpful to provide reference and decision‐making basis for the construction of sponge cities in the future.

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/jfr3.12834

Comprehensive performance evaluation of stormwater management measures for sponge city construction: A case study in Gui'an New District, China

Identify runoff generation patterns of check dams and terraces and the effects on runoff: a case study


 Northwest China has experienced dramatic changes in vegetation cover over the past few decades with the Yellow River Basin (YRB) being the most representative area. As the major climate-sensitive area in China, vegetation cover change is one reason for its impact on surface air temperature (SAT). This study uses the observation minus reanalysis (OMR) method to reveal the spatio-temporal variations of vegetation cover and its impact on SAT change over non-urban areas of the YRB from 1982 to 2015. The Global Inventory Modeling and Mapping Studies dataset, SAT derived from meteorological stations, and ERA-interim reanalysis temperature data were used to analyze the relationship between normalized difference vegetation index (NDVI) and temperature variation caused by vegetation change. The NDVI trend (SlopeNDVI) of the entire YRB reached 1.11×10−2 decade−1, which indicated the recovery of vegetation in general. The impact of variation in vegetation conditions on SAT change during 1982–2015 was estimated to be 0.037 °C decade−1, which contributed 7.62% to the temperature change. The mean annual NDVI (MNDVI) and SlopeNDVI in the YRB were significantly negatively correlated (P<0.001) with OMR temperature variation. A negative correlation was exhibited in semi-arid and semi-humid regions, whereas a positive correlation was found in the arid region. The observed changes in vegetation and SAT in the YRB support the theory of the impact of vegetation variation on SAT in China.

Qiang Wu

Papers

Research on Particle Swarm Optimization in LSTM Neural Networks for Rainfall-Runoff Simulation

Estimation of Relative Chlorophyll Content in Spring Wheat Based on Multi-Temporal UAV Remote Sensing

Comprehensive performance evaluation of stormwater management measures for sponge city construction: A case study in Gui'an New District, China

Identify runoff generation patterns of check dams and terraces and the effects on runoff: a case study

Spatio-temporal variations of vegetation cover and its influence on surface air temperature change over the Yellow River Basin, China