This study proposes novel Long Short-Term Memory (LSTM)-based classifiers through developing the internal structure of LSTM neural networks using 26 state activation functions as alternatives to the traditional hyperbolic tangent (tanh) activation function. The LSTM networks have high performance in solving the vanishing gradient problem that is observed in recurrent neural networks. Performance investigations were carried out utilizing three distinct deep learning optimization algorithms to evaluate the efficiency of the proposed state activation functions-based LSTM classifiers for two different classification tasks. The simulation results demonstrate that the proposed classifiers that use the Modified Elliott, Softsign, Sech, Gaussian, Bitanh1, Bitanh2 and Wave as state activation functions trump the tanh-based LSTM classifiers in terms of classification accuracy. The proposed classifiers are encouraged to be utilized and tested for other classification tasks.

https://ieeexplore.ieee.org/ielx7/6287639/6514899/09885032.pdf

Developing Novel Activation Functions Based Deep Learning LSTM for Classification

Wind turbine airfoils optimization design by generalized regression neural network under small sample

This study proposes novel Long Short-Term Memory (LSTM)-based classifiers through developing the internal structure of LSTM neural networks using 26 state activation functions as alternatives to the traditional hyperbolic tangent (tanh) activation function. The LSTM networks have high performance in solving the vanishing gradient problem that is observed in recurrent neural networks. Performance investigations were carried out utilizing three distinct deep learning optimization algorithms to evaluate the efficiency of the proposed state activation functions-based LSTM classifiers for two different classification tasks. The simulation results demonstrate that the proposed classifiers that use the Modified Elliott, Softsign, Sech, Gaussian, Bitanh1, Bitanh2 and Wave as state activation functions trump the tanh-based LSTM classifiers in terms of classification accuracy. The proposed classifiers are encouraged to be utilized and tested for other classification tasks. 

/pdf/developing-novel-activation-functions-based-deep-learning-1zcjr6ov.pdf

In this paper, we suggest improving the performance of developed activation function-based Deep Learning Long Short-Term Memory (DLLSTM) structures by employing robust loss functions like Mean Absolute Error <inline-formula> <tex-math notation="LaTeX">$(MAE)$ </tex-math></inline-formula> and Sum Squared Error <inline-formula> <tex-math notation="LaTeX">$(SSE)$ </tex-math></inline-formula> to create new classification layers. The classification layer is the last layer in any DLLSTM neural network structure where the loss function resides. The LSTM is an improved recurrent neural network that fixes the problem of the vanishing gradient that goes away and other issues. Fast convergence and optimum performance depend on the loss function. Three loss functions (default<inline-formula> <tex-math notation="LaTeX">$(Crossentropyex)$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$(MAE)$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$(SSE))$ </tex-math></inline-formula> that compute the error between the actual and desired output for two distinct applications were used to examine the effectiveness of the suggested DLLSTM classifier. The results show that one of the suggested classifiers’ specific loss functions <inline-formula> <tex-math notation="LaTeX">$(SSE))$ </tex-math></inline-formula> works better than other loss functions and does a great job. The suggested functions <inline-formula> <tex-math notation="LaTeX">$Softsign$ </tex-math></inline-formula>, Modified-Elliott, Root-sig, Bi-tanh1, Bi-tanh2, <inline-formula> <tex-math notation="LaTeX">$Sech$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$wave$ </tex-math></inline-formula> are more accurate than the tanh function.

https://doi.org/10.1109/access.2023.3275964

Developing Novel Robust Loss Functions-Based Classification Layers for DLLSTM Neural Networks

In this paper, we suggest improving the performance of developed activation function-based Deep Learning Long Short-Term Memory (DLLSTM) structures by employing robust loss functions like Mean Absolute Error <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(MAE)$ </tex-math></inline-formula> and Sum Squared Error <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(SSE)$ </tex-math></inline-formula> to create new classification layers. The classification layer is the last layer in any DLLSTM neural network structure where the loss function resides. The LSTM is an improved recurrent neural network that fixes the problem of the vanishing gradient that goes away and other issues. Fast convergence and optimum performance depend on the loss function. Three loss functions (default <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(Crossentropyex)$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(MAE)$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(SSE))$ </tex-math></inline-formula> that compute the error between the actual and desired output for two distinct applications were used to examine the effectiveness of the suggested DLLSTM classifier. The results show that one of the suggested classifiers’ specific loss functions <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$(SSE))$ </tex-math></inline-formula> works better than other loss functions and does a great job. The suggested functions <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Softsign$ </tex-math></inline-formula> , Modified-Elliott, Root-sig, Bi-tanh1, Bi-tanh2, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Sech$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$wave$ </tex-math></inline-formula> are more accurate than the tanh function. 

/pdf/developing-novel-robust-loss-functions-based-classification-227lsgtu.pdf

Furkan G. Oztiryaki

Papers

Airfoil Performance Analysis Using Shallow Neural Networks