Logistic regression is a supervised binary classification algorithm in machine learning. It is an important part of neural network and convolutional neural network. An important part of the logistic regression algorithm is to find the optimal parameters of the loss function, which is often a non-linear convex optimization problem. Generally, the gradient descent method or the stochastic gradient descent method in linear search are used in this nonlinear problem, but these methods are easy to fall into the trap of local minimum and have weak global convergence. In order to preserve the gradient information better and enhance the global convergence of the algorithm, this paper introduces fixed memory step gradient descent method into the optimization part of logistic regression algorithm, and combines OVR strategy to solve the problem of multi-class classification of data.

Application of Logistic Regression with Fixed Memory Step Gradient Descent Method in Multi-Class Classification Problem

Ridge regression is a supervised biased estimation regression method in machine learning. In fact, its principle is similar to that of ordinary least squares(OLS) estimation, except that OLS is improved by giving up the unbiasedness of OLS plus a penalty term, which solves the problem of multicollinearity (approximate linear correlation of independent variables), between data. Ridge regression algorithms are the same in optimizing the loss function of linear regression, and usually use gradient descent or stochastic gradient descent. However, when dealing with complex nonlinear problems, these methods can easily fall into the trap of local minima, and global consciousness and convergence are not strong. In order to retain the gradient information well and enhance the global convergence, the gradient descent method of memory-dependent derivative is introduced into the optimized loss function part of the ridge regression algorithm.

Ridge Regression Based on Gradient Descent Method with Memory Dependent Derivative

Convolutional neural network is the most important algorithm in the field of deep learning. The traditional convolution neural network usually uses Sigmoid or Relu as the activation function, but the two sides of Sigmoid are saturated, and Relu has a dead zone, which is very easy to cause gradient disappearance and gradient explosion. In this paper, Mish activation function is introduced into LENET-5 convolutional neural network, which overcomes the shortcomings of traditional activation function. At the same time, the fixed memory step gradient descent method is used to replace the gradient descent method of the optimization part, which improves the global convergence of the algorithm.

Lenet-5 Convolution Neural Network with Mish Activation Function and Fixed Memory Step Gradient Descent Method

As is known to all, Generative Antagonism Network (GAN) plays an important role in the field of image generation and classification. However, there are some shortcomings in it, one of which is that the generator and discriminator cannot converge to the real data at the same time. Therefore, Triple-GAN is proposed. Compared with GAN, it is composed of three parts: classifier, generator and discriminator, which guarantees that the data distribution of generator and classifier can converge to the real distribution. Yet, there are still some traditional limitations in the field of activation function and gradient descent such as gradient disappearance and gradient explosion. Thus, this paper adopts the new activation function Mish, the gradient ascending method and the gradient descending method instead of the original activation function and the gradient ascending algorithm. When a certain sparsity prevents the gradient from exploding and disappearing, the smooth curve of negative input value near the origin makes up for the defect of inactivation of neurons flowing into large gradient,and thus achieving better activation efficiency and greatly improving the efficiency of deep learning.

Triple-GAN with Variable Fractional Order Gradient Descent Method and Mish Activation Function

Fractional calculus has gained a lot of attention in the last couple of years. Researchers have discovered that processes in various fields follow fractional dynamics rather than ordinary integer-ordered dynamics, meaning that the corresponding differential equations feature non-integer valued derivatives. There are several arguments for why this is the case, one of which is that fractional derivatives inherit spatiotemporal memory and/or the ability to express complex naturally occurring phenomena. Another popular topic nowadays is machine learning, i.e., learning behavior and patterns from historical data. In our ever-changing world with ever-increasing amounts of data, machine learning is a powerful tool for data analysis, problem-solving, modeling, and prediction. It has provided many further insights and discoveries in various scientific disciplines. As these two modern-day topics hold a lot of potential for combined approaches in terms of describing complex dynamics, this article review combines approaches from fractional derivatives and machine learning from the past, puts them into context, and thus provides a list of possible combined approaches and the corresponding techniques. Note, however, that this article does not deal with neural networks, as there is already extensive literature on neural networks and fractional calculus. We sorted past combined approaches from the literature into three categories, i.e., preprocessing, machine learning and fractional dynamics, and optimization. The contributions of fractional derivatives to machine learning are manifold as they provide powerful preprocessing and feature augmentation techniques, can improve physically informed machine learning, and are capable of improving hyperparameter optimization. Thus, this article serves to motivate researchers dealing with data-based problems, to be specific machine learning practitioners, to adopt new tools, and enhance their existing approaches.

/pdf/combining-fractional-derivatives-and-machine-learning-a-1uch0ipg.pdf

Combining Fractional Derivatives and Machine Learning: A Review

This paper proposes an affine scaling interior trust-region method in association with nonmonotone line search filter technique for solving nonlinear optimization problems subject to linear inequality constraints. Based on a Newton step which is derived from the complementarity conditions of linear inequality constrained optimization, a trust-region subproblem subject only to an ellipsoidal constraint is defined by minimizing a quadratic model with an appropriate quadratic function and scaling matrix. The nonmonotone schemes combining with trust-region strategy and line search filter technique can bring about speeding up the convergence progress in the case of high nonlinear. A new backtracking relevance condition is given which assures global convergence without using the switching condition used in the traditional line search filter technique. The fast local convergence rate of the proposed algorithm is achieved which is not depend on any external restoration procedure. The preliminary numerical experim...

An affine scaling interior trust-region method combining with nonmonotone line search filter technique for linear inequality constrained minimization

This paper proposes and analyzes an affine scaling trust-region method with line search filter technique for solving nonlinear optimization problems subject to bounds on variables. At the current iteration, the trial step is generated by the general trust-region subproblem which is defined by minimizing a quadratic function subject only to an affine scaling ellipsoidal constraint. Both trust-region strategy and line search filter technique will switch to trail backtracking step which is strictly feasible. Meanwhile, the proposed method does not depend on any external restoration procedure used in line search filter technique. A new backtracking relevance condition is given which is weaker than the switching condition to obtain the global convergence of the algorithm. The global convergence and fast local convergence rate of this algorithm are established under reasonable assumptions. Preliminary numerical results are reported indicating the practical viability and show the effectiveness of the proposed algorithm.

Dan Li

Papers

An affine scaling interior trust-region method combining with nonmonotone line search filter technique for linear inequality constrained minimization

An affine scaling interior trust-region method combining with line search filter technique for optimization subject to bounds on variables