scispace - formally typeset
Search or ask a question
Author

Muhammad Zubair Rehman

Bio: Muhammad Zubair Rehman is an academic researcher from Universiti Tun Hussein Onn Malaysia. The author has contributed to research in topics: Backpropagation & Metaheuristic. The author has an hindex of 6, co-authored 20 publications receiving 140 citations. Previous affiliations of Muhammad Zubair Rehman include Information Technology University & Sohar University.

Papers
More filters
Journal ArticleDOI
30 Apr 2012
TL;DR: This research proposed an algorithm for improving the current working performance of Back-propagation algorithm by adaptively changing the momentum value and at the same time keeping the ‘gain’ parameter fixed for all nodes in the neural network.
Abstract: The traditional Gradient Descent Back-propagation Neural Network Algorithm is widely used in solving many practical applications around the globe. Despite providing successful solutions, it possesses a problem of slow convergence and sometimes getting stuck at local minima. Several modifications are suggested to improve the convergence rate of Gradient Descent Backpropagation algorithm such as careful selection of initial weights and biases, learning rate, momentum, network topology, activation function and ‘gain’ value in the activation function. In a certain variation, the previous researchers demonstrated that in “feed-forward algorithm”, the slope of activation function is directly influenced by ‘gain’ parameter. This research proposed an algorithm for improving the current working performance of Back-propagation algorithm by adaptively changing the momentum value and at the same time keeping the ‘gain’ parameter fixed for all nodes in the neural network. The performance of the proposed method known as ‘Gradient Descent Method with Adaptive Momentum (GDAM)’ is compared with the performances of ‘Gradient Descent Method with Adaptive Gain (GDM-AG)’ and ‘Gradient Descent with Simple Momentum (GDM)’. The learning rate is kept fixed while sigmoid activation function is used throughout the experiments. The efficiency of the proposed method is demonstrated by simulations on three classification problems. Results show that GDAM is far better than previous methods with an accuracy ratio of 1.0 for classification problems and can be used as an alternative approach of BPNN.

50 citations

Book ChapterDOI
01 Jan 2014
TL;DR: This paper investigates the use of Bat algorithm in combination with Back-propagation neural network (BPNN) algorithm to solve the local minima problem in gradient descent trajectory and to increase the convergence rate.
Abstract: Metaheuristic algorithm such as BAT algorithm is becoming a popular method in solving many hard optimization problems. This paper investigates the use of Bat algorithm in combination with Back-propagation neural network (BPNN) algorithm to solve the local minima problem in gradient descent trajectory and to increase the convergence rate. The performance of the proposed Bat based Back-Propagation (Bat-BP) algorithm is compared with Artificial Bee Colony using BPNN algorithm (ABC-BP) and simple BPNN algorithm. Specifically, OR and XOR datasets are used for training the network. The simulation results show that the computational efficiency of BPNN training process is highly enhanced when combined with BAT algorithm.

41 citations

Journal ArticleDOI
TL;DR: This current study focuses on proposing a new framework on using Gradient Descent Back Propagation Neural Network model with an improvement on the momentum value to identify the important factors that directly affect the hearing ability of industrial workers.
Abstract: Noise-Induced Hearing Loss (NIHL) has become a major source of health problem in industrial workers due to continuous exposure to high frequency sounds emitting from the machines. In the past, several studies have been carried-out to identify NIHL industrial workers. Unfortunately, these studies neglected some important factors that directly affect hearing ability in human. Artificial Neural Network (ANN) provides very effective way to predict hearing loss in humans. However, the training process for an ANN required the designers to arbitrarily select parameters such as network topology, initial weights and biases, learning rate value, the activation function, value for gain in activation function and momentum. An improper choice of any of these parameters can result in slow convergence or even network paralysis, where the training process comes to a standstill or get stuck at local minima. Therefore, this current study focuses on proposing a new framework on using Gradient Descent Back Propagation Neural Network model with an improvement on the momentum value to identify the important factors that directly affect the hearing ability of industrial workers. Results from the prediction will be used in determining the environmental health hazards which affect the workers health.

15 citations

Book ChapterDOI
01 Jan 2014
TL;DR: The proposed Cuckoo Search Recurrent Neural Network (CSRNN) algorithm performs better than other algorithms used in this study in terms of convergence rate and accuracy.
Abstract: Selecting the optimal topology of neural network for a particular application is a difficult task. In case of recurrent neural networks (RNN), most methods only introduce topologies in which their neurons are fully connected. However, recurrent neural network training algorithm has some drawbacks such as getting stuck in local minima, slow speed of convergence and network stagnancy. This paper propose an improved recurrent neural network trained with Cuckoo Search (CS) algorithm to achieve fast convergence and high accuracy. The performance of the proposed Cuckoo Search Recurrent Neural Network (CSRNN) algorithm is compared with Artificial Bee Colony (ABC) and similar hybrid variants. The simulation results show that the proposed CSRNN algorithm performs better than other algorithms used in this study in terms of convergence rate and accuracy.

14 citations

Journal ArticleDOI
TL;DR: This paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process to overcome the limitations of BP.
Abstract: Back Propagation (BP) is commonly used algorithm that optimize the performance of network for training multilayer feed-forward artificial neural networks. However, BP is inherently slow in learning and it sometimes gets trapped at local minima. These problems occur mailnly due to a constant and non-optimum learning rate (a fixed step size) in which the fixed value of learning rate is set to an initial starting value before training patterns for an input layer and an output layer. This fixed learning rate often leads the BP network towrds failure during steepest descent. Therefore to overcome the limitations of BP, this paper introduces an improvement to back propagation gradient descent with adapative learning rate (BPGD-AL) by changing the values of learning rate locally during the learning process. The simulation results on selected benchmark datasets show that the adaptive learning rate significantly improves the learning efficiency of the Back Propagation Algorithm

13 citations


Cited by
More filters
Journal ArticleDOI
TL;DR: Simulation results show that the computational efficiency of ANN training process is highly enhanced when coupled with different preprocessing techniques, particularly Min-Max, Z-Score and Decimal Scaling Normalization preprocessing technique.

163 citations

Journal ArticleDOI
03 Mar 2020
TL;DR: This paper explores the ability of ML to produce computationally efficient surrogate models of physical applications that circumvent the need for the more expensive simulation techniques entirely and how ML can be used to process large amounts of data.
Abstract: The re-kindled fascination in machine learning (ML), observed over the last few decades, has also percolated into natural sciences and engineering. ML algorithms are now used in scientific computing, as well as in data-mining and processing. In this paper, we provide a review of the state-of-the-art in ML for computational science and engineering. We discuss ways of using ML to speed up or improve the quality of simulation techniques such as computational fluid dynamics, molecular dynamics, and structural analysis. We explore the ability of ML to produce computationally efficient surrogate models of physical applications that circumvent the need for the more expensive simulation techniques entirely. We also discuss how ML can be used to process large amounts of data, using as examples many different scientific fields, such as engineering, medicine, astronomy and computing. Finally, we review how ML has been used to create more realistic and responsive virtual reality applications.

98 citations

Posted Content
TL;DR: This work revisits the momentum SGD algorithm and shows that hand-tuning a single learning rate and momentum makes it competitive with Adam, and designs YellowFin, an automatic tuner for momentum and learning rate in SGD.
Abstract: Hyperparameter tuning is one of the most time-consuming workloads in deep learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable. Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better test metrics. Motivated by this trend, we ask: can simple adaptive methods based on SGD perform as well or better? We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam. We then analyze its robustness to learning rate misspecification and objective curvature variation. Based on these insights, we design YellowFin, an automatic tuner for momentum and learning rate in SGD. YellowFin optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the fly. We empirically show that YellowFin can converge in fewer iterations than Adam on ResNets and LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to 3.28x in synchronous and up to 2.69x in asynchronous settings.

96 citations

Book ChapterDOI
24 Jun 2013
TL;DR: The simulation results show that the computational efficiency of BP training process is highly enhanced when coupled with the proposed hybrid method and the performance of the proposed Cuckoo Search Back-Propagation (CSBP) is compared with artificial bee colony using BP algorithm, and other hybrid variants.
Abstract: Back-propagation Neural Network (BPNN) algorithm is one of the most widely used and a popular technique to optimize the feed forward neural network training. Traditional BP algorithm has some drawbacks, such as getting stuck easily in local minima and slow speed of convergence. Nature inspired meta-heuristic algorithms provide derivative-free solution to optimize complex problems. This paper proposed a new meta-heuristic search algorithm, called cuckoo search (CS), based on cuckoo bird's behavior to train BP in achieving fast convergence rate and to avoid local minima problem. The performance of the proposed Cuckoo Search Back-Propagation (CSBP) is compared with artificial bee colony using BP algorithm, and other hybrid variants. Specifically OR and XOR datasets are used. The simulation results show that the computational efficiency of BP training process is highly enhanced when coupled with the proposed hybrid method.

85 citations

Journal ArticleDOI
TL;DR: In this paper, a bioinspired metaheuristic is proposed to simulate how the coronavirus spreads and infects healthy people, where relevant terms such as re-infection probability, super-spreading rate or traveling rate are introduced in the model in order to simulate as accurately as possible the Coronavirus activity.
Abstract: A novel bioinspired metaheuristic is proposed in this work, simulating how the coronavirus spreads and infects healthy people. From an initial individual (the patient zero), the coronavirus infects new patients at known rates, creating new populations of infected people. Every individual can either die or infect and, afterwards, be sent to the recovered population. Relevant terms such as re-infection probability, super-spreading rate or traveling rate are introduced in the model in order to simulate as accurately as possible the coronavirus activity. The Coronavirus Optimization Algorithm has two major advantages compared to other similar strategies. First, the input parameters are already set according to the disease statistics, preventing researchers from initializing them with arbitrary values. Second, the approach has the ability of ending after several iterations, without setting this value either. Infected population initially grows at an exponential rate but after some iterations, when considering social isolation measures and the high number recovered and dead people, the number of infected people starts decreasing in subsequent iterations. Furthermore, a parallel multi-virus version is proposed in which several coronavirus strains evolve over time and explore wider search space areas in less iterations. Finally, the metaheuristic has been combined with deep learning models, in order to find optimal hyperparameters during the training phase. As application case, the problem of electricity load time series forecasting has been addressed, showing quite remarkable performance.

60 citations