Wangpeng An

Bio: Wangpeng An is an academic researcher from Tsinghua University. The author has contributed to research in topics: Artificial neural network & Sparse approximation. The author has an hindex of 8, co-authored 19 publications receiving 413 citations. Previous affiliations of Wangpeng An include Hong Kong Polytechnic University.

A PID Controller Approach for Stochastic Optimization of Deep Networks

Abstract: Deep neural networks have demonstrated their power in many computer vision applications. State-of-the-art deep architectures such as VGG, ResNet, and DenseNet are mostly optimized by the SGD-Momentum algorithm, which updates the weights by considering their past and current gradients. Nonetheless, SGD-Momentum suffers from the overshoot problem, which hinders the convergence of network training. Inspired by the prominent success of proportional-integral-derivative (PID) controller in automatic control, we propose a PID approach for accelerating deep network optimization. We first reveal the intrinsic connections between SGD-Momentum and PID based controller, then present the optimization algorithm which exploits the past, current, and change of gradients to update the network parameters. The proposed PID method reduces much the overshoot phenomena of SGD-Momentum, and it achieves up to 50% acceleration on popular deep network architectures with competitive accuracy, as verified by our experiments on the benchmark datasets including CIFAR10, CIFAR100, and Tiny-ImageNet.

Exponential decay sine wave learning rate for fast deep neural network training

Abstract: Most state-of-the-art results on image classification tasks were obtained by residual neural networks, which use stochastic gradient descent (SGD) with momentum for training. In most cases, the learning rate drops by a constant factor every pre-defined number of epochs. However, it is difficult and time-consuming to estimate how many epochs to drop the learning rate. To tackle this problem, cyclical learning rate is gaining popularity in gradient-based optimization to improve the convergence speed in accelerated gradient schemes. But cyclical learning rate scheme scans a broad range of learning rate, some of which are not suitable for deep neural network training. In this paper, we propose a simple yet effective exponential decay sine wave like learning rate technique for SGD to improve its convergence speed. In the training process, the learning rate would vary in sine wave way. While the maximum value of sine wave would decay exponentially along with training epochs. An ensemble of wide residual nets with our proposed learning scheme achieves 3.01% and 16.03% errors on CIFAR-10 and CIFAR-100 respectively. Furthermore, our proposed method uses far less number of epochs than most recent learning rate strategies, accelerating neural network training tremendously.

PifPaf: Composite Fields for Human Pose Estimation

Abstract: We propose a new bottom-up method for multi-person 2D human pose estimation that is particularly well suited for urban mobility such as self-driving cars and delivery robots. The new method, PifPaf, uses a Part Intensity Field (PIF) to localize body parts and a Part Association Field (PAF) to associate body parts with each other to form full human poses. Our method outperforms previous methods at low resolution and in crowded, cluttered and occluded scenes thanks to (i) our new composite field PAF encoding fine-grained information and (ii) the choice of Laplace loss for regressions which incorporates a notion of uncertainty. Our architecture is based on a fully convolutional, single-shot, box-free design. We perform on par with the existing state-of-the-art bottom-up method on the standard COCO keypoint task and produce state-of-the-art results on a modified COCO keypoint task for the transportation domain.