Posted Content•
Neural Architecture Search with Reinforcement Learning
TL;DR: This paper uses a recurrent network to generate the model descriptions of neural networks and trains this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set.
Abstract: Neural networks are powerful and flexible models that work well for many difficult learning tasks in image, speech and natural language understanding. Despite their success, neural networks are still hard to design. In this paper, we use a recurrent network to generate the model descriptions of neural networks and train this RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. On the CIFAR-10 dataset, our method, starting from scratch, can design a novel network architecture that rivals the best human-invented architecture in terms of test set accuracy. Our CIFAR-10 model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the previous state-of-the-art model that used a similar architectural scheme. On the Penn Treebank dataset, our model can compose a novel recurrent cell that outperforms the widely-used LSTM cell, and other state-of-the-art baselines. Our cell achieves a test set perplexity of 62.4 on the Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model. The cell can also be transferred to the character language modeling task on PTB and achieves a state-of-the-art perplexity of 1.214.
Citations
More filters
25 Jul 2020
TL;DR: A system predicting hazard areas on the basis of confirmed infection cases with location information and achieving promising overall performance in terms of precision, recall, accuracy, F1 score, and AUC is built.
Abstract: Dating back from late December 2019, the Chinese city of Wuhan has reported an outbreak of atypical pneumonia, now known as lung inflammation caused by novel coronavirus (COVID-19). Cases have spread to other cities in China and more than 180 countries and regions internationally. World Health Organization (WHO) officially declares the coronavirus outbreak a pandemic and the public health emergency is perhaps one of the top concerns in the year of 2020 for governments all over the world. Till today, the coronavirus outbreak is still raging and has no sign of being under control in many countries. In this paper, we aim at drawing lessons from the COVID-19 outbreak process in China and using the experiences to help the interventions against the coronavirus wherever in need. To this end, we have built a system predicting hazard areas on the basis of confirmed infection cases with location information. The purpose is to warn people to avoid of such hot zones and reduce risks of disease transmission through droplets or contacts. We analyze the data from the daily official information release which are publicly accessible. Based on standard classification frameworks with reinforcements incrementally learned day after day, we manage to conduct thorough feature engineering from empirical studies, including geographical, demographic, temporal, statistical, and epidemiological features. Compared with heuristics baselines, our method has achieved promising overall performance in terms of precision, recall, accuracy, F1 score, and AUC. We expect that our efforts could be of help in the battle against the virus, the common opponent of human kind.
8 citations
Cites methods from "Neural Architecture Search with Rei..."
...A natural solution is that we utilize the reinforcement learning based on policy gradient to give feedback to the hyperparameter generator, which is inspired from Neural Architecture Search (NAS) [32]....
[...]
TL;DR: In this paper, a survey of mobile image processing and computer vision applications is presented, highlighting these constraints and explaining how the algorithms have been modified/adapted to meet accuracy and performance demands.
Abstract: Image processing and computer vision on mobile devices have a wide range of applications such as digital image enhancement and augmented reality. While images acquired by cameras on mobile devices can be processed with generic image processing algorithms, there are numerous constraints and external issues that call for customized algorithms for such devices. In this paper, we survey mobile image processing and computer vision applications while highlighting these constraints and explaining how the algorithms have been modified/adapted to meet accuracy and performance demands. We hope that this paper will be a useful resource for researchers who intend to apply image processing and computer vision algorithms to real-world scenarios and applications that involve mobile devices.
8 citations
09 Dec 2020
TL;DR: In this article, a light-weight object detector that outputs a depth and a color image from a stereo camera is presented. But the network architecture of YOLOv3 is extended to 3D in the middle, and intersection over union (IoU) is introduced to confirm the accuracy of region extraction.
Abstract: This paper aims at constructing a light-weight object detector that inputs a depth and a color image from a stereo camera. Specifically, by extending the network architecture of YOLOv3 to 3D in the middle, it is possible to output in the depth direction. In addition, Intersection over Union (IoU) in 3D space is introduced to confirm the accuracy of region extraction results. In the field of deep learning, object detectors that use distance information as input are actively studied for utilizing automated driving. However, the conventional detector has a large network structure, and the real-time property is impaired. The effectiveness of the detector constructed as described above is verified using datasets. The experiment verified that the proposed model is able to output 3D bounding boxes and detect people whose body is partly hidden. Further, the processing speed of the model reached 44.35 fps.
8 citations
TL;DR: In this article, a new search space is designed for feature pyramids in object detectors, which is formulated as a combinatorial optimization problem and solved by a Simulated Annealing-based Network Architecture Search method (SA-NAS).
Abstract: Feature pyramids have delivered significant improvement in object detection. However, building effective feature pyramids heavily relies on expert knowledge, and also requires strenuous efforts to balance effectiveness and efficiency. Automatic search methods, such as NAS-FPN, automates the design of feature pyramids, but the low search efficiency makes it difficult to apply in a large search space. In this paper, we propose a novel search framework for a feature pyramid network, called AutoDet, which enables to automatic discovery of informative connections between multi-scale features and configure detection architectures with both high efficiency and state-of-the-art performance. In AutoDet, a new search space is specifically designed for feature pyramids in object detectors, which is more general than NAS-FPN. Furthermore, the architecture search process is formulated as a combinatorial optimization problem and solved by a Simulated Annealing-based Network Architecture Search method (SA-NAS). Compared with existing NAS methods, AutoDet ensures a dramatic reduction in search times. For example, our SA-NAS can be up to 30x faster than reinforcement learning-based approaches. Furthermore, AutoDet is compatible with both one-stage and two-stage structures with all kinds of backbone networks. We demonstrate the effectiveness of AutoDet with outperforming single-model results on the COCO dataset. Without pre-training on OpenImages, AutoDet with the ResNet-101 backbone achieves an AP of 39.7 and 47.3 for one-stage and two-stage architectures, respectively, which surpass current state-of-the-art methods.
8 citations
TL;DR: In this paper, a one-shot neural architecture search (NAS) model is proposed for industrial fault diagnosis, where the supernet is trained to evaluate the actual performance of candidate networks by measuring the difference between its output probability and the true labels.
Abstract: Machine learning method has been widely applied in industrial fault diagnosis, especially the deep learning method. In the field of industrial fault diagnosis, deep learning is mostly used to extract features of vibration signals to achieve end-to-end fault diagnosis systems. Due to the complexity and variety of actual industrial datasets, some deep learning models are designed to be complicated. However, designing neural network architectures requires rich professional knowledge, experience, and a large number of experiments, increasing the difficulty of developing deep learning models. Fortunately, Neural Architecture Search (NAS), a branch of Automated Machine Learning (AutoML), is developing rapidly. Given a search space, NAS can search for networks that perform better than manually designed. In this paper, a one-shot NAS method for fault diagnosis is proposed. The one-shot model is a supernet that contains all candidate networks in a given search space. The supernet is trained to evaluate the actual performance of candidate networks by measuring the difference between its output probability and the true labels. According to the prediction of supernet, the networks with excellent performance can be searched, using some common search methods such as random search or evolutionary algorithm. Finally, the searched network is trained by reusing the weights of the supernet. To evaluate the proposed method, two search spaces are designed, ResNet and Inception search spaces, to search on PHM 2009 Data Challenge gearbox dataset. The state-of-the-art results are obtained, and accuracies of searched ResNet-A and Inception-A are 84.11% and 83.81%, which are 3.29% and 10.88% higher than Reinforcement Learning based NAS.
8 citations
References
More filters
27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
123,388 citations
Proceedings Article•
01 Jan 2015TL;DR: This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Abstract: We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.
111,197 citations
Proceedings Article•
04 Sep 2014TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.
55,235 citations
"Neural Architecture Search with Rei..." refers methods in this paper
...Along with this success is a paradigm shift from feature designing to architecture designing, i.e., from SIFT (Lowe, 1999), and HOG (Dalal & Triggs, 2005), to AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), and ResNet (He et al., 2016a)....
[...]
01 Jan 1998
TL;DR: In this article, a graph transformer network (GTN) is proposed for handwritten character recognition, which can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters.
Abstract: Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day.
42,067 citations
20 Jun 2005
TL;DR: It is shown experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection, and the influence of each stage of the computation on performance is studied.
Abstract: We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds.
31,952 citations
"Neural Architecture Search with Rei..." refers methods in this paper
...Along with this success is a paradigm shift from feature designing to architecture designing, i.e., from SIFT (Lowe, 1999), and HOG (Dalal & Triggs, 2005), to AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan & Zisserman, 2014), GoogleNet (Szegedy et al., 2015), and ResNet (He et al., 2016a)....
[...]