Proceedings ArticleDOI
Adaptive Lightweight Regularization Tool for Complex Analytics
Zhaojing Luo,Shaofeng Cai,Jinyang Gao,Meihui Zhang,Kee Yuan Ngiam,Gang Chen,Wang-Chien Lee +6 more
- pp 485-496
Reads0
Chats0
TLDR
This paper proposes a general adaptive regularization method based on Gaussian Mixture to learn the best regularization function according to the observed parameters, and develops an effective update algorithm which integrates Expectation Maximization with Stochastic Gradient Descent.Abstract:
Deep Learning and Machine Learning models have recently been shown to be effective in many real world applications. While these models achieve increasingly better predictive performance, their structures have also become much more complex. A common and difficult problem for complex models is overfitting. Regularization is used to penalize the complexity of the model in order to avoid overfitting. However, in most learning frameworks, regularization function is usually set as some hyper parameters, and therefore the best setting is difficult to find. In this paper, we propose an adaptive regularization method, as part of a large end-to-end healthcare data analytics software stack, which effectively addresses the above difficulty. First, we propose a general adaptive regularization method based on Gaussian Mixture (GM) to learn the best regularization function according to the observed parameters. Second, we develop an effective update algorithm which integrates Expectation Maximization (EM) with Stochastic Gradient Descent (SGD). Third, we design a lazy update algorithm to reduce the computational cost by 4x. The overall regularization framework is fast, adaptive and easy-to-use. We validate the effectiveness of our regularization method through an extensive experimental study over 13 standard benchmark datasets and three kinds of deep learning/machine learning models. The results illustrate that our proposed adaptive regularization method achieves significant improvement over state-of-the-art regularization methods.read more
Citations
More filters
Elastic-net regularization in learning theory
TL;DR: Zou and Hastie as discussed by the authors proposed an elastic-net regularization scheme for random-design regression, where the response variable is vector-valued and the prediction functions are linear combinations of elements (features) in an infinite-dimensional dictionary.
Posted ContentDOI
SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis
Naili Xing,Sai Ho Yeung,Cheng-Hao Cai,Teck Khim Ng,Wei Wang,Kaiyuan Yang,Nan Yang,Meihui Zhang,Gang Chen,Beng Chin Ooi +9 more
TL;DR: SINGA-Easy as discussed by the authors is a new deep learning framework that provides distributed hyper-parameter tuning at the training stage, dynamic computational cost control at the inference stage, and intuitive user interactions with multimedia contents facilitated by model explanation.
Proceedings ArticleDOI
Overfitting and Underfitting Analysis for Deep Learning Based End-to-end Communication Systems
TL;DR: In considered DL-based communication systems, it is proposed to use the average, variance and minimum of transmitted signals' minimum Euclidean distance to estimate the effects of underfitting and overfitting on the error rate performances in terms of the energy per bit to noise power spectral ratio Eb/ No of signals.
Journal ArticleDOI
Database Meets Deep Learning: Challenges and Opportunities
TL;DR: In this article, the authors discuss possible improvements for deep learning systems from a database perspective, and analyze database applications that may benefit from deep learning techniques, and discuss research problems at the intersection of the two fields.
Proceedings ArticleDOI
MLCask: Efficient Management of Component Evolution in Collaborative Data Analytics Pipelines
Zhaojing Luo,Sai Ho Yeung,Meihui Zhang,Kaiping Zheng,Lei Zhu,Gang Chen,Feiyi Fan,Qian Lin,Kee Yuan Ngiam,Beng Chin Ooi +9 more
TL;DR: In this paper, the authors identify two main challenges that arise during the deployment of machine learning pipelines, and address them with the design of versioning for an end-to-end analytics system MLCask.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
Journal ArticleDOI
Regression Shrinkage and Selection via the Lasso
TL;DR: A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.