Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Frechet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium

Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications, where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. The complexity of graph data has imposed significant challenges on the existing machine learning algorithms. Recently, many studies on extending deep learning approaches for graph data have emerged. In this article, we provide a comprehensive overview of graph neural networks (GNNs) in data mining and machine learning fields. We propose a new taxonomy to divide the state-of-the-art GNNs into four categories, namely, recurrent GNNs, convolutional GNNs, graph autoencoders, and spatial–temporal GNNs. We further discuss the applications of GNNs across various domains and summarize the open-source codes, benchmark data sets, and model evaluation of GNNs. Finally, we propose potential research directions in this rapidly growing field.

/pdf/a-comprehensive-survey-on-graph-neural-networks-4lw9i4v183.pdf

A Comprehensive Survey on Graph Neural Networks

admission. This proportion could already be greater in some parts of the country and may increase if referrals of cases of self-poisoning increase faster than the facilities for their assessment and management. The provision of social work and psychiatric expertise in casualty departments may be one means of preventing unnecessary medical admissions without risk to the patients. Dr Blake's and Dr Bramble's figures do not demonstrate, however, that any advantage would attach to medical teams taking over assessment from psychiatrists except that, by implication, assessments would be completed sooner by staff working on the ward full time. What the figures actually suggest is that if assessment of overdoses were left to house doctors there would be an increase in admissions to psychiatric units (by 19°U), outpatients (by 5O°'), and referrals to social services (by 140o). So for house doctors to assess overdoses would provide no economy for the psychiatric or social services. The study does not tell us what the consequences would have been for the six patients who the psychiatrists would have admitted but to whom the house doctors would have offered outpatient appointments. E J SALTER

Vitamin D deficiency.

“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

The PASCAL Visual Object Classes Challenge

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

Improved Training of Wasserstein GANs

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only poor samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models with continuous generators. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

/pdf/improved-training-of-wasserstein-gans-cd7mwfs3if.pdf

Improved training of wasserstein GANs

Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64x64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.

PixelVAE: A Latent Variable Model for Natural Images

One major challenge in training Deep Neural Networks is preventing overfitting. Many techniques such as data augmentation and novel regularizers such as Dropout have been proposed to prevent overfitting without requiring a massive amount of training data. In this work, we propose a new regularizer called DeCov which leads to significantly reduced overfitting (as indicated by the difference between train and val performance), and better generalization. Our regularizer encourages diverse or non-redundant representations in Deep Neural Networks by minimizing the cross-covariance of hidden activations. This simple intuition has been explored in a number of past works but surprisingly has never been applied as a regularizer in supervised learning. Experiments across a range of datasets and network architectures show that this loss always reduces overfitting while almost always maintaining or increasing generalization performance and often improving performance over Dropout.

Reducing Overfitting in Deep Networks by Decorrelating Representations

Background. CVD is a principal cause of mortality and disability globally. Objective. To analyse the epidemiological data on CHD, strokes, and the associated risk factors among adult population in the Gulf countries. Methods. A systematic review of published articles between 1990 and 2014 was conducted. Results. The analysis included 62 relevant studies. The prevalence of CHD was reported to be 5.5% in Saudi Arabia. The annual incidence of strokes ranged from 27.6 to 57 per 100 000 in the Gulf countries with ischaemic stroke being the most common subtype and hypertension and diabetes being the most common risk factors among stroke and ACS patients. The prevalence of overweight and obesity ranged from 31.2% to 43.3% and 22% to 34.1% in males and from 28% to 34.3% and 26.1% to 44% in females, respectively. In males, the prevalence of hypertension and diabetes ranged from 26.0% to 50.7% and 9.3% to 46.8%, respectively; in females these ranged from 20.9% to 57.2% and 6% to 53.2%, respectively. The prevalence of inactivity was from 24.3% to 93.9% and 56.7% to 98.1% in males and females, respectively. Relatively more males (13.4% to 37.4%) than females (0.5% to 20.7%) were current smokers. Available data indicate poor dietary habits with high consumption of snacks, fatty foods, sugar, and fast food. Conclusion. Effective preventative strategies and education programs are crucial in the Gulf region to reduce the risk of CVD mortality and morbidity in the coming years.

/pdf/prevalence-of-cardiovascular-disease-and-associated-risk-2ac0itmis3.pdf

Faruk Ahmed

Papers

Improved Training of Wasserstein GANs

Improved training of wasserstein GANs

PixelVAE: A Latent Variable Model for Natural Images

Reducing Overfitting in Deep Networks by Decorrelating Representations

Prevalence of Cardiovascular Disease and Associated Risk Factors among Adult Population in the Gulf Region: A Systematic Review