“Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告

The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.

Analyzing and Improving the Image Quality of StyleGAN

/pdf/analyzing-and-improving-the-image-quality-of-stylegan-15vb065a0b.pdf

Uncertainty quantification (UQ) plays a pivotal role in reduction of uncertainties during both optimization and decision making processes. It can be applied to solve a variety of real-world applications in science and engineering. Bayesian approximation and ensemble learning techniques are two most widely-used UQ methods in the literature. In this regard, researchers have proposed different UQ methods and examined their performance in a variety of applications such as computer vision (e.g., self-driving cars and object detection), image processing (e.g., image restoration), medical image analysis (e.g., medical image classification and segmentation), natural language processing (e.g., text classification, social media texts and recidivism risk-scoring), bioinformatics, etc. This study reviews recent advances in UQ methods used in deep learning. Moreover, we also investigate the application of these methods in reinforcement learning (RL). Then, we outline a few important applications of UQ methods. Finally, we briefly highlight the fundamental research challenges faced by UQ methods and discuss the future research directions in this field.

A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges

DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection

Deep Neural Networks (DNNs) have gained considerable attention in the past decades due to their astounding performance in different applications, such as natural language modeling, self-driving assistance, and source code understanding. With rapid exploration, more and more complex DNN architectures have been proposed along with huge pre-trained model parameters. A common way to use such DNN models in user-friendly devices (e.g., mobile phones) is to perform model compression before deployment. However, recent research has demonstrated that model compression, e.g., model quantization, yields accuracy degradation as well as output disagreements when tested on unseen data. Since the unseen data always include distribution shifts and often appear in the wild, the quality and reliability of models after quantization are not ensured. In this paper, we conduct a comprehensive study to characterize and help users understand the behaviors of quantization models. Our study considers four datasets spanning from image to text, eight DNN architectures including both feed-forward neural networks and recurrent neural networks, and 42 shifted sets with both synthetic and natural distribution shifts. The results reveal that 1) data with distribution shifts lead to more disagreements than without. 2) Quantization-aware training can produce more stable models than standard, adversarial, and Mixup training. 3) Disagreements often have closer top-1 and top-2 output probabilities, and Margin is a better indicator than other uncertainty metrics to distinguish disagreements. 4) Retraining the model with disagreements has limited efficiency in removing disagreements. We release our code and models as a new benchmark for further study of model quantization.

Towards Understanding Model Quantization for Reliable Deep Neural Network Deployment

Testing Autonomous Driving Systems (ADSs) is a critical task for ensuring the reliability and safety of autonomous vehicles. Existing methods mainly focus on searching for safety violations while the diversity of the generated test cases is ignored, which may generate many redundant test cases and failures. Such redundant failures can reduce testing performance and increase failure analysis costs. In this paper, we present a novel behavior-guided fuzzing technique (BehAVExplor) to explore the different behaviors of the ego vehi- cle (i.e., the vehicle controlled by the ADS under test) and detect diverse violations. Specifically, we design an efficient unsupervised model, called BehaviorMiner, to characterize the behavior of the ego vehicle. BehaviorMiner extracts the temporal features from the given scenarios and performs a clustering-based abstraction to group behaviors with similar features into abstract states. A new test case will be added to the seed corpus if it triggers new behav- iors (e.g., cover new abstract states). Due to the potential conflict between the behavior diversity and the general violation feedback, we further propose an energy mechanism to guide the seed selec- tion and the mutation. The energy of a seed quantifies how good it is. We evaluated BehAVExplor on Apollo, an industrial-level ADS, and LGSVL simulation environment. Empirical evaluation results show that BehAVExplor can effectively find more diverse violations than the state-of-the-art.

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

Continuous game quality assurance is of great importance to satisfy the increasing demands of users. To respond to game issues reported by users timely, game com-panies often create and maintain a large number of releases, updates, and tweaks in a short time. Regression testing is an essential technique adopted to detect regression issues during the evolution of the game software. However, due to the special characteristics of game software (e.g., frequent updates and long-running tests), traditional regression testing techniques are not directly applicable. To bridge this gap, in this paper, we perform an early exploratory study to investigate the challenges in regression testing of video games. We first performed empirical studies to better understand the game development process, bugs introduced during game evolution, and the context sensitivity. Based on the results of the study, we proposed the first regression test selection (RTS) technique for game software, which is a compromise between safety and practicality. In particular, we model the test suite of game software as a State Transition Graph (STG) and then perform the RTS on the STG. We establish the dependencies between the states/actions of STG and game files, including game art resources, game design files, and source code, and perform change impact analysis to identify the states/actions (in the STG) that potentially execute such changes. We implemented our framework in a tool, named GameRTS, and evaluated its usefulness on 10 tasks of a large-scale commercial game, including a total of 1,429 commits over three versions. The experimental results demonstrate the usefulness and effectiveness of GameRTS in game RTS. For most tasks, GameRTS only selected one trace from STG, which can significantly reduce the testing time. Furthermore, GameRTS detects all the regression bugs from the test evaluation suites. Compared with the file-level RTS, GameRTS selected fewer states/actions/traces (i.e., 13.77%, 23.97%, 6.85%). In addition, GameRTS identified 2 new critical regression bugs in the game.

GameRTS: A Regression Testing Framework for Video Games

The fast advances in mobile hardware and widespread smartphone usage have fueled the growth of global mobile gaming in the past decade. As a result, the need for quality assurance of mobile gaming has become increasingly pressing. While general-purpose testing methods have been developed for mobile applications, they become struggling when being applied to mobile games due to the unique characteristics of mobile games, such as dynamic loading and stunning visual effects. There comes a growing industrial demand for automated testing techniques with high compatibility (compatible with various resolutions, and platforms) and non-intrusive characteristics (without packaging external modules into the source code, e.g., POCO). To fulfill these demands, in this paper, we introduce our experience in adopting the widget detection-based testing technique WDTEST, for mobile games at NetEase Games. To this end, we have constructed by far the largest graphical user interface (GUI) dataset for mobile games and conducted comprehensive evaluations on the performance of state-of-the-art widget detection techniques in the context of mobile gaming.We leverage widget detection techniques to develop WDTEST, which performs automated testing using only screenshots as input. Our evaluation shows that WDTEST outperforms the widely used tool Monkey in achieving three times more coverage of unique UI in gaming scenarios. Our further experiments demonstrate that WDTEST can be applied to general mobile applications without additional fine-tuning. Furthermore, we conducted a thorough survey at NetEase Games to gain a comprehensive understanding of widget detection-based testing techniques and identify challenges in industrial mobile game testing. The results show that testers are overall satisfied with the compatibility testing aspect of widget detection-based testing, but not much with functionality testing. This survey also highlights several unique characteristics of mobile games, providing valuable insights for future research directions.

Widget Detection-based Testing for Industrial Mobile Games

Deep learning (DL) plays a more and more important role in our daily life due to its competitive performance in industrial application domains. As the core of DL-enabled systems, deep neural networks (DNNs) need to be carefully evaluated to ensure the produced models match the expected requirements. In practice, the de facto standard to assess the quality of DNNs in the industry is to check their performance (accuracy) on a collected set of labeled test data. However, preparing such labeled data is often not easy partly because of the huge labeling effort, i.e., data labeling is labor-intensive, especially with the massive new incoming unlabeled data every day. Recent studies show that test selection for DNN is a promising direction that tackles this issue by selecting minimal representative data to label and using these data to assess the model. However, it still requires human effort and cannot be automatic. In this paper, we propose a novel technique, named Aries, that can estimate the performance of DNNs on new unlabeled data using only the information obtained from the original test data. The key insight behind our technique is that the model should have similar prediction accuracy on the data which have similar distances to the decision boundary. We performed a large-scale evaluation of our technique on two famous datasets, CIFAR-10 and Tiny-ImageNet, four widely studied DNN models including ResNetl0l and DenseNetl21, and 13 types of data transformation methods. Results show that the estimated accuracy by Aries is only 0.03% - 2.60% off the true accuracy. Besides, Aries also outperforms the state-of-the-art labeling-free methods in 50 out of 52 cases and selection-labeling-based methods in 96 out of 128 cases.

Xiaofei Xie

Papers

Towards Understanding Model Quantization for Reliable Deep Neural Network Deployment

BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

GameRTS: A Regression Testing Framework for Video Games

Widget Detection-based Testing for Industrial Mobile Games

Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation