General Topology-I

Deep learning is at the heart of the current rise of artificial intelligence. In the field of computer vision, it has become the workhorse for applications ranging from self-driving cars to surveillance and security. Whereas, deep neural networks have demonstrated phenomenal success (often beyond human capabilities) in solving complex problems, recent studies show that they are vulnerable to adversarial attacks in the form of subtle perturbations to inputs that lead a model to predict incorrect outputs. For images, such perturbations are often too small to be perceptible, yet they completely fool the deep learning models. Adversarial attacks pose a serious threat to the success of deep learning in practice. This fact has recently led to a large influx of contributions in this direction. This paper presents the first comprehensive survey on adversarial attacks on deep learning in computer vision. We review the works that design adversarial attacks, analyze the existence of such attacks and propose defenses against them. To emphasize that adversarial attacks are possible in practical conditions, we separately review the contributions that evaluate adversarial attacks in the real-world scenarios. Finally, drawing on the reviewed literature, we provide a broader outlook of this research direction.

Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey

Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

/pdf/synthetic-and-natural-noise-both-break-neural-machine-41z8rg2kva.pdf

Synthetic and Natural Noise Both Break Neural Machine Translation

Machine Learning (ML) models are applied in a variety of tasks such as network intrusion detection or Malware classification. Yet, these models are vulnerable to a class of malicious inputs known as adversarial examples. These are slightly perturbed inputs that are classified incorrectly by the ML model. The mitigation of these adversarial inputs remains an open problem. As a step towards understanding adversarial examples, we show that they are not drawn from the same distribution than the original data, and can thus be detected using statistical tests. Using thus knowledge, we introduce a complimentary approach to identify specific inputs that are adversarial. Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs. We evaluate our approach on multiple adversarial example crafting methods (including the fast gradient sign and saliency map methods) with several datasets. The statistical test flags sample sets containing adversarial inputs confidently at sample sizes between 10 and 100 data points. Furthermore, our augmented model either detects adversarial examples as outliers with high accuracy (> 80%) or increases the adversary's cost - the perturbation added - by more than 150%. In this way, we show that statistical properties of adversarial examples are essential to their detection.

On the (Statistical) Detection of Adversarial Examples

We address the problem of adversarial attacks on text classification, which is rarely studied comparing to attacks on image classification. The challenge of this task is to generate adversarial examples that maintain lexical correctness, grammatical correctness and semantic similarity. Based on the synonyms substitution strategy, we introduce a new word replacement order determined by both the word saliency and the classification probability, and propose a greedy algorithm called probability weighted word saliency (PWWS) for text adversarial attack. Experiments on three popular datasets using convolutional as well as LSTM models show that PWWS reduces the classification accuracy to the most extent, and keeps a very low word substitution rate. A human evaluation study shows that our generated adversarial examples maintain the semantic similarity well and are hard for humans to perceive. Performing adversarial training using our perturbed datasets improves the robustness of the models. At last, our method also exhibits a good transferability on the generated adversarial examples.

/pdf/generating-natural-language-adversarial-examples-through-1ftt7y125j.pdf

Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency.

Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to a black-box attack, which is a more realistic scenario. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We develop novel scoring strategies to find the most important words to modify such that the deep classifier makes a wrong prediction. Simple character-level transformations are applied to the highest-ranked words in order to minimize the edit distance of the perturbation. We evaluated DeepWordBug on two real-world text datasets: Enron spam emails and IMDB movie reviews. Our experimental results indicate that DeepWordBug can reduce the classification accuracy from 99% to 40% on Enron and from 87% to 26% on IMDB. Our results strongly demonstrate that the generated adversarial sequences from a deep-learning model can similarly evade other deep models.

/pdf/black-box-generation-of-adversarial-text-sequences-to-evade-4fdr7angdo.pdf

Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Recent studies have shown that deep neural networks (DNN) are vulnerable to adversarial samples: maliciously-perturbed samples crafted to yield incorrect model outputs. Such attacks can severely undermine DNN systems, particularly in security-sensitive settings. It was observed that an adversary could easily generate adversarial samples by making a small perturbation on irrelevant feature dimensions that are unnecessary for the current classification task. To overcome this problem, we introduce a defensive mechanism called DeepCloak. By identifying and removing unnecessary features in a DNN model, DeepCloak limits the capacity an attacker can use generating adversarial samples and therefore increase the robustness against such inputs. Comparing with other defensive approaches, DeepCloak is easy to implement and computationally efficient. Experimental results show that DeepCloak can increase the performance of state-of-the-art DNN models against adversarial samples.

/pdf/deepcloak-masking-deep-neural-network-models-for-robustness-uo95xpsgmz.pdf

DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples

Most machine learning classifiers, including deep neural networks, are vulnerable to adversarial examples. Such inputs are typically generated by adding small but purposeful modifications that lead to incorrect outputs while imperceptible to human eyes. The goal of this paper is not to introduce a single method, but to make theoretical steps towards fully understanding adversarial examples. By using concepts from topology, our theoretical analysis brings forth the key reasons why an adversarial example can fool a classifier ($f_1$) and adds its oracle ($f_2$, like human eyes) in such analysis. By investigating the topological relationship between two (pseudo)metric spaces corresponding to predictor $f_1$ and oracle $f_2$, we develop necessary and sufficient conditions that can determine if $f_1$ is always robust (strong-robust) against adversarial examples according to $f_2$. Interestingly our theorems indicate that just one unnecessary feature can make $f_1$ not strong-robust, and the right feature representation learning is the key to getting a classifier that is both accurate and strong-robust.

A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Examples

Adversarial samples are maliciously created inputs that lead a learning-based classifier to produce incorrect output labels. An adversarial sample is often generated by adding adversarial perturbation (AP) to a normal test sample. Recent studies that tried to analyze classifiers under such AP are mostly empirical and provide little understanding of why. To fill this gap, we propose a theoretical framework for analyzing learning-based classifiers, especially deep neural networks (DNN) in the face of such AP. By using concepts from topology, this framework brings forth the key reasons why an adversarial can fool a classifier ($f_1$) and suggests a new focus on its oracle ($f_2$, like human annotators of that specific task). By investigating the topology relationship between two (pseudo)metric spaces corresponding to predictor $f_1$ and oracle $f_2$, we develop several ideal conditions that can determine if $f_1$ is always robust (strong-robust) against adversarial samples according to its $f_2$. The theoretical framework leads to a set of novel and complementary insights that have not been uncovered by the literature. Surprisingly our theorems find that just one extra irrelevant feature can make a classifier not strong-robust, and the right feature representation learning is the key to getting a classifier that is both accurate and strong robust. Empirically we find that "Siamese architecture" can be used to help DNN models get close to the desired topological relationship for strong-robustness, which in turn effectively improves its performance against AP.

/pdf/a-theoretical-framework-for-robustness-of-deep-classifiers-21u9dpfnxl.pdf

Ji Gao

Papers

Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples

DeepCloak: Masking Deep Neural Network Models for Robustness Against Adversarial Samples

A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Examples

A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Samples