Backdoor attack intends to embed hidden backdoor into deep neural networks (DNNs), such that the attacked model performs well on benign samples, whereas its prediction will be maliciously changed if the hidden backdoor is activated by the attacker-defined trigger. This threat could happen when the training process is not fully controlled, such as training on third-party datasets or adopting third-party models, which poses a new and realistic threat. Although backdoor learning is an emerging and rapidly growing research area, its systematic review, however, remains blank. In this paper, we present the first comprehensive survey of this realm. We summarize and categorize existing backdoor attacks and defenses based on their characteristics, and provide a unified framework for analyzing poisoning-based backdoor attacks. Besides, we also analyze the relation between backdoor attacks and relevant fields ($i.e.,$ adversarial attacks and data poisoning), and summarize widely adopted benchmark datasets. Finally, we briefly outline certain future research directions relying upon reviewed works. A curated list of backdoor-related resources is also available at \url{this https URL}.

Backdoor Learning: A Survey

We introduce a new sublinear space data structure--the count-min sketch--for summarizing data streams. Our sketch allows fundamental queries in data stream summarization such as point, range, and inner product queries to be approximately answered very quickly; in addition, it can be applied to solve several important problems in data streams such as finding quantiles, frequent items, etc. The time and space bounds we show for using the CM sketch to solve these problems significantly improve those previously known--typically from 1/e2 to 1/e in factor.

An improved data stream summary: The Count-Min Sketch and its applications

Backdoor learning is an emerging and vital topic for studying deep neural networks' vulnerability (DNNs). Many pioneering backdoor attack and defense methods are being proposed, successively or concurrently, in the status of a rapid arms race. However, we find that the evaluations of new methods are often unthorough to verify their claims and accurate performance, mainly due to the rapid development, diverse settings, and the difficulties of implementation and reproducibility. Without thorough evaluations and comparisons, it is not easy to track the current progress and design the future development roadmap of the literature. To alleviate this dilemma, we build a comprehensive benchmark of backdoor learning called BackdoorBench. It consists of an extensible modular-based codebase (currently including implementations of 8 state-of-the-art (SOTA) attacks and 9 SOTA defense algorithms) and a standardized protocol of complete backdoor learning. We also provide comprehensive evaluations of every pair of 8 attacks against 9 defenses, with 5 poisoning ratios, based on 5 models and 4 datasets, thus 8,000 pairs of evaluations in total. We present abundant analysis from different perspectives about these 8,000 evaluations, studying the effects of different factors in backdoor learning. All codes and evaluations of BackdoorBench are publicly available at \url{https://backdoorbench.github.io}.

BackdoorBench: A Comprehensive Benchmark of Backdoor Learning

Visual object tracking (VOT) has been widely adopted in mission-critical applications, such as autonomous driving and intelligent surveillance systems. In current practice, third-party resources such as datasets, backbone networks, and training platforms are frequently used to train high-performance VOT models. Whilst these resources bring certain convenience, they also introduce new security threats into VOT models. In this paper, we reveal such a threat where an adversary can easily implant hidden backdoors into VOT models by tempering with the training process. Specifically, we propose a simple yet effective few-shot backdoor attack (FSBA) that optimizes two losses alternately: 1) a \emph{feature loss} defined in the hidden feature space, and 2) the standard \emph{tracking loss}. We show that, once the backdoor is embedded into the target model by our FSBA, it can trick the model to lose track of specific objects even when the \emph{trigger} only appears in one or a few frames. We examine our attack in both digital and physical-world settings and show that it can significantly degrade the performance of state-of-the-art VOT trackers. We also show that our attack is resistant to potential defenses, highlighting the vulnerability of VOT models to potential backdoor attacks.

Few-Shot Backdoor Attacks on Visual Object Tracking

The success of machine learning is fueled by the increasing availability of computing power and large training datasets. The training data is used to learn new models or update existing ones, assuming that it is sufficiently representative of the data that will be encountered at test time. This assumption is challenged by the threat of poisoning, an attack that manipulates the training data to compromise the model’s performance at test time. Although poisoning has been acknowledged as a relevant threat in industry applications, and a variety of different attacks and defenses have been proposed so far, a complete systematization and critical review of the field is still missing. In this survey, we provide a comprehensive systematization of poisoning attacks and defenses in machine learning, reviewing more than 100 papers published in the field in the last 15 years. We start by categorizing the current threat models and attacks, and then organize existing defenses accordingly. While we focus mostly on computer-vision applications, we argue that our systematization also encompasses state-of-the-art attacks and defenses for other data modalities. Finally, we discuss existing resources for research in poisoning, and shed light on the current limitations and open research questions in this research field.

https://dl.acm.org/doi/pdf/10.1145/3585385

Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning

Recent studies have revealed that deep neural networks (DNNs) are vulnerable to backdoor attacks, where attackers embed hidden backdoors in the DNN model by poisoning a few training samples. The attacked model behaves normally on benign samples, whereas its prediction will be maliciously changed when the backdoor is activated. We reveal that poisoned samples tend to cluster together in the feature space of the attacked DNN model, which is mostly due to the end-to-end supervised training paradigm. Inspired by this observation, we propose a novel backdoor defense via decoupling the original end-to-end training process into three stages. Specifically, we first learn the backbone of a DNN model via \emph{self-supervised learning} based on training samples without their labels. The learned backbone will map samples with the same ground-truth label to similar locations in the feature space. Then, we freeze the parameters of the learned backbone and train the remaining fully connected layers via standard training with all (labeled) training samples. Lastly, to further alleviate side-effects of poisoned samples in the second stage, we remove labels of some `low-credible' samples determined based on the learned model and conduct a \emph{semi-supervised fine-tuning} of the whole model. Extensive experiments on multiple benchmark datasets and DNN models verify that the proposed defense is effective in reducing backdoor threats while preserving high accuracy in predicting benign samples. Our code is available at \url{https://github.com/SCLBD/DBD}.

Backdoor Defense via Decoupling the Training Process

In Central Differential Privacy (CDP), there is a trusted analyst who collects the data from users and publishes the datasets/statistics after the procedure of random perturbation. However, in this paradigm, the submitted users' raw data is completely revealed to the analyst. In Local Differential Privacy (LDP), each user locally perturbs the data before submitting it to the data collector. Users no longer need to fully trust the analyst. Nevertheless, the LDP paradigm suffers from a strong constraint on the utility of statistical functions. To tackle the conflicts, recent works propose the shuffle model to enhance the utility of LDP mechanism. A new participant, i.e., shuffler, is introduced between users and the analyst to produce the privacy amplification. In this paper, we propose DUMP (\underline{DUM}my-\underline{P}oint-based), a framework for privacy-preserving histogram estimation in shuffle model. DUMP can summarize all existing histogram estimation protocols in shuffle model. We introduce $dummy\ blanket$ intuition to analyze the advantage of dummy points in improving utility. Then we design two protocols: pureDUMP and mixDUMP under DUMP framework, which achieve better trade-offs between privacy, accuracy, and communication than existing protocols. We also prove that dummy points have privacy amplification locally, which can achieve enhanced privacy protection on the shuffler. Besides, existing related studies lacks experimental evaluation that results are still in the theoretical stage. We conduct a comprehensive experimental evaluation to evaluate our proposed protocols and existing other protocols. Experimental results on both synthetic and real-world datasets show that our proposed protocols achieve better utility for both accuracy and communication under the same privacy guarantee than existing protocols.

DUMP: A Dummy-Point-Based Framework for Histogram Estimation in Shuffle Model.

The shuffle model is recently proposed to address the issue of severe utility loss in Local Differential Privacy (LDP) due to distributed data randomization.In the shuffle model, a shuffler is utilized to break the link between the user identity and the message uploaded to the data analyst. Since less noise needs to be introduced to achieve the same privacy guarantee, following this paradigm, the utility of privacy-preserving data collection is improved. 
We propose DUMP (\underline{DUM}my-\underline{P}oint-based), a framework for privacy-preserving histogram estimation in the shuffle model. The core of DUMP is a new concept of \emph{dummy blanket}, which enables enhancing privacy by just introducing \textit{points }on the user side and further improving the utility of the shuffle model.We instantiate DUMP by proposing two protocols: pureDUMP and mixDUMP, and conduct a comprehensive experimental evaluation to compare them with existing protocols. The experimental results show that, under the same privacy guarantee, (1) the proposed protocols have significant improvements in communication efficiency over all existing multi-message protocols, by at least 3 orders of magnitude; (2) they achieve competitive utility, while the only known protocol (Ghazi \textit{et al.}, PMLR 2020) having better utility than ours employs hard-to-exactly-sample distributions which are vulnerable to floating-point attacks (CCS 2012).

Kunzhe Huang

Papers

Backdoor Defense via Decoupling the Training Process

DUMP: A Dummy-Point-Based Framework for Histogram Estimation in Shuffle Model.

Privacy Enhancement via Dummy Points in the Shuffle Model