Good data stewardship requires removal of data at the request of the data's owner. This raises the question if and how a trained machine-learning model, which implicitly stores information about its training data, should be affected by such a removal request. Is it possible to "remove" data from a machine-learning model? We study this problem by defining certified removal: a very strong theoretical guarantee that a model from which data is removed cannot be distinguished from a model that never observed the data to begin with. We develop a certified-removal mechanism for linear classifiers and empirically study learning settings in which this mechanism is practical.

Certified Data Removal from Machine Learning Models

The EU Proposal for a General Data Protection Regulation has caused a wide debate between lawyers and legal scholars and many opinions have been voiced on the issue of the right to be forgotten. In order to analyse the relevance of the new rule provided by Article 17 of the Proposal, this paper considers the original idea of the right to be forgotten, pre-existing in both European and U.S. legal frameworks. This article focuses on the new provisions of Article 17 of the EU Proposal for a General Data Protection Regulation and evaluates its effects on court decisions. The author assumes that the new provisions do not seem to represent a revolutionary change to the existing rules with regard to the right granted to the individual, but instead have an impact on the extension of the protection of the information disseminated on-line.

The EU Proposal for a General Data Protection Regulation and the Roots of the ‘Right to Be Forgotten’

Artificial intelligence (AI) and algorithmic decision making are having a profound impact on our daily lives. These systems are vastly used in different high-stakes applications like healthcare, business, government, education, and justice, moving us toward a more algorithmic society. However, despite so many advantages of these systems, they sometimes directly or indirectly cause harm to the users and society. Therefore, it has become essential to make these systems safe, reliable, and trustworthy. Several requirements, such as fairness, explainability, accountability, reliability, and acceptance, have been proposed in this direction to make these systems trustworthy. This survey analyzes all of these different requirements through the lens of the literature. It provides an overview of different approaches that can help mitigate AI risks and increase trust and acceptance of the systems by utilizing the users and society. It also discusses existing strategies for validating and verifying these systems and the current standardization efforts for trustworthy AI. Finally, we present a holistic view of the recent advancements in trustworthy AI to help the interested researchers grasp the crucial facets of the topic efficiently and offer possible future research directions.

Trustworthy Artificial Intelligence: A Review

Training machine learning (ML) models is expensive in terms of computational power, amounts of labeled data and human expertise. Thus, ML models constitute intellectual property (IP) and business value for their owners. Embedding digital watermarks during model training allows a model owner to later identify their models in case of theft or misuse. However, model functionality can also be stolen via model extraction, where an adversary trains a surrogate model using results returned from a prediction API of the original model. Recent work has shown that model extraction is a realistic threat. Existing watermarking schemes are ineffective against IP theft via model extraction since it is the adversary who trains the surrogate model. In this paper, we introduce DAWN (Dynamic Adversarial Watermarking of Neural Networks), the first approach to use watermarking to deter model extraction IP theft. Unlike prior watermarking schemes, DAWN does not impose changes to the training process but it operates at the prediction API of the protected model, by dynamically changing the responses for a small subset of queries (e.g., 1- 2^{-64}$), incurring negligible loss of prediction accuracy (0.03-0.5%).

DAWN: Dynamic Adversarial Watermarking of Neural Networks

We study the data deletion problem for convex models. By leveraging techniques from convex optimization and reservoir sampling, we give the first data deletion algorithms that are able to handle an arbitrarily long sequence of adversarial updates while promising both per-deletion run-time and steady-state error that do not grow with the length of the update sequence. We also introduce several new conceptual distinctions: for example, we can ask that after a deletion, the entire state maintained by the optimization algorithm is statistically indistinguishable from the state that would have resulted had we retrained, or we can ask for the weaker condition that only the observable output is statistically indistinguishable from the observable output that would have resulted from retraining. We are able to give more efficient deletion algorithms under this weaker deletion criterion.

Descent-to-Delete: Gradient-Based Methods for Machine Unlearning

Once users have shared their data online, it is generally difficult for them to revoke access and ask for the data to be deleted. Machine learning (ML) exacerbates this problem because any model trained with said data may have memorized it, putting users at risk of a successful privacy attack exposing their information. Yet, having models unlearn is notoriously difficult.We introduce SISA training, a framework that expedites the unlearning process by strategically limiting the influence of a data point in the training procedure. While our framework is applicable to any learning algorithm, it is designed to achieve the largest improvements for stateful algorithms like stochastic gradient descent for deep neural networks. SISA training reduces the computational overhead associated with unlearning, even in the worst-case setting where unlearning requests are made uniformly across the training set. In some cases, the service provider may have a prior on the distribution of unlearning requests that will be issued by users. We may take this prior into account to partition and order data accordingly, and further decrease overhead from unlearning.Our evaluation spans several datasets from different domains, with corresponding motivations for unlearning. Under no distributional assumptions, for simple learning tasks, we observe that SISA training improves time to unlearn points from the Purchase dataset by 4.63×, and 2.45× for the SVHN dataset, over retraining from scratch. SISA training also provides a speed-up of 1.36× in retraining for complex learning tasks such as ImageNet classification; aided by transfer learning, this results in a small degradation in accuracy. Our work contributes to practical data governance in machine unlearning.

Machine Unlearning

Machine learning involves expensive data collection and training procedures. Model owners may be concerned that valuable intellectual property can be leaked if adversaries mount model extraction attacks. As it is difficult to defend against model extraction without sacrificing significant prediction accuracy, watermarking instead leverages unused model capacity to have the model overfit to outlier input-output pairs. Such pairs are watermarks, which are not sampled from the task distribution and are only known to the defender. The defender then demonstrates knowledge of the input-output pairs to claim ownership of the model at inference. The effectiveness of watermarks remains limited because they are distinct from the task distribution and can thus be easily removed through compression or other forms of knowledge transfer. 
We introduce Entangled Watermarking Embeddings (EWE). Our approach encourages the model to learn features for classifying data that is sampled from the task distribution and data that encodes watermarks. An adversary attempting to remove watermarks that are entangled with legitimate data is also forced to sacrifice performance on legitimate data. Experiments on MNIST, Fashion-MNIST, CIFAR-10, and Speech Commands validate that the defender can claim model ownership with 95\% confidence with less than 100 queries to the stolen copy, at a modest cost below 0.81 percentage points on average in the defended model's performance.

Entangled Watermarks as a Defense against Model Extraction

Training machine learning (ML) models typically involves expensive iterative optimization. Once the model’s final parameters are released, there is currently no mechanism for the entity which trained the model to prove that these parameters were indeed the result of this optimization procedure. Such a mechanism would support security of ML applications in several ways. For instance, it would simplify ownership resolution when multiple parties contest ownership of a specific model. It would also facilitate the distributed training across untrusted workers where Byzantine workers might otherwise mount a denial-ofservice by returning incorrect model updates.In this paper, we remediate this problem by introducing the concept of proof-of-learning in ML. Inspired by research on both proof-of-work and verified computations, we observe how a seminal training algorithm, stochastic gradient descent, accumulates secret information due to its stochasticity. This produces a natural construction for a proof-of-learning which demonstrates that a party has expended the compute require to obtain a set of model parameters correctly. In particular, our analyses and experiments show that an adversary seeking to illegitimately manufacture a proof-of-learning needs to perform at least as much work than is needed for gradient descent itself.We also instantiate a concrete proof-of-learning mechanism in both of the scenarios described above. In model ownership resolution, it protects the intellectual property of models released publicly. In distributed training, it preserves availability of the training procedure. Our empirical evaluation validates that our proof-of-learning mechanism is robust to variance induced by the hardware (e.g., ML accelerators) and software stacks.

Proof-of-Learning: Definitions and Practice

Once users have shared their data online, it is generally difficult for them
to revoke access and ask for the data to be deleted. Machine learning (ML)
exacerbates this problem because any model trained with said data may have
memorized it, putting users at risk of a successful privacy attack exposing
their information. Yet, having models unlearn is notoriously difficult. We
introduce SISA training, a framework that expedites the unlearning process by
strategically limiting the influence of a data point in the training procedure.
While our framework is applicable to any learning algorithm, it is designed to
achieve the largest improvements for stateful algorithms like stochastic
gradient descent for deep neural networks. SISA training reduces the
computational overhead associated with unlearning, even in the worst-case
setting where unlearning requests are made uniformly across the training set.
In some cases, the service provider may have a prior on the distribution of
unlearning requests that will be issued by users. We may take this prior into
account to partition and order data accordingly, and further decrease overhead
from unlearning. Our evaluation spans several datasets from different domains,
with corresponding motivations for unlearning. Under no distributional
assumptions, for simple learning tasks, we observe that SISA training improves
time to unlearn points from the Purchase dataset by 4.63x, and 2.45x for the
SVHN dataset, over retraining from scratch. SISA training also provides a
speed-up of 1.36x in retraining for complex learning tasks such as ImageNet
classification; aided by transfer learning, this results in a small degradation
in accuracy. Our work contributes to practical data governance in machine
unlearning.

/pdf/entangled-watermarks-as-a-defense-against-model-extraction-btpy35ivqb.pdf

Hengrui Jia

Papers

Machine Unlearning

Entangled Watermarks as a Defense against Model Extraction

Proof-of-Learning: Definitions and Practice

Machine Unlearning

Entangled Watermarks as a Defense against Model Extraction