Open AccessPosted Content
A Theorem of the Alternative for Personalized Federated Learning.
TLDR
This paper shows how the excess risks of personalized federated learning with a smooth, strongly convex loss depend on data heterogeneity from a minimax point of view, and reveals a surprising theorem of the alternative for personalized federation learning.Abstract:
A widely recognized difficulty in federated learning arises from the statistical heterogeneity among clients: local datasets often come from different but not entirely unrelated distributions, and personalization is, therefore, necessary to achieve optimal results from each individual's perspective. In this paper, we show how the excess risks of personalized federated learning with a smooth, strongly convex loss depend on data heterogeneity from a minimax point of view. Our analysis reveals a surprising theorem of the alternative for personalized federated learning: there exists a threshold such that (a) if a certain measure of data heterogeneity is below this threshold, the FedAvg algorithm [McMahan et al., 2017] is minimax optimal; (b) when the measure of heterogeneity is above this threshold, then doing pure local training (i.e., clients solve empirical risk minimization problems on their local datasets without any communication) is minimax optimal. As an implication, our results show that the presumably difficult (infinite-dimensional) problem of adapting to client-wise heterogeneity can be reduced to a simple binary decision problem of choosing between the two baseline algorithms. Our analysis relies on a new notion of algorithmic stability that takes into account the nature of federated learning.read more
Citations
More filters
Proceedings ArticleDOI
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
TL;DR: The reason behind generalizability of the FedAvg’s output is its power in learning the common data representation among the clients’ tasks, by leveraging the diversity among client data distributions via local updates, in the multi-task linear representation setting.
Journal Article
Adaptive and Robust Multi-task Learning
Yaqiong Duan,Kaizheng Wang +1 more
TL;DR: In this article , a family of adaptive methods that automatically utilize possible similarities among those tasks while carefully handling their differences is proposed. But their robustness against outlier tasks is questionable.
Proceedings ArticleDOI
Privacy-Preserving Federated Multi-Task Linear Regression: A One-Shot Linear Mixing Approach Inspired By Graph Regularization
TL;DR: This work focuses on the federated multi-task linear regression setting, where each machine possesses its own data for individual tasks and sharing the full local data between machines is prohibited, and proposes a novel fusion framework that only requires a one-shot communication of local estimates.
Posted Content
Personalized Federated Learning with Gaussian Processes
TL;DR: In this paper, a solution to PFL that is based on Gaussian processes (GPs) with deep kernel learning is presented, where a shared kernel function across all clients, parameterized by a neural network, with a personal GP classifier for each client.
Journal ArticleDOI
Personalized Federated Learning with Multiple Known Clusters
TL;DR: This work develops an algorithm that allows each cluster to communicate independently and derive the convergence results, and studies a hierarchical linear model to theoretically demonstrate that this approach outperforms agents learning independently and agents learning a single shared weight.
References
More filters
Proceedings Article
Communication-Efficient Learning of Deep Networks from Decentralized Data
TL;DR: In this paper, the authors presented a decentralized approach for federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets.
Journal ArticleDOI
A theory of learning from different domains
Shai Ben-David,John Blitzer,Koby Crammer,Alex Kulesza,Fernando Pereira,Jennifer Wortman Vaughan +5 more
TL;DR: A classifier-induced divergence measure that can be estimated from finite, unlabeled samples from the domains and shows how to choose the optimal combination of source and target error as a function of the divergence, the sample sizes of both domains, and the complexity of the hypothesis class.
Book ChapterDOI
Introduction to the non-asymptotic analysis of random matrices.
TL;DR: This is a tutorial on some basic non-asymptotic methods and concepts in random matrix theory, particularly for the problem of estimating covariance matrices in statistics and for validating probabilistic constructions of measurementMatrices in compressed sensing.
Book
Multitask learning
TL;DR: Multitask learning as discussed by the authors is an approach to inductive transfer that improves learning for one task by using the information contained in the training signals of other related tasks, and it does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better.
Proceedings Article
Analysis of Representations for Domain Adaptation
TL;DR: The theory illustrates the tradeoffs inherent in designing a representation for domain adaptation and gives a new justification for a recently proposed model which explicitly minimizes the difference between the source and target domains, while at the same time maximizing the margin of the training set.