Proximal splitting algorithms are well suited to solving large-scale nonsmooth optimization problems, in particular those arising in machine learning. We propose a new primal-dual algorithm, in which the dual update is randomized; equivalently, the proximity operator of one of the function in the problem is replaced by a stochastic oracle. For instance, some randomly chosen dual variables, instead of all, are updated at each iteration. Or, the proximity operator of a function is called with some small probability only. A nonsmooth variance-reduction technique is implemented so that the algorithm finds an exact minimizer of the general problem involving smooth and nonsmooth functions, possibly composed with linear operators. We derive linear convergence results in presence of strong convexity; these results are new even in the deterministic case, when our algorithms reverts to the recently proposed Primal-Dual Davis-Yin algorithm. Some randomized algorithms of the literature are also recovered as particular cases (e.g., Point-SAGA). But our randomization technique is general and encompasses many unbiased mechanisms beyond sampling and probabilistic updates, including compression. Since the convergence speed depends on the slowest among the primal and dual contraction mechanisms, the iteration complexity might remain the same when randomness is used. On the other hand, the computation complexity can be significantly reduced. Overall, randomness helps getting faster algorithms. This has long been known for stochastic-gradient-type algorithms, and our work shows that this fully applies in the more general primal-dual setting as well.

RandProx: Primal-Dual Optimization Algorithms with Randomized Proximal Updates

Distributed Mean Estimation (DME), in which $n$ clients communicate vectors to a parameter server that estimates their average, is a fundamental building block in communication-efficient federated learning. In this paper, we improve on previous DME techniques that achieve the optimal $O(1/n)$ Normalized Mean Squared Error (NMSE) guarantee by asymptotically improving the complexity for either encoding or decoding (or both). To achieve this, we formalize the problem in a novel way that allows us to use off-the-shelf mathematical solvers to design the quantization.

QUIC-FL: Quick Unbiased Compression for Federated Learning

In federated learning, a large number of users are involved in a global learning task, in a collaborative way. They alternate local computations and communication with a distant server. Communication, which can be slow and costly, is the main bottleneck in this setting. To ac-celerate distributed gradient descent, the popular strategy of local training is to communicate less frequently; that is, to perform several iterations of local computations between the communication steps. A recent breakthrough in this ﬁeld was made by Mishchenko et al. (2022): their Scaﬀnew algorithm is the ﬁrst to probably beneﬁt from local training, with accelerated communication complexity. However, it was an open and challenging question to know whether the powerful mechanism behind Scaﬀnew would be compatible with partial participation , the desirable feature that not all clients need to participate to every round of the training process. We answer this question positively and propose a new algorithm, which handles local training and partial participation, with state-of-the-art communication complexity.

TAMUNA: Accelerated Federated Learning with Local Training and Partial Participation

Federated Learning is a collaborative training framework that leverages heterogeneous data distributed across a vast number of clients. Since it is practically infeasible to request and process all clients during the aggregation step, partial participation must be supported. In this setting, the communication between the server and clients poses a major bottleneck. To reduce communication loads, there are two main approaches: compression and local steps. Recent work by Mishchenko et al. [2022] introduced the new ProxSkip method, which achieves an accelerated rate using the local steps technique. Follow-up works successfully combined local steps acceleration with partial participation [Grudzie\'n et al., 2023, Condat et al. 2023] and gradient compression [Condat et al. [2022]. In this paper, we finally present a complete method for Federated Learning that incorporates all necessary ingredients: Local Training, Compression, and Partial Participation. We obtain state-of-the-art convergence guarantees in the considered setting. Moreover, we analyze the general sampling framework for partial participation and derive an importance sampling scheme, which leads to even better performance. We experimentally demonstrate the advantages of the proposed method in practice.

Improving Accelerated Federated Learning with Compression and Importance Sampling

Assume that we have to minimize f ( x, y ) , where f is smooth and µ x -strongly convex in x and µ y -strongly convex in y . Optimal fast gradient method of Yurii Nesterov requires ∼ (min ( µ x , µ y )) − 1 / 2 calculations of ∇ x f and ∇ y f . In this paper we propose an algorithm that requires ∼ µ − 1 / 2 x calculations of ∇ x f and ∼ µ − 1 / 2 y calculations of ∇ y f . In some applications µ x (cid:29) µ y and calculation of ∇ y f is much cheaper than ∇ x f . In this case, we have signiﬁ-cant wall-clock time acceleration.

Optimal algorithm for min-min strongly convex optimization

In the modern paradigm of federated learning, a large number of users are involved in a global learning task, in a collaborative way. They alternate local computations and two-way communication with a distant orchestrating server. Communication, which can be slow and costly, is the main bottleneck in this setting. To reduce the communication load and therefore accelerate distributed gradient descent, two strategies are popular: 1) communicate less frequently; that is, perform several iterations of local computations between the communication rounds; and 2) communicate compressed information instead of full-dimensional vectors. In this paper, we propose the ﬁrst algorithm for distributed optimization and federated learning, which harnesses these two strategies jointly and converges linearly to an exact solution, with a doubly accelerated rate: our algorithm beneﬁts from the two acceleration mechanisms provided by local training and compression, namely a better dependency on the condition number of the functions and on the dimension of the model, respectively.

Provably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Compressed Communication

One powerful strategy of how to increase the complexity of cellular proteomes is through post-translational modifications (PTMs) of proteins. Currently there are approximately 400 types of PTMs, the different combinations of which yield a large variety of protein isoforms with distinct biochemical properties. Although mitochondrial proteins undergoing PTMs were identified nearly six decades ago, studies on the roles and extent of PTMs on mitochondrial functions lagged behind the other cellular compartments. The application of mass spectrometry for the characterization of the mitochondrial proteome as well as for the detection of various PTMs resulted in the identification of thousands of amino acid positions that can be modified by different chemical groups. However, the data on mitochondrial PTMs are scattered in several datasets, and the available databases do not contain a complete list of modified residues. To integrate information on PTMs of the mitochondrial proteome of the yeast Saccharomyces cerevisiae, we built the y-mtPTM (Yeast Mitochondrial Post-Translational Modification) Database (http://compbio.fmph.uniba.sk/y-mtptm/). It lists nearly 20,000 positions on mitochondrial proteins affected by approximately twenty various PTMs, with phosphorylated, succinylated, acetylated and ubiquitylated sites being the most abundant. A simple search of a protein of interest reveals the modified amino acid residues, their position within the primary sequence as well as on its 3D structure and links to the source reference(s). The database will serve yeast mitochondrial researchers as a comprehensive platform to investigate the functional significance of the PTMs of mitochondrial proteins.

Ivan Agarsky

Papers

Provably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Compressed Communication

y-mtPTM: Yeast Mitochondrial Post-Translational Modification Database.