A state-of-the-art survey on solving non-IID data in Federated Learning

To avoid the original private data uploading in cloud-edgecomputing, the federated learning (FL) scheme is recently proposed which enhances the privacy preservation. However, the attacks against the uploaded model updates in FL can still cause private data leakage which demotivates the privacy-sensitive participating edge devices. To address this issue, we aim to design a privacy-preserving incentive mechanism for the federated cloud-edge learning (PFCEL) system such that 1) the privacy-sensitive edge devices are motivated to contribute to the local training and model uploading, 2) a trade-off between the private data leakage and the model accuracy is achieved. We first model the data leakage quantitatively from an adversarial perspective, and then formulate the incentive design problem as a three-layer Stackelberg game, where the interaction between the edge servers and edge devices is further formulated as an optimal contract design problem. Extensive theoretical analysis and numerical evaluations demonstrate the effectiveness of our designed mechanism in terms of privacy preservation and system utility.

Privacy-Preserving Incentive Mechanism Design for Federated Cloud-Edge Learning

This paper proposes an efficient federated distillation learning system (EFDLS) for multi-task time series classification (TSC). EFDLS consists of a central server and multiple mobile users, where different users may run different TSC tasks. EFDLS has two novel components: a feature-based student-teacher (FBST) framework and a distance-based weights matching (DBWM) scheme. For each user, the FBST framework transfers knowledge from its teacher&#x2019;s hidden layers to its student&#x2019;s hidden layers via knowledge distillation, where the teacher and student have identical network structures. For each connected user, its student model&#x2019;s hidden layers&#x2019; weights are uploaded to the EFDLS server periodically. The DBWM scheme is deployed on the server, with the least square distance used to measure the similarity between the weights of two given models. This scheme finds a partner for each connected user such that the user&#x2019;s and its partner&#x2019;s weights are the closest among all the weights uploaded. The server exchanges and sends back the user&#x2019;s and its partner&#x2019;s weights to these two users which then load the received weights to their teachers&#x2019; hidden layers. Experimental results show that compared with a number of state-of-the-art federated learning algorithms, our proposed EFDLS wins 20 out of 44 standard UCR2018 datasets and achieves the highest mean accuracy (70.14%) on these datasets. In particular, compared with a single-task Baseline, EFDLS obtains 32/4/8 regarding &#x2019;win&#x2019;/&#x2019;tie&#x2019;/&#x2019;lose&#x2019; and results in an improvement of approximately 4% in terms of mean accuracy. 

/pdf/an-efficient-federated-distillation-learning-system-for-4u8rxqb9.pdf

An Efficient Federated Distillation Learning System for Multitask Time Series Classification

Federated Recommendation (FR) has received con-siderable popularity and attention in the past few years. In FR, for each user, its feature vector and interaction data are kept locally on its own client thus are private to others. Without the access to above information, most existing poisoning attacks against recommender systems or federated learning lose validity. Benifiting from this characteristic, FR is commonly considered fairly secured. However, we argue that there is still possible and necessary security improvement could be made in FR. To prove our opinion, in this paper we present FedRecAttack, a model poisoning attack to FR aiming to raise the exposure ratio of target items. In most recommendation scenarios, apart from pri-vate user-item interactions (e.g., clicks, watches and purchases), some interactions are public (e.g., likes, follows and comments). Motivated by this point, in FedRecAttack we make use of the public interactions to approximate users' feature vectors, thereby attacker can generate poisoned gradients accordingly and control malicious users to upload the poisoned gradients in a well-designed way. To evaluate the effectiveness and side effects of FedRecAttack, we conduct extensive experiments on three real-world datasets of different sizes from two completely different scenarios. Experimental results demonstrate that our proposed FedRecAttack achieves the state-of-the-art effectiveness while its side effects are negligible. Moreover, even with small proportion (3%) of malicious users and small proportion (1%) of public interactions, FedRecAttack remains highly effective, which reveals that FR is more vulnerable to attack than people commonly considered.

FedRecAttack: Model Poisoning Attack to Federated Recommendation

Machine learning models have been deployed in mobile networks to deal with massive data from different layers to enable automated network management and intelligence on devices. To overcome high communication cost and severe privacy concerns of centralized machine learning, federated learning (FL) has been proposed to achieve distributed machine learning among networked devices. While the computation and communication limitation has been widely studied, the impact of on-device storage on the performance of FL is still not explored. Without an effective data selection policy to filter the massive streaming data on devices, classical FL can suffer from much longer model training time (4 ×) and significant inference accuracy reduction (7%), observed in our experiments. In this work, we take the first step to consider the online data selection for FL with limited on-device storage. We first define a new data valuation metric for data evaluation and selection in FL with theoretical guarantees for speeding up model convergence and enhancing final model accuracy, simultaneously. We further design ODE, a framework of Online Data sElection for FL, to coordinate networked devices to store valuable data samples. Experimental results on one industrial dataset and three public datasets show the remarkable advantages of ODE over the state-of-the-art approaches. Particularly, on the industrial dataset, ODE achieves as high as 2.5 × speedup of training time and 6% increase in inference accuracy, and is robust to various factors in practical environments.

/pdf/to-store-or-not-online-data-selection-for-federated-learning-2unq7gyu.pdf

To Store or Not? Online Data Selection for Federated Learning with Limited Storage

User modeling aims to capture the latent characteristics of users from their behaviors, and is widely applied in numerous applications. Usually, centralized user modeling suffers from the risk of privacy leakage. Instead, federated user modeling expects to provide a secure multi-client collaboration for user modeling through federated learning. Existing federated learning methods are mainly designed for consistent clients, which cannot be directly applied to practical scenarios, where different clients usually store inconsistent user data. Therefore, it is a crucial demand to design an appropriate federated solution that can better adapt to user modeling tasks, and however, meets following critical challenges: 1) Statistical heterogeneity. The distributions of user data in different clients are not always independently identically distributed which leads to personalized clients; 2) Privacy heterogeneity. User data contains both public and private information, which have different levels of privacy. It means we should balance different information to be shared and protected; 3) Model heterogeneity. The local user models trained with client records are heterogeneous which need flexible aggregation in the server. In this paper, we propose a novel client-server architecture framework, namely Hierarchical Personalized Federated Learning (HPFL) to serve federated learning in user modeling with inconsistent clients. In the framework, we first define hierarchical information to finely partition the data with privacy heterogeneity. On this basis, the client trains a user model which contains different components designed for hierarchical information. Moreover, client processes a fine-grained personalized update strategy to update personalized user model for statistical heterogeneity. Correspondingly, the server completes a differentiated component aggregation strategy to flexibly aggregate heterogeneous user models in the case of privacy and model heterogeneity. Finally, we conduct extensive experiments on real-world datasets, which demonstrate the effectiveness of the HPFL framework.

Hierarchical Personalized Federated Learning for User Modeling

High-efficiency control of pesticide and heavy metal combined pollution in paddy soil using biochar/g-C3N4 photoresponsive soil remediation agent

In the Natural Language for Optimization (NL4Opt) NeurIPS 2022 competition, competitors focus on improving the accessibility and usability of optimization solvers, with the aim of subtask 1: recognizing the semantic entities that correspond to the components of the optimization problem; subtask 2: generating formulations for the optimization problem. In this paper, we present the solution of our team. First, we treat subtask 1 as a named entity recognition (NER) problem with the solution pipeline including pre-processing methods, adversarial training, post-processing methods and ensemble learning. Besides, we treat subtask 2 as a generation problem with the solution pipeline including specially designed prompts, adversarial training, post-processing methods and ensemble learning. Our proposed methods have achieved the F1-score of 0.931 in subtask 1 and the accuracy of 0.867 in subtask 2, which won the fourth and third places respectively in this competition. Our code is available at https://github.com/bigdata-ustc/nl4opt.

A Novel Approach for Auto-Formulation of Optimization Problems

Large language models (LLMs), like ChatGPT, have shown some human-like cognitive abilities. For comparing these abilities of different models, several benchmarks (i.e. sets of standard test questions) from different fields (e.g., Literature, Biology and Psychology) are often adopted and the test results under traditional metrics such as accuracy, recall and F1, are reported. However, such way for evaluating LLMs can be inefficient and inaccurate from the cognitive science perspective. Inspired by Computerized Adaptive Testing (CAT) used in psychometrics, we propose an adaptive testing framework for LLM evaluation. Rather than using a standard test set and simply reporting accuracy, this approach dynamically adjusts the characteristics of the test questions, such as difficulty, based on the model's performance. This allows for a more accurate estimation of the model's abilities, using fewer questions. More importantly, it allows LLMs to be compared with humans easily, which is essential for NLP models that aim for human-level ability. Our diagnostic reports have found that ChatGPT often behaves like a ``careless student'', prone to slip and occasionally guessing the questions. We conduct a fine-grained diagnosis and rank the latest 6 instruction-tuned LLMs from three aspects of Subject Knowledge, Mathematical Reasoning, and Programming, where GPT4 can outperform other models significantly and reach the cognitive ability of middle-level students. Different tests for different models using efficient adaptive testing -- we believe this has the potential to become a new norm in evaluating large language models.

Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Understanding mathematical questions effectively is a crucial task, which can benefit many applications, such as difficulty estimation. Researchers have drawn much attention to designing pre-training models for question representations due to the scarcity of human annotations (e.g., labeling difficulty). However, unlike general free-format texts (e.g., user comments), mathematical questions are generally designed with explicit purposes and mathematical logic, and usually consist of more complex content, such as formulas, and related mathematical knowledge (e.g., Function). Therefore, the problem of holistically representing mathematical questions remains underexplored. To this end, in this paper, we propose a novel contrastive pre-training approach for mathematical question representations, namely QuesCo, which attempts to bring questions with more similar purposes closer. Specifically, we first design two-level question augmentations, including content-level and structure-level, which generate literally diverse question pairs with similar purposes. Then, to fully exploit hierarchical information of knowledge concepts, we propose a knowledge hierarchy-aware rank strategy (KHAR), which ranks the similarities between questions in a fine-grained manner. Next, we adopt a ranking contrastive learning task to optimize our model based on the augmented and ranked questions. We conduct extensive experiments on two real-world mathematical datasets. The experimental results demonstrate the effectiveness of our model.

Yuting Ning

Papers

Hierarchical Personalized Federated Learning for User Modeling

High-efficiency control of pesticide and heavy metal combined pollution in paddy soil using biochar/g-C3N4 photoresponsive soil remediation agent

A Novel Approach for Auto-Formulation of Optimization Problems

Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective

Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training