Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target-domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning research studies, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey article reviews more than 40 representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over 20 representative transfer learning models are used for experiments. The models are performed on three different data sets, that is, Amazon Reviews, Reuters-21578, and Office-31, and the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

A Comprehensive Survey on Transfer Learning

We propose an independence criterion based on the eigen-spectrum of covariance operators in reproducing kernel Hilbert spaces (RKHSs), consisting of an empirical estimate of the Hilbert-Schmidt norm of the cross-covariance operator (we term this a Hilbert-Schmidt Independence Criterion, or HSIC). This approach has several advantages, compared with previous kernel-based independence criteria. First, the empirical estimate is simpler than any other kernel dependence test, and requires no user-defined regularisation. Second, there is a clearly defined population quantity which the empirical estimate approaches in the large sample limit, with exponential convergence guaranteed between the two: this ensures that independence tests based on HSIC do not suffer from slow learning rates. Finally, we show in the context of independent component analysis (ICA) that the performance of HSIC is competitive with that of previously published kernel-based criteria, and of other recently published ICA methods.

Measuring statistical dependence with Hilbert-Schmidt norms

For a target task where the labeled data are unavailable, domain adaptation can transfer a learner from a different source domain. Previous deep domain adaptation methods mainly learn a global domain shift, i.e., align the global source and target distributions without considering the relationships between two subdomains within the same category of different domains, leading to unsatisfying transfer learning performance without capturing the fine-grained information. Recently, more and more researchers pay attention to subdomain adaptation that focuses on accurately aligning the distributions of the relevant subdomains. However, most of them are adversarial methods that contain several loss functions and converge slowly. Based on this, we present a deep subdomain adaptation network (DSAN) that learns a transfer network by aligning the relevant subdomain distributions of domain-specific layer activations across different domains based on a local maximum mean discrepancy (LMMD). Our DSAN is very simple but effective, which does not need adversarial training and converges fast. The adaptation can be achieved easily with most feedforward network models by extending them with LMMD loss, which can be trained efficiently via backpropagation. Experiments demonstrate that DSAN can achieve remarkable results on both object recognition tasks and digit classification tasks. Our code will be available at https://github.com/easezyc/deep-transfer-learning .

Deep Subdomain Adaptation Network for Image Classification

Deep Learning and Medical Image Processing for Coronavirus (COVID-19) Pandemic: A Survey.

Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. Such approaches offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information. However, the simultaneous learning of multiple tasks presents new design and optimization challenges, and choosing which tasks should be learned jointly is in itself a non-trivial problem. In this survey, we give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field. Our discussion is structured according to a partition of the existing deep MTL techniques into three groups: architectures, optimization methods, and task relationship learning. We also provide a summary of common multi-task benchmarks.

Multi-Task Learning with Deep Neural Networks: A Survey

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning researches, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey paper reviews more than forty representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over twenty representative transfer learning models are used for experiments. The models are performed on three different datasets, i.e., Amazon Reviews, Reuters-21578, and Office-31. And the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

Zeroing neural dynamics (ZND), a special class of neural dynamics, is a powerful methodology for time-varying problems solving. On the basis of this methodology, different continuous-time ZND models are obtained for various time-varying problems solving. Continuous-time ZND models are supposed to be discretized for the sake of prevalent digital-equipment applications, and a discretization formula is needed to transform a continuous-time ZND model into a discrete-time ZND model. In this article, continuous-time ZND models, new discretization formulas and various discrete-time ZND models are presented. The time-varying minimization problem, which is a representative time-varying issue, is also discussed as an example throughout this article. The relationship between ZND and Zhao-Lu-Swamy-Feng (ZLSF) models is identified; i.e., the ZLSF models are minimization-type and Euler-type special cases of ZND models. In addition, ZND models are compared with other models to demonstrate their differences. The article aims to introduce the ZND methodology and illustrate the manner by which it is used, provide readers with new discretization formulas and various continuous-time and discrete-time ZND models for time-varying problems solving, discuss the factors affecting the performance of the aforementioned models, exemplify the differences between ZND models and other models, and point out future research directions.

Zeroing Neural Dynamics and Models for Various Time-Varying Problems Solving with ZLSF Models as Minimization-Type and Euler-Type Special Cases [Research Frontier]

Time-variant problems, which can be classified into future and non-future problems, are often encountered in academia and industry. In a future problem, we only know the information on the current and past time instants, and we have to acquire the next-time-instant solution before the next time instant arrives. Zeroing neural dynamics (ZND) and Zhang et al. discretization (ZeaD) formula group are two essential tools to build discrete-time ZND (DT-ZND) models for future problems solving. The former uses a systematical design formula to build a continuous-time ZND (CT-ZND) model, and the latter is used to transform the CT-ZND model into the discrete-time forms. Many DT-ZND models have been developed to solve various time-variant problems. In light of DT-ZND models and correction strategy, in this paper, we mainly focus on designing and building improved models for future problems solving. Based on the ZND method, extrapolation formulas, and correction steps, new models and corresponding computational algorithms are proposed to solve future optimization and future matrix inversion problems. The numerical experiments are also carried out to demonstrate the superiority of the proposed algorithms.

/pdf/new-models-for-future-problems-solving-by-using-znd-method-3fs8gz9bkc.pdf

New Models for Future Problems Solving by Using ZND Method, Correction Strategy and Extrapolation Formulas

Knowledge Graph (KG) alignment is to discover the mappings (i.e., equivalent entities, relations, and others) between two KGs. The existing methods can be divided into the embedding-based models, and the conventional reasoning and lexical matching based systems. The former compute the similarity of entities via their cross-KG embeddings, but they usually rely on an ideal supervised learning setting for good performance and lack appropriate reasoning to avoid logically wrong mappings; while the latter address the reasoning issue but are poor at utilizing the KG graph structures and the entity contexts. In this study, we aim at combining the above two solutions and thus propose an iterative framework named PRASE which is based on probabilistic reasoning and semantic embedding. It learns the KG embeddings via entity mappings from a probabilistic reasoning system named PARIS, and feeds the resultant entity mappings and embeddings back into PARIS for augmentation. The PRASE framework is compatible with different embedding-based models, and our experiments on multiple datasets have demonstrated its state-of-the-art performance.

/pdf/unsupervised-knowledge-graph-alignment-by-probabilistic-4lw25kt890.pdf

Zhiyuan Qi

Papers

A Comprehensive Survey on Transfer Learning

A Comprehensive Survey on Transfer Learning

Zeroing Neural Dynamics and Models for Various Time-Varying Problems Solving with ZLSF Models as Minimization-Type and Euler-Type Special Cases [Research Frontier]

New Models for Future Problems Solving by Using ZND Method, Correction Strategy and Extrapolation Formulas

Unsupervised Knowledge Graph Alignment by Probabilistic Reasoning and Semantic Embedding