This paper reviews concepts, modeling approaches, and recent findings along a spectrum of different levels of abstraction of neural network models including generalization across (1) Samples, (2) Distributions, (3) Domains, (4) Tasks, (5) Modalities, and (6) Scopes. Results on (1) sample generalization show that, in the case of ImageNet, nearly all the recent improvements reduced training error while overfitting stayed flat; with nearly all the training error eliminated, future progress will require a focus on reducing overfitting. Perspectives from statistics highlight how (2) distribution generalization can be viewed alternately as a change in sample weights or a change in the input-output relationship; thus, techniques that have been successful in domain generalization have the potential to be applied to difficult forms of sample or distribution generalization. Transfer learning approaches to (3) domain generalization are summarized, as are recent advances and the wealth of domain adaptation benchmark datasets available. Recent breakthroughs surveyed in (4) task generalization include few-shot meta-learning approaches and the BERT NLP engine, and recent (5) modality generalization studies are discussed that integrate image and text data and that apply a biologically-inspired network across olfactory, visual, and auditory modalities. Recent (6) scope generalization results are reviewed that embed knowledge graphs into deep NLP approaches. Additionally, concepts from neuroscience are discussed on the modular architecture of brains and the steps by which dopamine-driven conditioning leads to abstract thinking.

Generalization in Neural Networks: A Broad Survey

A Multi-objective Optimization Approach for the Synthesis of Granular Computing-Based Classification Systems in the Graph Domain

In the real world, it is challenging to calculate a trade-off alternative with traditional classical methods for complex non-linear systems, which always involve multiple conflicting objectives. Such complicated systems urgently desire advanced methods to conquer the multi-objective optimization problems (MOPs). As a promising AI method, the development and application of Machine Learning (ML) attract increasingly more attention from researchers. The natures of ML methods, such as parallel computation possibility, no need for any priori assumptions, etc., ensure the effectiveness and efficiency for solving MOPs. However, as we know, there is no literature related to the comprehensive review of ML in multi-objective optimization domain until now. This literature review aims to provide researchers a global view of mainstream ML methods for MOO in a general domain and a reference for applying ML methods to solve a specific type of MOPs. In this paper, the general ML mainstream methods are summarized, based on which the literature relating to ML on MOPs are retrieved in comprehensive domains. The relevant literature is categorized according to the emphasis of object types, purposes and methods, and the categorization results are finally analyzed and discussed.

Qu Qu

Papers

A Comprehensive Review of Machine Learning in Multi-objective Optimization