Showing papers on "Benchmark (computing) published in 2021"

Bottleneck Transformers for Visual Recognition

Abstract: We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt [67] evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in "compute"1 time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision.2

Reptile Search Algorithm (RSA): A nature-inspired meta-heuristic optimizer

Abstract: This paper proposes a novel nature-inspired meta-heuristic optimizer, called Reptile Search Algorithm (RSA), motivated by the hunting behaviour of Crocodiles. Two main steps of Crocodile behaviour are implemented, such as encircling, which is performed by high walking or belly walking, and hunting, which is performed by hunting coordination or hunting cooperation. The mentioned search methods of the proposed RSA are unique compared to other existing algorithms. The performance of the proposed RSA is evaluated using twenty-three classical test functions, thirty CEC2017 test functions, ten CEC2019 test functions, and seven real-world engineering problems. The obtained results of the proposed RSA are compared to various existing optimization algorithms in the literature. The results of the tested three benchmark functions revealed that the proposed RSA achieved better results than the other competitive optimization algorithms. The results of the Friedman ranking test proved that the RSA is a significantly superior method than other comparative methods. Finally, the results of the examined engineering problems showed that the RSA obtained better results compared to other various methods.

Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

Abstract: There is an ever-growing demand for artificial intelligence. Optical processors, which compute with photons instead of electrons, can fundamentally accelerate the development of artificial intelligence by offering substantially improved computing performance. There has been long-term interest in optically constructing the most widely used artificial-intelligence architecture, that is, artificial neural networks, to achieve brain-inspired information processing at the speed of light. However, owing to restrictions in design flexibility and the accumulation of system errors, existing processor architectures are not reconfigurable and have limited model complexity and experimental performance. Here, we propose the reconfigurable diffractive processing unit, an optoelectronic fused computing architecture based on the diffraction of light, which can support different neural networks and achieve a high model complexity with millions of neurons. Along with the developed adaptive training approach to circumvent system errors, we achieved excellent experimental accuracies for high-speed image and video recognition over benchmark datasets and a computing performance superior to that of cutting-edge electronic computing platforms. Linear diffractive structures are by themselves passive systems but researchers here exploit the non-linearity of a photodetector to realize a reconfigurable diffractive ‘processing’ unit. High-speed image and video recognition is demonstrated.

SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud

Abstract: We present Self-Ensembling Single-Stage object Detector (SE-SSD) for accurate and efficient 3D object detection in outdoor point clouds. Our key focus is on exploiting both soft and hard targets with our formulated constraints to jointly optimize the model, without introducing extra computation in the inference. Specifically, SE-SSD contains a pair of teacher and student SSDs, in which we design an effective IoU-based matching strategy to filter soft targets from the teacher and formulate a consistency loss to align student predictions with them. Also, to maximize the distilled knowledge for ensembling the teacher, we design a new augmentation scheme to produce shape-aware augmented samples to train the student, aiming to encourage it to infer complete object shapes. Lastly, to better exploit hard targets, we design an ODIoU loss to supervise the student with constraints on the predicted box centers and orientations. Our SE-SSD attains top performance compared with all prior published works. Also, it attains top precisions for car detection in the KITTI benchmark (ranked 1st and 2nd on the BEV and 3D leaderboards1, respectively) with an ultra-high inference speed. The code is available at https://github.com/Vegeta2020/SE-SSD.

Algorithms for Verifying Deep Neural Networks

Abstract: Neural networks have been widely used in many applications, such as image classification and understanding, language processing, and control of autonomous systems. These networks work by mapping inputs to outputs through a sequence of layers. At each layer, the input to that layer undergoes an affine transformation followed by a simple nonlinear transformation before being passed to the next layer. Neural networks are being used for increasingly important tasks, and in some cases, incorrect outputs can lead to costly consequences, hence validation of correctness at each layer is vital. The sheer size of the networks makes this not feasible using traditional methods. In this monograph, the authors survey a class of methods that are capable of formally verifying properties of deep neural networks. In doing so, they introduce a unified mathematical framework for verifying neural networks, classify existing methods under this framework, provide pedagogical implementations of existing methods, and compare those methods on a set of benchmark problems. Algorithms for Verifying Deep Neural Networks serves as a tutorial for students and professionals interested in this emerging field as well as a benchmark to facilitate the design of new verification algorithms.

A Survey of Modern Deep Learning based Object Detection Models

Abstract: Object Detection is the task of classification and localization of objects in an image or video. It has gained prominence in recent years due to its widespread applications. This article surveys recent developments in deep learning based object detectors. Concise overview of benchmark datasets and evaluation metrics used in detection is also provided along with some of the prominent backbone architectures used in recognition tasks. It also covers contemporary lightweight classification models used on edge devices. Lastly, we compare the performances of these architectures on multiple metrics.

Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning

Abstract: Computer-aided detection, localisation, and segmentation methods can help improve colonoscopy procedures. Even though many methods have been built to tackle automatic detection and segmentation of polyps, benchmarking of state-of-the-art methods still remains an open problem. This is due to the increasing number of researched computer vision methods that can be applied to polyp datasets. Benchmarking of novel methods can provide a direction to the development of automated polyp detection and segmentation tasks. Furthermore, it ensures that the produced results in the community are reproducible and provide a fair comparison of developed methods. In this paper, we benchmark several recent state-of-the-art methods using Kvasir-SEG, an open-access dataset of colonoscopy images for polyp detection, localisation, and segmentation evaluating both method accuracy and speed. Whilst, most methods in literature have competitive performance over accuracy, we show that the proposed ColonSegNet achieved a better trade-off between an average precision of 0.8000 and mean IoU of 0.8100, and the fastest speed of 180 frames per second for the detection and localisation task. Likewise, the proposed ColonSegNet achieved a competitive dice coefficient of 0.8206 and the best average speed of 182.38 frames per second for the segmentation task. Our comprehensive comparison with various state-of-the-art methods reveals the importance of benchmarking the deep learning methods for automated real-time polyp identification and delineations that can potentially transform current clinical practices and minimise miss-detection rates.

An optical neural chip for implementing complex-valued neural network.

Abstract: Complex-valued neural networks have many advantages over their real-valued counterparts. Conventional digital electronic computing platforms are incapable of executing truly complex-valued representations and operations. In contrast, optical computing platforms that encode information in both phase and magnitude can execute complex arithmetic by optical interference, offering significantly enhanced computational speed and energy efficiency. However, to date, most demonstrations of optical neural networks still only utilize conventional real-valued frameworks that are designed for digital computers, forfeiting many of the advantages of optical computing such as efficient complex-valued operations. In this article, we highlight an optical neural chip (ONC) that implements truly complex-valued neural networks. We benchmark the performance of our complex-valued ONC in four settings: simple Boolean tasks, species classification of an Iris dataset, classifying nonlinear datasets (Circle and Spiral), and handwriting recognition. Strong learning capabilities (i.e., high accuracy, fast convergence and the capability to construct nonlinear decision boundaries) are achieved by our complex-valued ONC compared to its real-valued counterpart.

Benchmark and Survey of Automated Machine Learning Frameworks

Abstract: Machine learning (ML) has become a vital part in many aspects of our daily life. However, building well performing machine learning applications requires highly specialized data scientists and doma...

A Survey of Evolutionary Algorithms for Multi-Objective Optimization Problems With Irregular Pareto Fronts

Abstract: Evolutionary algorithms have been shown to be very successful in solving multi-objective optimization problems (MOPs). However, their performance often deteriorates when solving MOPs with irregular Pareto fronts. To remedy this issue, a large body of research has been performed in recent years and many new algorithms have been proposed. This paper provides a comprehensive survey of the research on MOPs with irregular Pareto fronts. We start with a brief introduction to the basic concepts, followed by a summary of the benchmark test problems with irregular problems, an analysis of the causes of the irregularity, and real-world optimization problems with irregular Pareto fronts. Then, a taxonomy of the existing methodologies for handling irregular problems is given and representative algorithms are reviewed with a discussion of their strengths and weaknesses. Finally, open challenges are pointed out and a few promising future directions are suggested.

SUPERB: Speech processing Universal PERformance Benchmark

Abstract: Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

Automatic COVID-19 detection from X-ray images using ensemble learning with convolutional neural network

Abstract: COVID-19 continues to have catastrophic effects on the lives of human beings throughout the world. To combat this disease it is necessary to screen the affected patients in a fast and inexpensive way. One of the most viable steps towards achieving this goal is through radiological examination, Chest X-Ray being the most easily available and least expensive option. In this paper, we have proposed a Deep Convolutional Neural Network-based solution which can detect the COVID-19 +ve patients using chest X-Ray images. Multiple state-of-the-art CNN models—DenseNet201, Resnet50V2 and Inceptionv3, have been adopted in the proposed work. They have been trained individually to make independent predictions. Then the models are combined, using a new method of weighted average ensembling technique, to predict a class value. To test the efficacy of the solution we have used publicly available chest X-ray images of COVID +ve and –ve cases. 538 images of COVID +ve patients and 468 images of COVID –ve patients have been divided into training, test and validation sets. The proposed approach gave a classification accuracy of 91.62% which is higher than the state-of-the-art CNN models as well the compared benchmark algorithm. We have developed a GUI-based application for public use. This application can be used on any computer by any medical personnel to detect COVID +ve patients using Chest X-Ray images within a few seconds.

A Dynamic Neighborhood-Based Switching Particle Swarm Optimization Algorithm.

Abstract: In this article, a dynamic-neighborhood-based switching PSO (DNSPSO) algorithm is proposed, where a new velocity updating mechanism is designed to adjust the personal best position and the global best position according to a distance-based dynamic neighborhood to make full use of the population evolution information among the entire swarm. In addition, a novel switching learning strategy is introduced to adaptively select the acceleration coefficients and update the velocity model according to the searching state at each iteration, thereby contributing to a thorough search of the problem space. Furthermore, the differential evolution algorithm is successfully hybridized with the particle swarm optimization (PSO) algorithm to alleviate premature convergence. A series of commonly used benchmark functions (including unimodal, multimodal, and rotated multimodal cases) is utilized to comprehensively evaluate the performance of the DNSPSO algorithm. The experimental results demonstrate that the developed DNSPSO algorithm outperforms a number of existing PSO algorithms in terms of the solution accuracy and convergence performance, especially for complicated multimodal optimization problems.

MOSOA: A new multi-objective seagull optimization algorithm

Abstract: This study introduces the extension of currently developed Seagull Optimization Algorithm (SOA) in terms of multi-objective problems, which is entitled as Multi-objective Seagull Optimization Algorithm (MOSOA). In this algorithm, a concept of dynamic archive is introduced, which has the feature to cache the non-dominated Pareto optimal solutions. The roulette wheel selection approach is utilized to choose the effective archived solutions by simulating the migration and attacking behaviors of seagulls. The proposed algorithm is approved by testing it with twenty-four benchmark test functions, and its performance is compared with existing metaheuristic algorithms. The developed algorithm is analyzed on six constrained problems of engineering design to assess its appropriateness for finding the solutions of real-world problems. The outcomes from the empirical analyzes depict that the proposed algorithm is better than other existing algorithms. The proposed algorithm also considers those Pareto optimal solutions, which demonstrate high convergence.

Self-Supervised Hypergraph Convolutional Networks for Session-based Recommendation

Abstract: Session-based recommendation (SBR) focuses on next-item prediction at a certain time point. As user profiles are generally not available in this scenario, capturing the user intent lying in the item transitions plays a pivotal role. Recent graph neural networks (GNNs) based SBR methods regard the item transitions as pairwise relations, which neglect the complex high-order information among items. Hypergraph provides a natural way to capture beyond-pairwise relations, while its potential for SBR has remained unexplored. In this paper, we fill this gap by modeling session-based data as a hypergraph and then propose a hypergraph convolutional network to improve SBR. Moreover, to enhance hypergraph modeling, we devise another graph convolutional network which is based on the line graph of the hypergraph and then integrate self-supervised learning into the training of the networks by maximizing mutual information between the session representations learned via the two networks, serving as an auxiliary task to improve the recommendation task. Since the two types of networks both are based on hypergraph, which can be seen as two channels for hypergraph modeling, we name our model DHCN (Dual Channel Hypergraph Convolutional Networks). Extensive experiments on three benchmark datasets demonstrate the superiority of our model over the SOTA methods, and the results validate the effectiveness of hypergraph modeling and self-supervised task. The implementation of our model is available via https://github.com/xiaxin1998/DHCN.

Achievable Rate Optimization for MIMO Systems With Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable intelligent surfaces (RISs) represent a new technology that can shape the radio wave propagation in wireless networks and offers a great variety of possible performance and implementation gains Motivated by this, we study the achievable rate optimization for multi-stream multiple-input multiple-output (MIMO) systems equipped with an RIS, and formulate a joint optimization problem of the covariance matrix of the transmitted signal and the RIS elements To solve this problem, we propose an iterative optimization algorithm that is based on the projected gradient method (PGM) We derive the step size that guarantees the convergence of the proposed algorithm and we define a backtracking line search to improve its convergence rate Furthermore, we introduce the total free space path loss (FSPL) ratio of the indirect and direct links as a first-order measure of the applicability of RISs in the considered communication system Simulation results show that the proposed PGM achieves the same achievable rate as a state-of-the-art benchmark scheme, but with a significantly lower computational complexity In addition, we demonstrate that the RIS application is particularly suitable to increase the achievable rate in indoor environments, as even a small number of RIS elements can provide a substantial achievable rate gain

Image Matching Across Wide Baselines: From Paper to Practice

Abstract: We introduce a comprehensive benchmark for local features and robust estimation algorithms, focusing on the downstream task—the accuracy of the reconstructed camera pose—as our primary metric. Our pipeline’s modular structure allows easy integration, configuration, and combination of different methods and heuristics. This is demonstrated by embedding dozens of popular algorithms and evaluating them, from seminal works to the cutting edge of machine learning research. We show that with proper settings, classical solutions may still outperform the perceived state of the art. Besides establishing the actual state of the art, the conducted experiments reveal unexpected properties of structure from motion pipelines that can help improve their performance, for both algorithmic and learned methods. Data and code are online ( https://github.com/ubc-vision/image-matching-benchmark ), providing an easy-to-use and flexible framework for the benchmarking of local features and robust estimation methods, both alongside and against top-performing methods. This work provides a basis for the Image Matching Challenge ( https://image-matching-challenge.github.io ).

Intelligent Reflecting Surface Assisted Mobile Edge Computing for Internet of Things

Abstract: This letter studies the impact of an intelligent reflecting surface (IRS) on computational performance in a mobile edge computing (MEC) system. Specifically, an access point (AP) equipped with an edge server provides MEC services to multiple Internet of Thing (IoT) devices that choose to offload a portion of their own computational tasks to the AP with the remaining portion being locally computed. We deploy an IRS to enhance the computational performance of the MEC system by intelligently adjusting the phase shift of each reflecting element. A joint design problem is formulated for the considered IRS assisted MEC system, aiming to optimize its sum computational bits and taking into account the CPU frequency, the offloading time allocation, transmit power of each device as well as the phase shifts of the IRS. To deal with the non-convexity of the formulated problem, we conduct our algorithm design by finding the optimized phase shifts first and then achieving the jointly optimal solution of the CPU frequency, the transmit power and the offloading time allocation by considering the Lagrange dual method and Karush-Kuhn-Tucker (KKT) conditions. Numerical evaluations highlight the advantage of the IRS-assisted MEC system in comparison with the benchmark schemes.

WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition

Abstract: In this paper, we contribute a new million-scale face benchmark containing noisy 4M identities/260M faces (WebFace260M) and cleaned 2M identities/42M faces (WebFace42M) training data, as well as an elaborately designed time-constrained evaluation protocol. Firstly, we collect 4M name list and download 260M faces from the Internet. Then, a Cleaning Automatically utilizing Self-Training (CAST) pipeline is devised to purify the tremendous WebFace260M, which is efficient and scalable. To the best of our knowledge, the cleaned WebFace42M is the largest public face recognition training set and we expect to close the data gap between academia and industry. Referring to practical scenarios, Face Recognition Under Inference Time conStraint (FRUITS) protocol and a test set are constructed to comprehensively evaluate face matchers.Equipped with this benchmark, we delve into million-scale face recognition problems. A distributed framework is developed to train face recognition models efficiently without tampering with the performance. Empowered by Web-Face42M, we reduce relative 40% failure rate on the challenging IJB-C set, and rank the 3rd among 430 entries on NIST-FRVT. Even 10% data (WebFace4M) shows superior performance compared with public training set. Furthermore, comprehensive baselines are established on our rich-attribute test set under FRUITS-100ms/500ms/1000ms protocol, including MobileNet, EfficientNet, AttentionNet, ResNet, SENet, ResNeXt and RegNet families. Benchmark website is https://www.face-benchmark.org.

Neural Architecture Transfer

Abstract: Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings ( $\leq$ ≤ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to conventional transfer learning of fine-tuning weights of an existing network architecture learned on standard datasets. Code is available at https://github.com/human-analysis/neural-architecture-transfer .

Deep learning in multi-object detection and tracking: state of the art

Abstract: Object detection and tracking is one of the most important and challenging branches in computer vision, and have been widely applied in various fields, such as health-care monitoring, autonomous driving, anomaly detection, and so on. With the rapid development of deep learning (DL) networks and GPU’s computing power, the performance of object detectors and trackers has been greatly improved. To understand the main development status of object detection and tracking pipeline thoroughly, in this survey, we have critically analyzed the existing DL network-based methods of object detection and tracking and described various benchmark datasets. This includes the recent development in granulated DL models. Primarily, we have provided a comprehensive overview of a variety of both generic object detection and specific object detection models. We have enlisted various comparative results for obtaining the best detector, tracker, and their combination. Moreover, we have listed the traditional and new applications of object detection and tracking showing its developmental trends. Finally, challenging issues, including the relevance of granular computing, in the said domain are elaborated as a future scope of research, together with some concerns. An extensive bibliography is also provided.

NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size.

Abstract: Neural architecture search (NAS) has attracted much attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both of those aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms. This facilitates a much larger community of researchers to focus on developing better algorithms in a more comparable environment.

An Enhanced IEEE 33 Bus Benchmark Test System for Distribution System Studies

Abstract: The transformation of passive distribution systems to more active ones thanks to the increased penetration of distributed energy resources, such as dispersed generators, flexible demand, distributed storage, and electric vehicles, creates the necessity of an enhanced test system for distribution systems planning and operation studies. The value of the proposed test system, is that it provides an appropriate and comprehensive benchmark for future researches concerning distribution systems. The proposed test system is developed by modifying and updating the well-known 33 bus distribution system from Baran & Wu. It comprises both forms of balanced and unbalanced three-phase power systems, including new details on the integration of distributed and renewable generation units, reactive power compensation assets, reconfiguration infrastructures and appropriate datasets of load and renewable generation profiles for different case studies.

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Abstract: We introduce DexYCB, a new dataset for capturing hand grasping of objects. We first compare DexYCB with a related one through cross-dataset evaluation. We then present a thorough benchmark of state-of-the-art approaches on three relevant tasks: 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation. Finally, we evaluate a new robotics-relevant task: generating safe robot grasps in human-to-robot object handover. 1

NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search

Abstract: The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical evaluations. As a remedy, several tabular NAS benchmarks were proposed to simulate runs of NAS methods in seconds. However, all existing tabular NAS benchmarks are limited to extremely small architectural spaces since they rely on exhaustive evaluations of the space. This leads to unrealistic results that do not transfer to larger search spaces. To overcome this fundamental limitation, we propose NAS-Bench-301, the first surrogate NAS benchmark, using a search space containing 1018 architectures, many orders of magnitude larger than any previous tabular NAS benchmark. After motivating the benefits of a surrogate benchmark over a tabular one, we fit various regression models on our dataset, which consists of ∼60k architecture evaluations, and build surrogates via deep ensembles to model uncertainty. We benchmark a wide range of NAS algorithms using NAS-Bench-301 and obtain comparable results to the true benchmark at a fraction of the real cost. Finally, we show how NAS-Bench-301 can be used to generate new scientific insights.

GPT Understands, Too

Abstract: While GPTs with traditional fine-tuning fail to achieve strong results on natural language understanding (NLU), we show that GPTs can be better than or comparable to similar-sized BERTs on NLU tasks with a novel method P-tuning -- which employs trainable continuous prompt embeddings. On the knowledge probing (LAMA) benchmark, the best GPT recovers 64\% (P@1) of world knowledge without any additional text provided during test time, which substantially improves the previous best by 20+ percentage points. On the SuperGlue benchmark, GPTs achieve comparable and sometimes better performance to similar-sized BERTs in supervised learning. Importantly, we find that P-tuning also improves BERTs' performance in both few-shot and supervised settings while largely reducing the need for prompt engineering. Consequently, P-tuning outperforms the state-of-the-art approaches on the few-shot SuperGlue benchmark.

Next generation reservoir computing.

Abstract: Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metaparameters, and provides interpretable results. Here, we demonstrate that nonlinear vector autoregression excels at reservoir computing benchmark tasks and requires even shorter training data sets and training time, heralding the next generation of reservoir computing.

EMoSOA: a new evolutionary multi-objective seagull optimization algorithm for global optimization

Abstract: This study introduces the evolutionary multi-objective version of seagull optimization algorithm (SOA), entitled Evolutionary Multi-objective Seagull Optimization Algorithm (EMoSOA). In this algorithm, a dynamic archive concept, grid mechanism, leader selection, and genetic operators are employed with the capability to cache the solutions from the non-dominated Pareto. The roulette-wheel method is employed to find the appropriate archived solutions. The proposed algorithm is tested and compared with state-of-the-art metaheuristic algorithms over twenty-four standard benchmark test functions. Four real-world engineering design problems are validated using proposed EMoSOA algorithm to determine its adequacy. The findings of empirical research indicate that the proposed algorithm is better than other algorithms. It also takes into account those optimal solutions from the Pareto which shows high convergence.

Prediction of COVID-19 Confirmed Cases Combining Deep Learning Methods and Bayesian Optimization.

Abstract: COVID-19 virus has encountered people in the world with numerous problems. Given the negative impacts of COVID-19 on all aspects of people's lives, especially health and economy, accurately forecasting the number of cases infected with this virus can help governments to make accurate decisions on the interventions that must be taken. In this study, we propose three hybrid approaches for forecasting COVID-19 time series methods based on combining three deep learning models such as multi-head attention, long short-term memory (LSTM), and convolutional neural network (CNN) with the Bayesian optimization algorithm. All models are designed based on the multiple-output forecasting strategy, which allows the forecasting of the multiple time points. The Bayesian optimization method automatically selects the best hyperparameters for each model and enhances forecasting performance. Using the publicly available epidemical data acquired from Johns Hopkins University's Coronavirus Resource Center, we conducted our experiments and evaluated the proposed models against the benchmark model. The results of experiments exhibit the superiority of the deep learning models over the benchmark model both for short-term forecasting and long-horizon forecasting. In particular, the mean SMAPE of the best deep learning model is 0.25 for the short-term forecasting (10 days ahead). Also, for long-horizon forecasting, the best deep learning model obtains the mean SMAPE of 2.59.