Showing papers by "Terrance E. Boult published in 2022"

Few-Shot Class Incremental Learning Leveraging Self-Supervised Features

Abstract: Few-Shot Class Incremental Learning (FSCIL) is a recently introduced Class Incremental Learning (CIL) setting that operates under more constrained assumptions: only very few samples per class are available in each incremental session, and the number of samples/classes is known ahead of time. Due to limited data for class incremental learning, FSCIL suffers more from over-fitting and catastrophic forgetting than general CIL. In this paper we study leveraging the advances due to self-supervised learning to remedy overfitting and catastrophic forgetting and significantly advance the state-of-the-art FSCIL. We explore training a lightweight feature fusion plus classifier on a concatenation of features emerging from supervised and self-supervised models. The supervised model is trained on data from a base session, where a relatively larger amount of data is available in FSCIL. Whereas a self-supervised model is learned using an abundance of unlabeled data. We demonstrate a classifier trained on the fusion of such features outperforms classifiers trained independently on either of these representations. We experiment with several existing self-supervised models and provide results for three popular benchmarks for FSCIL including Caltech-UCSD Birds-200-2011 (CUB200), miniImageNet, and CIFAR100 where we advance the state-ofthe-art for each benchmark. Code is available at: https://github.com/TouqeerAhmad/FeSSSS

Variable Few Shot Class Incremental and Open World Learning

Abstract: Prior work on few-shot class incremental learning has operated with an unnatural assumption: the number of ways and number of shots are assumed to be known and fixed e.g., 10-ways 5-shots, 5-ways 5-shots, etc. Hence, we refer to this setting as Fixed-Few-Shot Class Incremental Learning (FFSCIL). In practice, the pre-specified fixed number of classes and examples per class may not be available, meaning one cannot update the model. Evaluation of FSCIL approaches in such unnatural settings renders their applicability questionable for practical scenarios where such assumptions do not hold. To mitigate the limitation of FFSCIL, we propose Variable-Few-Shot Class Incremental Learning (VFSCIL) and demonstrate it with Up-to N-Ways, Up-to K-Shots class incremental learning; wherein each incremental session, a learner may have up to N classes and up to K samples per class. Consequently, conventional FFSCIL is a special case of herein introduced VFSCIL. Further, we extend VFSCIL to a more practical problem of Variable-Few-Shot Open-World Learning (VFSOWL), where an agent is not only required to perform incremental learning, but must detect unknown samples and enroll only those that it detects correctly. We formulate and study VFSCIL and VFSOWL on two benchmark datasets conventionally employed for FFSCIL i.e., Caltech-UCSD Birds-200-2011 (CUB200) and miniImageNet. First, to serve as a strong baseline, we extend the state-of-the-art FSCIL approach to operate in Up-to N-Ways, Up-to K-Shots class incremental and open-world settings. Then, we propose a novel but simple approach for VFSCIL/VFSOWL where we leverage the current advancements in self-supervised feature learning. Utilizing both benchmark datasets, our proposed approach outperforms the strong baseline on the conventional FFSCIL setting and newly introduced VFSCIL/VFSOWL settings. Our code is available at: https://github.com/TouqeerAhmad/VFSOWL

Agglomerative Clustering with Threshold Optimization via Extreme Value Theory

Abstract: Clustering is a critical part of many tasks and, in most applications, the number of clusters in the data are unknown and must be estimated. This paper presents an Extreme Value Theory-based approach to threshold selection for clustering, proving that the “correct” linkage distances must follow a Weibull distribution for smooth feature spaces. Deep networks and their associated deep features have transformed many aspects of learning, and this paper shows they are consistent with our extreme-linkage theory and provide Unreasonable Clusterability. We show how our novel threshold selection can be applied to both classic agglomerative clustering and the more recent FINCH (First Integer Neighbor Clustering Hierarchy) algorithm. Our evaluation utilizes over a dozen different large-scale vision datasets/subsets, including multiple face-clustering datasets and ImageNet for both in-domain and, more importantly, out-of-domain object clustering. Across multiple deep features clustering tasks with very different characteristics, our novel automated threshold selection performs well, often outperforming state-of-the-art clustering techniques even when they select parameters on the test set.

Carbon-Neutral Distributed Ledger

Abstract: This work introduces a carbon-neutral approach for Distributed Ledger. With computationally intensive consensus models creating exceptional levels of wasted energy worldwide, it is crucial that distributed ledger technology progresses in a direction that alleviates this trend. This work offers both a novel consensus model and a user incentive that departs from the usage of computational processes for consensus. This work considers the need for longevity of distributed ledger and reflects the desires of the different user archetypes engaged in the recent upswing of cryptocurrency. We anticipate that some users engage for investment or growth speculation, some engage for transactional purposes, and others engage for analytics and transaction verification. We introduce a sustainable distributed ledger that provides engagement considerations for each archetype while eliminating wasteful side effects of computational proof-of-work algorithms. The incentive for contribution included for this novel distributed ledger is based on participation over time. This work shows the ability to incentivize each of the archetypes aligned for the purpose of maintaining ledger data and transactions over time. In the model introduced, the key user engagement focus is redirected from computational waste found in current cryptocurrencies to a model that incentivizes constant uptime to receive tokens released for use over time. In addition, we demonstrate that current incentive models which result in global energy waste are converted to a user incentive where all rewards are aligned to maintain continually available data storage.

Consistency and Accuracy of CelebA Attribute Values

Abstract: We report the first systematic analysis of the experimental foundations of facial attribute classification. Two annotators independently assigning attribute values shows that only 12 of 40 common attributes are assigned values with>= 95% consistency, and three (high cheekbones, pointed nose, oval face) have essentially random consistency. Of 5,068 duplicate face appearances in CelebA, attributes have contradicting values on from 10 to 860 of the 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with>= 95% consistency. Selecting the mouth slightly open (MSO) for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and (MSO=false) at about 2%. A corrected version of the MSO attribute values enables learning a model that achieves higher accuracy than previously reported for MSO. Corrected values for CelebA MSO are available at https://github.com/HaiyuWu/CelebAMSO.

On the Effect of Atmospheric Turbulence in the Feature Space of Deep Face Recognition

Abstract: When captured over long distances, image quality is degraded by inconsistent refractive indexes in the atmosphere. This effect, known as Atmospheric Turbulence (AT), leads to lower performance for vision-based biometric systems such as face recognition. To account for AT, the literature has proposed methods to restore face-images from atmospheric turbulence, but has limited success. There is still a need to understand how atmospheric turbulence breaks recognition performance. We offer a first-look in this direction by providing a study on the effect of atmospheric turbulence in the feature space of deep-learning-based face recognition. We present results on recognition performance and feature space transformation caused by a wide range of AT levels. In deep feature space, we find interesting phenomena such as increasing feature magnitudes, which contradicts the expected result from the literature. From our results, we are able to identify an effect that makes face recognition under atmospheric turbulence uniquely difficult, which we call feature defection. In total, our findings suggest several areas of available improvement which can be used as a guideline for further progress in building models that are robust to AT.

Enhanced Performance of Pre-Trained Networks by Matched Augmentation Distributions

Abstract: There exists a distribution discrepancy between training and testing, in the way images are fed to modern CNNs. Recent work tried to bridge this gap either by fine-tuning or re-training the network at different resolutions. However retraining a network is rarely cheap and not always viable. To this end, we propose a simple solution to address the train-test distributional shift and enhance the performance of pretrained models - which commonly ship as a package with deep learning platforms e.g., PyTorch. Specifically, we demonstrate that running inference on the center crop of an image is not always the best as important discriminatory information may be cropped-off. Instead we propose to combine results for multiple random crops for a test image. This not only matches the train time augmentation but also provides the full coverage of the input image. We explore combining representation of random crops through averaging at different levels i.e., deep feature level, logit level, and softmax level. We demonstrate that, for various families of modern deep networks, such averaging results in better validation accuracy compared to using a single central crop per image. The softmax averaging results in the best performance for various pre-trained networks without requiring any re-training or fine-tuning whatsoever. On modern GPUs with batch processing, the paper's approach to inference of pre-trained networks, is essentially free as all images in a batch can all be processed at once. Our code is available at: https://github.com/TouqeerAhmad/MID

Weibull-Open-World (WOW) Multi-Type Novelty Detection in CartPole3D

Abstract: Algorithms for automated novelty detection and management are of growing interest but must address the inherent uncertainty from variations in non-novel environments while detecting the changes from the novelty. This paper expands on a recent unified framework to develop an operational theory for novelty that includes multiple (sub)types of novelty. As an example, this paper explores the problem of multi-type novelty detection in a 3D version of CartPole, wherein the cart Weibull-Open-World control-agent (WOW-agent) is confronted by different sub-types/levels of novelty from multiple independent agents moving in the environment. The WOW-agent must balance the pole and detect and characterize the novelties while adapting to maintain that balance. The approach develops static, dynamic, and prediction-error measures of dissimilarity to address different signals/sources of novelty. The WOW-agent uses the Extreme Value Theory, applied per dimension of the dissimilarity measures, to detect outliers and combines different dimensions to characterize the novelty. In blind/sequestered testing, the system detects nearly 100% of the non-nuisance novelties, detects many nuisance novelties, and shows it is better than novelty detection using a Gaussian-based approach. We also show the WOW-agent’s lookahead collision avoiding control is significantly better than a baseline Deep-Q-learning Networktrained controller.

Overleaf Example

Abstract: We report the first analysis of the experimental foundations of facial attribute classification. An experiment with two annotators independently assigning values shows that only 12 of 40 commonly-used attributes are assigned values with ≥ 95% consistency, and that three (high cheekbones, pointed nose, oval face) have random consistency (50%). These results show that the binary face attributes currently used in this research area could re-focused to be more objective. We identify 5,068 duplicate face appearances in CelebA, the most widely used dataset in this research area, and find that individual attributes have contradicting values on from 10 to 860 of 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with ≥ 95% consistency. Selecting the mouth slightly open (MSO) attribute for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and for (MSO=false) at about 2%. We create a corrected version of the MSO attribute values, and compare classification models created using the original versus corrected values. The corrected values enable a model that achieves higher accuracy than has been previously reported for MSO. Also, ScoreCAM visualizations show that the model created using the corrected attribute values is in fact more focused on the mouth region of the face. These results show that the error rate in the current CelebA attribute values should be reduced in order to enable learning of better models. The corrected attribute values for CelebA’s MSO and the CelebA facial hair attributes will be made available upon publication.