We propose a novel Generative Adversarial Network (XingGAN or CrossingGAN) for person image generation tasks, i.e., translating the pose of a given person to a desired one. The proposed Xing generator consists of two generation branches that model the person’s appearance and shape information, respectively. Moreover, we propose two novel blocks to effectively transfer and update the person’s shape and appearance embeddings in a crossing way to mutually improve each other, which has not been considered by any other existing GAN-based image generation work. Extensive experiments on two challenging datasets, i.e., Market-1501 and DeepFashion, demonstrate that the proposed XingGAN advances the state-of-the-art performance both in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/XingGAN.

XingGAN for Person Image Generation

Vanilla convolutions in modern deep networks are known to operate locally and at fixed scale (e.g., the widely-adopted 3× 3 kernels in image-oriented tasks). This causes low efficacy in connecting two distant locations in the network. In this work, we propose a novel convolutional operator dubbed as fast Fourier convolution (FFC), which has the main hallmarks of non-local receptive fields and cross-scale fusion within the convolutional unit. According to spectral convolution theorem in Fourier theory, point-wise update in the spectral domain globally affects all input features involved in Fourier transform, which sheds light on neural architectural design with non-local receptive field. Our proposed FFC is inspired to capsulate three different kinds of computations in a single operation unit: a local branch that conducts ordinary small-kernel convolution, a semi-global branch that processes spectrally stacked image patches, and a global branch that manipulates image-level spectrum. All branches complementarily address different scales. A multi-branch aggregation step is included in FFC for cross-scale fusion. FFC is a generic operator that can directly replace vanilla convolutions in a large body of existing networks, without any adjustments and with comparable complexity metrics (e.g., FLOPs). We experimentally evaluate FFC in three major vision benchmarks (ImageNet for image recognition, Kinetics for video action recognition, MSCOCO for human keypoint detection). It consistently elevates accuracies in all above tasks by significant margins.

/pdf/fast-fourier-convolution-20fo726w8i.pdf

Fast Fourier Convolution

Recently, device-free human behavior recognition has become a hot research topic and has achieved significant progress in the field of ubiquitous computing. Among various implementation, behavior recognition based on WiFi CSI (channel state information) has drawn wide attention due to its major advantages. This paper investigates more than 100 latest CSI based behavior recognition applications within the last 6 years and presents a comprehensive survey from every aspect of human behavior recognition. Firstly, this paper reviews general behavior recognition applications using the WiFi signal and presents the basic concept of CSI and the fundamental principle of CSI-based behavior recognition. This paper analyzes the key components and core characteristics of the system architecture of human behavior recognition using CSI. Afterward, we divide the sensing procedures into many steps and summarize the typical studies from these steps, including base signal selection, signal preprocessing, and identification approaches. Next, based on the recognition technique, we classify the applications into three groups, including pattern-based, model-based, and deep learning-based approach. In every group, we categorize the state-of-the-art applications into three groups, including coarse-grained specific behavior recognition, fine-grained specific behavior recognition, and activity inference. It elaborates the typical behavior recognition applications from five aspects, including experimental equipment, experimental scenario, behavior, classifier, and system performance. Then, this paper presents comprehensive discussions about representative applications from the implementation view and outlines the major consideration when developing a recognition system. Finally, this article concludes by analyzing the open issues of CSI-based behavior recognition applications and pointing out future research directions.

/pdf/a-survey-on-human-behavior-recognition-using-channel-state-17xs5detky.pdf

A Survey on Human Behavior Recognition Using Channel State Information

In recent years, wireless sensing has been exploited as a promising research direction for contactless human activity recognition. However, one major issue hindering the real deployment of these systems is that the signal variation patterns induced by the human activities with different devices and environmental settings are neither stable nor consistent, resulting in unstable system performance. The existing machine learning based methods usually take the "black box" approach and fails to achieve consistent performance. In this paper, we argue that a deep understanding of radio signal propagation in wireless sensing is needed, and it may be possible to develop a deterministic sensing model to make the signal variation patterns predictable. With this intuition, in this paper we investigate: 1) how wireless signals are affected by human activities taking transceiver location and environment settings into consideration; 2) a new deterministic sensing approach to model the received signal variation patterns for different human activities; 3) a proof-of-concept prototype to demonstrate our approach and a case study to detect diverse activities. In particular, we propose a diffraction-based sensing model to quantitatively determine the signal change with respect to a target's motions, which eventually links signal variation patterns with motions, and hence can be used to recognize human activities. Through our case study, we demonstrate that the diffraction-based sensing model is effective and robust in recognizing exercises and daily activities. In addition, we demonstrate that the proposed model improves the recognition accuracy of existing machine learning systems by above 10%.

Towards a Diffraction-based Sensing Approach on Human Activity Recognition

Wireless networks have been widely deployed with a high demand for wireless data traffic. The ubiquitous availability of wireless signals brings new opportunities for non-intrusive human activity sensing. To enhance a thorough understanding of existing wireless sensing techniques and provide insights for future directions, this survey conducts a review of the existing research on human activity sensing with wireless signals. We review and compare existing research of wireless human activity sensing from seven perspectives, including the types of wireless signals, theoretical models, signal preprocessing techniques, activity segmentation, feature extraction, classification, and application. With the development and deployment of new wireless technology, there will be more sensing opportunities in human activities. Based on the analysis of existing research, the survey points out seven challenges on wireless human activity sensing research: robustness, non-coexistence of sensing and communications, privacy, multiple user activity sensing, limited sensing range, complex deep learning, and lack of standard datasets. Finally, this survey presents four possible future research trends, including new theoretical models, the coexistence of sensing and communications, awareness of sensing on receivers, and constructing open datasets to enable new wireless sensing opportunities on human activities.

Human Activity Sensing with Wireless Signals: A Survey.

Zero-shot image segmentation refers to the task of segmenting pixels from specific unseen semantic class. Previous methods mainly rely on historic segmentation tasks, such as using semantic embedding or word embedding of class names to infer a new segmentation model. In this work we describe Cap2Seg, a novel solution of zero-shot image segmentation that harnesses accompanying image captions for intelligently inferring spatial and semantic context for the zero-shot image segmentation task. As our main insight, image captions often implicitly entail the occurrence of a new class in an image and its most-confident spatial distribution. We define a contextual entailment question (CEQ) that tailors BERT-like text models. In specific, the proposed networks for inferring unseen classes consists of three branches (global / local / semi-global), which infer labels of unseen class from image level, image-stripe level or pixel level respectively. Comprehensive experiments and ablation studies are conducted on two image benchmarks, COCO-stuff and Pascal VOC. All clearly demonstrate the effectiveness of the proposed Cap2Seg, including a set of hardest unseen classes (i.e., image captions do not literally contain the class names and direct matching for inference fails).

Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

Bodyweight exercises are effective and efficient ways to improve one's balance, flexibility, and strength without machinery or extra equipment. Prior works have been successful in monitoring aerobic exercises and free-weight exercises, but are not suitable for ubiquitous bodyweight exercise monitoring in order to provide fine-grained repetition counting information in each exercise set. In this work, we propose WiFit, a bodyweight exercises monitoring system that supports accurate repetition counting using a pair of commodity Wi-Fi devices without attaching anything to the human body. We first analyze the movement patterns of bodyweight exercises and couple them with detailed Doppler effect modeling to determine the most effective system settings. Then, by leveraging the human activity Doppler displacement stream extracted from Wi-Fi CSI signal, we have developed an impulse-based method to segment and count the number of repetitions, and analyzed specific features for classifying different types of bodyweight exercises. Extensive experiments show that WiFit achieves 99% accuracy for repetition counting and 95.8% accuracy for exercise type classification.

WiFit: Ubiquitous Bodyweight Exercise Monitoring with Commodity Wi-Fi Devices

Fusing multi-modality information is known to be able to effectively bring significant improvement in video classification. However, the most popular method up to now is still simply fusing each stream's prediction scores at the last stage. A valid question is whether there exists a more effective method to fuse information cross modality. With the development of attention mechanism in natural language processing, there emerge many successful applications of attention in the field of computer vision. In this paper, we propose a cross-modality attention operation, which can obtain information from other modality in a more effective way than two-stream. Correspondingly we implement a compatible block named CMA block, which is a wrapper of our proposed attention operation. CMA can be plugged into many existing architectures. In the experiments, we comprehensively compare our method with two-stream and non-local models widely used in video classification. All experiments clearly demonstrate strong performance superiority by our proposed method. We also analyze the advantages of the CMA block by visualizing the attention map, which intuitively shows how the block helps the final prediction.

/pdf/two-stream-video-classification-with-cross-modality-2xm73rcodp.pdf

Two-Stream Video Classification with Cross-Modality Attention

Effectively modeling long-range spatial correlation is crucial in context-sensitive visual computing tasks, such as human pose estimation and video classification. Enlarging receptive field is popularly adopted in building such non-local deep networks. However, current solutions, including dilation convolution or self-attention based operators, mostly suffer from either low computational efficacy or insufficient receptive field. This paper proposes spectral residual learning (SRL), a novel network architectural design for achieving fully global receptive field. A neural block that implements SRL has three key components: a local-to-global transform that projects some ordinary local features into a spectral domain, compiled operations in the spectral domain, and a global-to-local transform that converts all data back to the original local format. We show its equivalence to conducting residual learning in some spectral domain and carefully re-formulate a variety of neural layers into their spectral forms, such as ReLU or convolutions. The benefits of SRL is three-fold: first, all operations have global receptive field, namely any update affects all image positions. This can extract richer context information in various vision tasks; Secondly, the local-to-global / global-to-local transforms in SRL are defined by bi-linear unitary matrices, which is both computation and parameter economic; Lastly, SRL is a generic formulation, here instantiated by Fourier transform and real orthogonal matrix. We conduct comprehensive evaluations on two challenging tasks, including human pose estimation from images and video classification. All experiments clearly show performance improvement by large margins in comparison with conventional non-local network designs.

Guiyu Tian

Papers

Cap2Seg: Inferring Semantic and Spatial Context from Captions for Zero-Shot Image Segmentation

WiFit: Ubiquitous Bodyweight Exercise Monitoring with Commodity Wi-Fi Devices

Two-Stream Video Classification with Cross-Modality Attention

Fast Non-Local Neural Networks with Spectral Residual Learning

Two-Stream Video Classification with Cross-Modality Attention