How do humans recognize the action "opening a book" ? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects. In this paper, we propose to represent videos as space-time region graphs which capture these two important cues. Our graph nodes are defined by the object region proposals from different frames in a long range video. These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects. We perform reasoning on this graph representation via Graph Convolutional Networks. We achieve state-of-the-art results on both Charades and Something-Something datasets. Especially for Charades, we obtain a huge 4.4% gain when our model is applied in complex environments.

Videos as Space-Time Region Graphs.

The rapid development of remote sensing and space technology provides multisource remote sensing image data for earth observation in the same area. Information provided by these images, however, is often complementary and cooperative, and multisource image fusion is still challenging. This paper proposes a novel multisource remote sensing image fusion algorithm. It integrates the contrast saliency map (CSM) and the sum-modified-Laplacian (SML) in the nonsubsampled shearlet transform (NSST) domain. The NSST is utilized to decompose the source images into low-frequency sub-bands and high-frequency sub-bands. Low-frequency sub-bands reflect the contrast and brightness of the source images, while high-frequency sub-bands reflect the texture and details of the source images. Using this information, the contrast saliency map and SML fusion rules are introduced into the corresponding sub-bands. Finally, the inverse NSST reconstructs the fusion image. Experimental results demonstrate that the proposed multisource remote image fusion technique performs well in terms of contrast enhancement and detail preservation.

https://www.mdpi.com/1424-8220/21/5/1756/pdf

Saliency-Guided Nonsubsampled Shearlet Transform for Multisource Remote Sensing Image Fusion.

ABSTRACT Due to the randomness of weather conditions, the complexity of power demands and the nonlinearity of high-dimensional data, traditional regression analysis, and time series forecasting methods can no longer meet the current requirements of high-precision wind power load forecasting. Therefore, it is necessary to use an accurate short-term load forecasting method for power grid dispatching and planning. Taking the onshore wind power load data of Valencia, Spain (from January 1, 2015, to December 31, 2018), as the research object, this paper proposes a hybrid model of bidirectional gated recurrent units and squeeze-and-excitation networks (BiGRU-SENet) based on an attention mechanism, which is good at handling the nonlinearity of high-dimensional time series data, to predict short-term power loads. The results showed that (1) the attention mechanism could significantly improve the predictive performance of BiGRU by weighting the input of different time steps. (2) The GRU-CNN was more suitable for short-term load forecasting than CNN-GRU, which could make full use of time series information and not lose the key information at the beginning. (3) SENet enabled CNN to perform feature-wise recalibration, further focusing on effective features. (4) Compared with the second-best model (BiGRU-Attention-CNN), BiGRU-Attention-SENet was 12.849% and 3.096% lower in terms of the mean absolute percentage error (MAPE) and root mean square error (RMSE), respectively. In terms of computational overheads, BiGRU-Attention-SENet achieved greater improvement in prediction accuracy by adding relatively fewer computational overheads (the parameters increased by 0.338%, and the FLOPs increased by 0.621%). (5) Through fine tuning, the pre-trained model achieved almost the same accuracy as the no pre-trained one in a shorter time, which proved that BiGRU-Attention-SENet had strong generalization ability. The hybrid model proposed in this paper has application prospects in power load forecasting.

Short-term power load forecasting based on BiGRU-Attention-SENet model

A comprehensive framework of multiple semantics preservation in neural style transfer

Image segmentation is an important step in many computer vision and image processing algorithms. It is often adopted in tasks such as object detection, classification, and tracking. The segmentation of underwater images is a challenging problem as the water and particles present in the water scatter and absorb the light rays. These effects make the application of traditional segmentation methods cumbersome. Besides that, to use the state-of-the-art segmentation methods to face this problem, which are based on deep learning, an underwater image segmentation dataset must be proposed. So, in this paper, we develop a dataset of real underwater images, and some other combinations using simulated data, to allow the training of two of the best deep learning segmentation architectures, aiming to deal with segmentation of underwater images in the wild. In addition to models trained in these datasets, fine-tuning and image restoration strategies are explored too. To do a more meaningful evaluation, all the models are compared in the testing set of real underwater images. We show that methods obtain impressive results, mainly when trained with our real dataset, comparing with manually segmented ground truth, even using a relatively small number of labeled underwater training images.

/pdf/underwater-image-segmentation-in-the-wild-using-deep-4a4kdidyc4.pdf

Underwater image segmentation in the wild using deep learning

Behavior recognition is a fundamental yet challenging task in intelligent surveillance system, which plays an increasingly important role in the process of “Industry 4.0”. However, monitoring the workflow of both workers and machines in production procedure is quite difficult in complex industrial environments. In this paper, we propose a novel workflow recognition framework to recognize the behavior of working subjects based on the well-designed encoder-decoder structure. Namely, attention-based workflow recognition framework, termed as AWR. To improve the accuracy of workflow recognition, a temporal attention cell (AttCell) is introduced to draw dynamic attention distribution in the last stage of the framework. In addition, a Rough-to-Refine phase localization model is exploited to improve localization accuracy, which can effectively identify the boundaries of a specific phase instance in long untrimmed videos. Comprehensive experiments indicate a 1.4% mAP@IoU= 0.4 boost on THUMOS’14 dataset and a 3.4% mAP@IoU= 0.4 boost on hand-crafted workflow dataset detection challenge compared to the advanced GTAN pipeline respectively. More remarkably, the effectiveness of the workflow recognition system is validated in a real-world production scenario.

Attention-based encoder-decoder networks for workflow recognition

In the process of “Industry 4.0”, video analysis plays a vital role in a variety of industrial applications. Video-based action detection has obtained promising performance in computer vision community. However, in complex factory environment, how to detect workflow of both machines and workers in production process is not well resolved. To solve this issue, we propose a generic proposal based Graph Attention Networks for workflow detection. Specifically, an efficient and effective action proposal method is firstly employed to generate workflow proposals. Then, these proposals and their relations are exploited for proposal graph construction. Here, two types of relationships are considered for identifying the workflow phases, which are contextual and surrounding relations to capture context information and characterize the correlations between different workflow instances. To improve the recognition accuracy, within-category and between-category attention are incorporated to learn long-range and dynamic dependencies respectively. Thus, the capability of feature representation for workflow detection can be greatly enhanced. Experimental results verify that the proposed approach is considerably improved upon the state-of-the-arts on THUMOS’14 and a practical workflow dataset, achieving 6.7% and 3.9% absolute improvement compared to the advanced GTAN detector at tIoU threshold 0.4, respectively. Moreover, augmentation experiments are carried out on ActivityNet1.3 to prove the effectiveness of performance improvement by modeling workflow proposal relationships.

Proposal-Based Graph Attention Networks for Workflow Detection

Action detection with two-stream enhanced detector

Precisely localizing temporal intervals for each action segment in long raw videos is essential challenge in practical video content analysis (e.g., activity detection or video caption generation). Most of previous works often neglect the hierarchical action granularity and eventually fail to identify precise action boundaries. (e.g., embracing approaching or turning a screw in mechanical maintenance). In this paper, we introduce a simple yet efficient coarse-to-fine network (CFNet) to solve the challenging issue of temporal action localization by progressively refining action boundary at multiple action granularities. The proposed CFNet is mainly composed of three components: a coarse proposal module (CPM) to generate coarse action candidates, a fusion block (FB) to enhance feature representation by fusing the coarse candidate features and corresponding features of raw input frames, and a boundary transformer module (BTM) to further refine action boundaries. Specifically, CPM exploits framewise, matching and gated actionness curves to complement each other for coarse candidate generation at different levels, while FB is devised to enrich feature representation by fusing the last feature map of CPM and corresponding raw frame input. Finally, BTM learns long-term temporal dependency with a transformer structure to further refine action boundaries at a finer granularity. Thus, the fine-grained action intervals can be incrementally obtained. Compared with previous state-of-the-art techniques, the proposed coarse-to-fine network can asymptotically approach fine-grained action boundary. Comprehensive experiments are conducted on both publicly available THUMOS14 and ActivityNet-v1.3 datasets, and show the outstanding improvements of our method when compared with the prior methods on various video action parsing tasks.

https://ieeexplore.ieee.org/ielx7/6287639/6514899/09885204.pdf

Min Zhang

Papers

Attention-based encoder-decoder networks for workflow recognition

Proposal-Based Graph Attention Networks for Workflow Detection

Action detection with two-stream enhanced detector

Temporal Action Localization With Coarse-to-Fine Network