Understanding animals' behaviors is significant for a wide range of applications. However, existing animal behavior datasets have limitations in multiple aspects, including limited numbers of animal classes, data samples and provided tasks, and also limited variations in environmental conditions and viewpoints. To address these limitations, we create a large and diverse dataset, Animal Kingdom, that provides multiple annotated tasks to enable a more thorough understanding of natural animal behaviors. The wild animal footages used in our dataset record different times of the day in extensive range of environments containing variations in backgrounds, viewpoints, illumination and weather conditions. More specifically, our dataset contains 50 hours of annotated videos to localize relevant animal behavior segments in long videos for the video grounding task, 30K video sequences for the fine-grained multi-label action recognition task, and 33K frames for the pose estimation task, which correspond to a diverse range of animals with 850 species across 6 major animal classes. Such a challenging and comprehensive dataset shall be able to facilitate the community to develop, adapt, and evaluate various types of advanced methods for animal behavior analysis. Moreover, we propose a Collaborative Action Recognition (CARe) model that learns general and specific features for action recognition with unseen new animals. This method achieves promising performance in our experiments. Our dataset can be found at https://sutdcv.github.io/Animal-Kingdom.

Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understanding

Multiple fish tracking in unconstrained marine videos is a highly challenging task. Trajectories of fishes convey critical information for the analysis of fish behavior. In this article, we have proposed deep fish tracking network (DFTNet) that incorporates Siamese network for encoding the appearance similarity and attention long short-term memory network to capture the motion similarity across subsequent frames. Finally, intersection-over-union matching score is computed to amalgamate spatial similarity cue in the final score. The proposed framework can provide joint optimization score to maintain the tracklet information encoding appearance, motion, and spatial similarity cues. We perform exhaustive experiments and compare the proposed approach with competing techniques over Fish4knowledge videos and achieve significant average reduction in identification (ID) switches by 60.9%. The source code is made publicly available at https://github.com/hemanth-s17/Deep-Fish-Tracker-Network .

DFTNet: Deep Fish Tracker With Attention Mechanism in Unconstrained Marine Environments

Tracking and grasping moving objects are hot topics in the field of robots, which provides great potential in the industrial scene and human–computer cooperation. Based on the kernel correlation filter and vision 3-D reconstruction, this article proposes a visual-based tracking and grasping method for moving targets. An improved algorithm based on the kernel correlation filter is proposed for object tracking. A scale pool is constructed, and a scale filter is trained to solve the problem of algorithm scale adaptation. At the same time, the judgment mechanism of tracking results, secondary detection, and modification update mechanisms are added to improve the robustness of the algorithm. Combined with an RGB-D camera, the target is reconstructed to obtain the 3-D pose of the target. The tracking and intercepting strategy is adopted to grasp the moving target. The proposed method is proven to have good performance through dataset comparison tests and experiments on real robot systems.

Improved Kernel Correlation Filter Based Moving Target Tracking for Robot Grasping

RGB and thermal infrared (RGBT) tracking as a solution in complex environments has gradually become a research hotspot. The powerful complementarity between RGB and thermal infrared data enables trackers to work 24/7. Existing works usually adopt the symmetric network structure that deploys the identical strategy to mine modalities with different properties, ignoring the heterogeneity among modalities. In this article, we propose a novel asymmetric global–local mutual integration network via comprehensively considering symmetric structure, heterogeneity-based global association, and interframe communication. It consists of asymmetric mode-distinguishing parallel structure (AMPS), cross-modal global–local interaction, and interframe monitoring strategy (IMS). Specifically, the AMPS performs discriminative mining on the information of the two modalities by combining the discount module and the branch cement module, and extracts multiscale cues through the multiscale auxiliary module to handle the challenges of scale variation and small-size objects. Then, the global mining module is deployed in the cross-modal global–local interaction section to jointly perform intramodal and intermodal global correlation while acting as the global complement to local feature extraction. Finally, the IMS employs a fast optical flow algorithm to detect interframe displacement to assist the network in better handling camera and fast object motion. Extensive experiments on GTOT, RGBT234, and LasHeR datasets adequately verify the effectiveness of the proposed network, and further ablation experiments also confirm the efficacy of the asymmetric structure and components.

Asymmetric Global–Local Mutual Integration Network for RGBT Tracking

Through the advancement of observation systems, our vision has far extended its reach into the world of fishes, and how they interact with fishing gears—breaking through physical boundaries and visually adapting to challenging conditions in marine environments. As marine sciences step into the era of artificial intelligence (AI), deep learning models now provide tools for researchers to process a large amount of imagery data (i.e., image sequence, video) on fish behavior in a more time-efficient and cost-effective manner. The latest AI models to detect fish and categorize species are now reaching human-like accuracy. Nevertheless, robust tools to track fish movements in situ are under development and primarily focused on tropical species. Data to accurately interpret fish interactions with fishing gears is still lacking, especially for temperate fishes. At the same time, this is an essential step for selectivity studies to advance and integrate AI methods in assessing the effectiveness of modified gears. We here conduct a bibliometric analysis to review the recent advances and applications of AI in automated tools for fish tracking, classification, and behavior recognition, highlighting how they may ultimately help improve gear selectivity. We further show how transforming external stimuli that influence fish behavior, such as sensory cues and gears as background, into interpretable features that models learn to distinguish remains challenging. By presenting the recent advances in AI on fish behavior applied to fishing gear improvements (e.g., Long Short-Term Memory (LSTM), Generative Adversarial Network (GAN), coupled networks), we discuss the advances, potential and limits of AI to help meet the demands of fishing policies and sustainable goals, as scientists and developers continue to collaborate in building the database needed to train deep learning models.

/pdf/artificial-intelligence-for-fish-behavior-recognition-may-2aua99kl.pdf

Artificial intelligence for fish behavior recognition may unlock fishing gear selectivity

Optical Flow is a popular method of computer vision for motion estimation. In this paper, we present a refined optical flow estimation method. Central to our approach is exploiting contour information as most of the motion lies on the edges. Further, we have formulated it as sparse to dense motion estimation. Proposed method has been evaluated on challenging real life image sequences of KITTI and Fish4Knowledge database. Results demonstrate that method performs well in case of low contrast, highly cluttered background, dynamic background, occlusion and illumination change.

Shilpi Gupta

Papers

DFTNet: Deep Fish Tracker With Attention Mechanism in Unconstrained Marine Environments

U-RME: Underwater Refined Motion Estimation in Hazy, Cluttered and Dynamic Environments