Learning 3D global features by aggregating multiple views is important. Pooling is widely used to aggregate views in deep learning models. However, pooling disregards a lot of content information within views and the spatial relationship among the views, which limits the discriminability of learned features. To resolve this issue, 3D to Sequential Views (3D2SeqViews) is proposed to more effectively aggregate the sequential views using convolutional neural networks with a novel hierarchical attention aggregation. Specifically, the content information within each view is first encoded. Then, the encoded view content information and the sequential spatiality among the views are simultaneously aggregated by the hierarchical attention aggregation, where view-level attention and class-level attention are proposed to hierarchically weight sequential views and shape classes. View-level attention is learned to indicate how much attention is paid to each view by each shape class, which subsequently weights sequential views through a novel recursive view integration. Recursive view integration learns the semantic meaning of view sequence, which is robust to the first view position. Furthermore, class-level attention is introduced to describe how much attention is paid to each shape class, which innovatively employs the discriminative ability of the fine-tuned network. 3D2SeqViews learns more discriminative features than the state-of-the-art, which leads to the outperforming results in shape classification and retrieval under three large-scale benchmarks.

3D2SeqViews: Aggregating Sequential Views for 3D Global Feature Learning by CNN With Hierarchical Attention Aggregation

With the availability of low-cost and compact 2.5/3D visual sensing devices, computer vision community is experiencing a growing interest in visual scene understanding of indoor environments. This survey paper provides a comprehensive background to this research topic. We begin with a historical perspective, followed by a popular 3D data representation and a comparative analysis of available datasets. Before delving into the application specific details, this survey provides a succinct introduction to the core technologies that are the underlying methods extensively used in this paper. Afterwards, we review the developed techniques according to a taxonomy based on the scene understanding tasks. This covers holistic indoor scene understanding as well as subtasks, such as scene classification, object detection, pose estimation, semantic segmentation, 3D reconstruction, saliency detection, physics-based reasoning, and affordance prediction. Later on, we summarize the performance metrics used for evaluation in different tasks and a quantitative comparison among the recent state-of-the-art techniques. We conclude this review with the current challenges and an outlook toward the open research problems requiring further investigation.

/pdf/indoor-scene-understanding-in-2-5-3d-for-autonomous-agents-a-3uvenfwezk.pdf

Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey

A multi-view recurrent neural network for 3D mesh segmentation

We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.

Language-driven synthesis of 3D scenes from scene databases

Rapid development of artificial intelligence motivates researchers to expand the capabilities of intelligent and autonomous robots. In many robotic applications, robots are required to make planning decisions based on perceptual information to achieve diverse goals in an efficient and effective way. The planning problem has been investigated in active robot vision, in which a robot analyzes its environment and its own state in order to move sensors to obtain more useful information under certain constraints. View planning, which aims to find the best view sequence for a sensor, is one of the most challenging issues in active robot vision. The quality and efficiency of view planning are critical for many robot systems and are influenced by the nature of their tasks, hardware conditions, scanning states, and planning strategies. In this paper, we first summarize some basic concepts of active robot vision, and then review representative work on systems, algorithms and applications from four perspectives: object reconstruction, scene reconstruction, object recognition, and pose estimation. Finally, some potential directions are outlined for future work.

/pdf/view-planning-in-robot-active-vision-a-survey-of-systems-14fraq8m6e.pdf

View planning in robot active vision: A survey of systems, algorithms, and applications

We address the problem of autonomously exploring unknown objects in a scene by consecutive depth acquisitions. The goal is to reconstruct the scene while online identifying the objects from among a large collection of 3D shapes. Fine-grained shape identification demands a meticulous series of observations attending to varying views and parts of the object of interest. Inspired by the recent success of attention-based models for 2D recognition, we develop a 3D Attention Model that selects the best views to scan from, as well as the most informative regions in each view to focus on, to achieve efficient object recognition. The region-level attention leads to focus-driven features which are quite robust against object occlusion. The attention model, trained with the 3D shape collection, encodes the temporal dependencies among consecutive views with deep recurrent networks. This facilitates order-aware view planning accounting for robot movement cost. In achieving instance identification, the shape collection is organized into a hierarchy, associated with pre-trained hierarchical classifiers. The effectiveness of our method is demonstrated on an autonomous robot (PR) that explores a scene and identifies the objects to construct a 3D scene model.

3D attention-driven depth acquisition for object identification

Visual place recognition is a challenging problem in robotics and autonomous systems because the scene undergoes appearance and viewpoint changes in a changing world. Existing state-of-the-art methods heavily rely on CNN-based architectures. However, CNN cannot effectively model image spatial structure information due to the inherent locality. To address this issue, this paper proposes a novel Transformer-based place recognition method to combine local details, spatial context, and semantic information for image feature embedding. Firstly, to overcome the inherent locality of the convolutional neural network (CNN), a hybrid CNN-Transformer feature extraction network is introduced. The network utilizes the feature pyramid based on CNN to obtain the detailed visual understanding, while using the vision Transformer to model image contextual information and aggregate task-related features dynamically. Specifically, the multi-level output tokens from the Transformer are fed into a single Transformer encoder block to fuse multi-scale spatial information. Secondly, to acquire the multi-scale semantic information, a global semantic NetVLAD aggregation strategy is constructed. This strategy employs semantic enhanced NetVLAD, imposing prior knowledge on the terms of the Vector of Locally Aggregated Descriptors (VLAD), to aggregate multi-level token maps, and further concatenates the multi-level semantic features globally. Finally, to alleviate the disadvantage that the fixed margin of triplet loss leads to the suboptimal convergence, an adaptive triplet loss with dynamic margin is proposed. Extensive experiments on public datasets show that the learned features are robust to appearance and viewpoint changes and achieve promising performance compared to state-of-the-arts.

Hybrid CNN-Transformer Features for Visual Place Recognition

The invention provides an object identification method and an object identification device. The object identification method comprises the steps of S1, carrying out characteristics extraction on a depth map input characteristic extractor of a current optimal viewing angle of an object to be identified to obtain a first characteristic vector; S2, allowing the first characteristic vector to pass through a first hidden layer so as to obtain a pooling layer result, and inputting the pooling layer result into a hierarchical classifier to obtain a classification result; S3, generating a currently focused local region according to the pooling layer result and an observation vector of a viewing angle observation parameter; S4, inputting the local region into a second hidden layer to obtain a second characteristic vector, wherein the second characteristic vector includes the pooling layer result and information of the current local observation viewing angle; S5, generating a next optimal viewing angle by use of a fully connected layer and the second characteristic vector; and S6, repeating steps S1-S5 until classification results are traversed to leaf nodes of the hierarchical classifier. According to the object identification method and the object identification device, an unknown scene is scanned and identified, not only the problem of identification of the object is solved, but also the problem of planning of a next viewpoint is solved.

Object identification method and device

Mineral identification is an important part of geological research. Traditional mineral identification methods heavily rely on the identification ability of the identifier and external instruments, and therefore require expensive labor expenditures and equipment capabilities. Deep learning-based mineral identification brings a new solution to the problem, which not only saves labor costs, but also reduces identification errors. However, the accuracy of existing recognition efforts is often affected by various factors such as Mohs hardness, color, picture scale, and especially light intensity. To reduce the impact of light intensity on recognition accuracy, we propose an efficient deep learning-based mineral recognition method using the luminance equalization algorithm. In this paper, we first propose a new algorithm combining histogram equalization (HE) and the Laplace algorithm, and use this algorithm to process the luminance of the identified samples, and finally use the YOLOv5 model to identify the samples. The experimental results show that our method achieves 95.6% accuracy for the identification of 50 common minerals, achieving a luminance equalization-based deep learning mineral identification method.

/pdf/mineral-identification-based-on-deep-learning-using-image-24x0yjsp.pdf

Mineral Identification Based on Deep Learning Using Image Luminance Equalization

This inventor provides a kind of recognition method and device, including the following stages: S1: get the first feature vector by putting the depth image of current best view of the object to be recognized to feature extractor; S2: get the output of pooling layer by putting the first feature vector to first hidden layer, then get the classification by putting the output of pooling layer to a hierarchical classifier; S3: generate the partial region of current view according to the output of pooling payer and view observation vector; S4: get the second feature vector by putting the partial region to second hidden layer, and the second feature vector includes the output of pooling layer and information of current view; S5: generate the next best view using a fully connected layer and the second feature vector; S6: repeat S1 to S5 until reach the leaf node of the hierarchical classifier. This inventor scans and recognizes the unknown scene, while solving the problem of next best view.

Junyu Zhang

Papers

3D attention-driven depth acquisition for object identification

Hybrid CNN-Transformer Features for Visual Place Recognition

Object identification method and device

Mineral Identification Based on Deep Learning Using Image Luminance Equalization

Object recognition method and device