Showing papers by "Hao Su published in 2017"

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Abstract: Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Abstract: Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Abstract: Few prior works study deep learning on point sets. PointNet is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Abstract: Generation of 3D data by deep neural network has been attracting increasing attention in the research community. The majority of extant works resort to regular representations such as volumetric grids or collection of images, however, these representations obscure the natural invariance of 3D shapes under geometric transformations, and also suffer from a number of other issues. In this paper we address the problem of 3D reconstruction from a single image, generating a straight-forward form of output – point cloud coordinates. Along with this problem arises a unique and interesting issue, that the groundtruth shape for an input image may be ambiguous. Driven by this unorthordox output form and the inherent ambiguity in groundtruth, we design architecture, loss function and learning paradigm that are novel and effective. Our final solution is a conditional shape sampler, capable of predicting multiple plausible 3D point clouds from an input image. In experiments not only can our system outperform state-of-the-art methods on single image based 3D reconstruction benchmarks, but it also shows strong performance for 3D shape completion and promising ability in making multiple plausible predictions.

SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

Abstract: In this paper, we study the problem of semantic annotation on 3D models that are represented as shape graphs. A functional view is taken to represent localized information on graphs, so that annotations such as part segment or keypoint are nothing but 0-1 indicator vertex functions. Compared with images that are 2D grids, shape graphs are irregular and non-isomorphic data structures. To enable the prediction of vertex functions on them by convolutional neural networks, we resort to spectral CNN method that enables weight sharing by parametrizing kernels in the spectral domain spanned by graph Laplacian eigenbases. Under this setting, our network, named SyncSpecCNN, strives to overcome two key challenges: how to share coefficients and conduct multi-scale analysis in different parts of the graph for a single shape, and how to share information across related but different shapes that may be represented by very different graphs. Towards these goals, we introduce a spectral parametrization of dilated convolutional kernels and a spectral transformer network. Experimentally we tested SyncSpecCNN on various tasks, including 3D shape part segmentation and keypoint prediction. State-of-the-art performance has been achieved on all benchmark datasets.

Learning Shape Abstractions by Assembling Volumetric Primitives

Abstract: We present a learning framework for abstracting complex shapes by learning to assemble objects using 3D volumetric primitives. In addition to generating simple and geometrically interpretable explanations of 3D objects, our framework also allows us to automatically discover and exploit consistent structure in the data. We demonstrate that using our method allows predicting shape representations which can be leveraged for obtaining a consistent parsing across the instances of a shape collection and constructing an interpretable shape similarity measure. We also examine applications for image-based prediction as well as shape manipulation.

Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories

Abstract: We focus on the non-Lambertian object-level intrinsic problem of recovering diffuse albedo, shading, and specular highlights from a single image of an object. Based on existing 3D models in the ShapeNet database, a large-scale object intrinsics database is rendered with HDR environment maps. Millions of synthetic images of objects and their corresponding albedo, shading, and specular ground-truth images are used to train an encoder-decoder CNN, which can decompose an image into the product of albedo and shading components along with an additive specular component. Our CNN delivers accurate and sharp results in this classical inverse problem of computer vision. Evaluated on our realistically synthetic dataset, our method consistently outperforms the state-of-the-art by a large margin. We train and test our CNN across different object categories. Perhaps surprising especially from the CNN classification perspective, our intrinsics CNN generalizes very well across categories. Our analysis shows that feature learning at the encoder stage is more crucial for developing a universal representation across categories. We apply our model to real images and videos from Internet, and observe robust and realistic intrinsics results. Quality non-Lambertian intrinsics could open up many interesting applications such as realistic product search based on material properties and image-based albedo/specular editing.

ComplementMe: weakly-supervised component suggestions for 3D modeling

Abstract: Assembly-based tools provide a powerful modeling paradigm for non-expert shape designers. However, choosing a component from a large shape repository and aligning it to a partial assembly can become a daunting task. In this paper we describe novel neural network architectures for suggesting complementary components and their placement for an incomplete 3D part assembly. Unlike most existing techniques, our networks are trained on unlabeled data obtained from public online repositories, and do not rely on consistent part segmentations or labels. Absence of labels poses a challenge in indexing the database of parts for the retrieval. We address it by jointly training embedding and retrieval networks, where the first indexes parts by mapping them to a low-dimensional feature space, and the second maps partial assemblies to appropriate complements. The combinatorial nature of part arrangements poses another challenge, since the retrieval network is not a function: several complements can be appropriate for the same input. Thus, instead of predicting a single output, we train our network to predict a probability distribution over the space of part embeddings. This allows our method to deal with ambiguities and naturally enables a UI that seamlessly integrates user preferences into the design process. We demonstrate that our method can be used to design complex shapes with minimal or no user input. To evaluate our approach we develop a novel benchmark for component suggestion systems demonstrating significant improvement over state-of-the-art techniques.

Frustum PointNets for 3D Object Detection from RGB-D Data

Abstract: In this work, we study 3D object detection from RGB-D data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.

Noise adaptive wavelet thresholding for speckle noise removal in optical coherence tomography

Abstract: Optical coherence tomography (OCT) is based on coherence detection of interferometric signals and hence inevitably suffers from speckle noise. To remove speckle noise in OCT images, wavelet domain thresholding has demonstrated significant advantages in suppressing noise magnitude while preserving image sharpness. However, speckle noise in OCT images has different characteristics in different spatial scales, which has not been considered in previous applications of wavelet domain thresholding. In this study, we demonstrate a noise adaptive wavelet thresholding (NAWT) algorithm that exploits the difference of noise characteristics in different wavelet sub-bands. The algorithm is simple, fast, effective and is closely related to the physical origin of speckle noise in OCT image. Our results demonstrate that NAWT outperforms conventional wavelet thresholding.

Opening a Can of Worm(-like Micelle)s: The Effect of Temperature of Solutions of Functionalized Dipeptides.

Abstract: A simple heat/cool cycle can be used to significantly affect the properties of a solution of a low-molecular-weight gelator at high pH. The viscosity and extensional viscosity are increased markedly, leading to materials with very different properties than when the native solution is used.

Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55.

Abstract: We introduce a large-scale 3D shape understanding benchmark using data and annotation from ShapeNet 3D object database. The benchmark consists of two tasks: part-level segmentation of 3D shapes and 3D reconstruction from single view images. Ten teams have participated in the challenge and the best performing teams have outperformed state-of-the-art approaches on both tasks. A few novel deep learning architectures have been proposed on various 3D representations on both tasks. We report the techniques used by each team and the corresponding performances. In addition, we summarize the major discoveries from the reported results and possible trends for the future work in the field.

Fiber-Optic Force Sensors for MRI-Guided Interventions and Rehabilitation: A Review

Abstract: Magnetic resonance imaging (MRI) provides both anatomical imaging with excellent soft tissue contrast and functional MRI imaging (fMRI) of physiological parameters. The last two decades have witnessed the manifestation of increased interest in MRI-guided minimally invasive intervention procedures and fMRI for rehabilitation and neuroscience research. Accompanying the aspiration to utilize MRI to provide imaging feedback during interventions and brain activity for neuroscience study, there is an accumulated effort to utilize force sensors compatible with the MRI environment to meet the growing demand of these procedures, with the goal of enhanced interventional safety and accuracy, improved efficacy and rehabilitation outcome. This paper summarizes the fundamental principles, the state of the art development, and challenges of fiber-optic force sensors for MRI-guided interventions and rehabilitation. It provides an overview of MRI-compatible fiber-optic force sensors based on different sensing principles, including light intensity modulation, wavelength modulation, and phase modulation. Extensive design prototypes are reviewed to illustrate the detailed implementation of these principles. Advantages and disadvantages of the sensor designs are compared and analyzed. A perspective on the future development of fiber-optic sensors is also presented, which may have additional broad clinical applications. Future surgical interventions or rehabilitation will rely on intelligent force sensors to provide situational awareness to augment or complement human perception in these procedures.

An MRI-Guided Telesurgery System Using a Fabry-Perot Interferometry Force Sensor and a Pneumatic Haptic Device

Abstract: This paper presents a surgical master-slave teleoperation system for percutaneous interventional procedures under continuous magnetic resonance imaging (MRI) guidance. The slave robot consists of a piezoelectrically actuated 6-degree-of-freedom (DOF) robot for needle placement with an integrated fiber optic force sensor (1-DOF axial force measurement) using the Fabry-Perot interferometry (FPI) sensing principle; it is configured to operate inside the bore of the MRI scanner during imaging. By leveraging the advantages of pneumatic and piezoelectric actuation in force and position control respectively, we have designed a pneumatically actuated master robot (haptic device) with strain gauge based force sensing that is configured to operate the slave from within the scanner room during imaging. The slave robot follows the insertion motion of the haptic device while the haptic device displays the needle insertion force as measured by the FPI sensor. Image interference evaluation demonstrates that the telesurgery system presents a signal to noise ratio reduction of less than 17% and less than 1% geometric distortion during simultaneous robot motion and imaging. Teleoperated needle insertion and rotation experiments were performed to reach 10 targets in a soft tissue-mimicking phantom with 0.70 ± 0.35 mm Cartesian space error.

Learning Hierarchical Shape Segmentation and Labeling from Online Repositories

Abstract: We propose a method for converting geometric shapes into hierarchically segmented parts with part labels. Our key idea is to train category-specific models from the scene graphs and part names that accompany 3D shapes in public repositories. These freely-available annotations represent an enormous, untapped source of information on geometry. However, because the models and corresponding scene graphs are created by a wide range of modelers with different levels of expertise, modeling tools, and objectives, these models have very inconsistent segmentations and hierarchies with sparse and noisy textual tags. Our method involves two analysis steps. First, we perform a joint optimization to simultaneously cluster and label parts in the database while also inferring a canonical tag dictionary and part hierarchy. We then use this labeled data to train a method for hierarchical segmentation and labeling of new 3D shapes. We demonstrate that our method can mine complex information, detecting hierarchies in man-made objects and their constituent parts, obtaining finer scale details than existing alternatives. We also show that, by performing domain transfer using a few supervised examples, our technique outperforms fully-supervised techniques that require hundreds of manually-labeled models.

A Noncrystallization Approach toward Uniform Thylakoids-like 2D "Nano-coins" and Their Grana-like 3D Suprastructures.

Abstract: Two-dimensional (2D) circular shape nanostructures (e.g., “nano-coins”) are ubiquitously present in thylakoids and grana within chloroplasts of plant cells in nature. The design and fabrication of 2D nano-coins with controlled sizes and thicknesses yet remain challenging tasks. Herein, we present a noncrystallization approach to achieve 2D nano-coins from assemblies of a set of zwitterionic giant surfactants. Distinguished from traditional crystallization approaches where the 2D nanostructures with specific crystallographic symmetries are fabricated, the noncrystallization assembly of giant surfactants results in 2D nano-coins that are derived from the separation of assembled 3D multiple lamellar cylindrical colloids with uniform diameters. The diameters and thicknesses of these nano-coins can be readily tailored by varying the molecular length of giant surfactants’ tails. The formation of 2D nano-coins or 3D cylindrical colloid suprastructures is controlled by tuning the pH value of added selective solve...

Learning hierarchical shape segmentation and labeling from online repositories

Abstract: We propose a method for converting geometric shapes into hierarchically segmented parts with part labels. Our key idea is to train category-specific models from the scene graphs and part names that accompany 3D shapes in public repositories. These freely-available annotations represent an enormous, untapped source of information on geometry. However, because the models and corresponding scene graphs are created by a wide range of modelers with different levels of expertise, modeling tools, and objectives, these models have very inconsistent segmentations and hierarchies with sparse and noisy textual tags. Our method involves two analysis steps. First, we perform a joint optimization to simultaneously cluster and label parts in the database while also inferring a canonical tag dictionary and part hierarchy. We then use this labeled data to train a method for hierarchical segmentation and labeling of new 3D shapes. We demonstrate that our method can mine complex information, detecting hierarchies in man-made objects and their constituent parts, obtaining finer scale details than existing alternatives. We also show that, by performing domain transfer using a few supervised examples, our technique outperforms fully-supervised techniques that require hundreds of manually-labeled models.

ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling

Abstract: Assembly-based tools provide a powerful modeling paradigm for non-expert shape designers. However, choosing a component from a large shape repository and aligning it to a partial assembly can become a daunting task. In this paper we describe novel neural network architectures for suggesting complementary components and their placement for an incomplete 3D part assembly. Unlike most existing techniques, our networks are trained on unlabeled data obtained from public online repositories, and do not rely on consistent part segmentations or labels. Absence of labels poses a challenge in indexing the database of parts for the retrieval. We address it by jointly training embedding and retrieval networks, where the first indexes parts by mapping them to a low-dimensional feature space, and the second maps partial assemblies to appropriate complements. The combinatorial nature of part arrangements poses another challenge, since the retrieval network is not a function: several complements can be appropriate for the same input. Thus, instead of predicting a single output, we train our network to predict a probability distribution over the space of part embeddings. This allows our method to deal with ambiguities and naturally enables a UI that seamlessly integrates user preferences into the design process. We demonstrate that our method can be used to design complex shapes with minimal or no user input. To evaluate our approach we develop a novel benchmark for component suggestion systems demonstrating significant improvement over state-of-the-art techniques.

Cross-Modal Attribute Transfer for Rescaling 3D Models

Abstract: We present an algorithm for transferring physical attributes between webpages and 3D shapes. We crawl product catalogues and other webpages with structured metadata containing physical attributes such as dimensions and weights. Then we transfer physical attributes between shapes and real-world objects using a joint embedding of images and 3D shapes and a view-based weighting and aspect ratio filtering scheme for instance-level linking of 3D models and real-world counterpart objects. We evaluate our approach on a large-scale dataset of unscaled 3D models, and show that we outperform prior work on rescaling 3D models that considers only category-level size priors.

Recent progress in exploiting small molecule peptides as supramolecular hydrogelators

Abstract: Small molecule peptides and their derivatives are an emerging class of supramolecular hydrogelators that have attracted rapidly growing interest in the fields of drug delivery and regenerative medicine due to their inherent biodegradability and biocompatibility, as well as versatility in molecular design and ease of synthesis. Built upon the directional, intermolecular interactions such as hydrogen bonding and π-π stacking, peptide-based molecular units can associate in aqueous solution into filamentous assemblies of various sizes and shapes. Under appropriate conditions, these filamentous assemblies can percolate into a 3D network with materials properties tailorable for specific biomedical applications. In this review, we survey the literature published over the past three years in the development of peptide-based hydrogelators for biomedical applications. We highlight several representative examples and center our discussion on the fundamentals of molecular design, assembly, and gelation conditions.

Design and preliminary evaluation of a multi-robotic system with pelvic and hip assistance for pediatric gait rehabilitation

Abstract: This paper presents a modular, computationally-distributed “multi-robot” cyberphysical system designed to assist children with developmental delays in learning to walk. The system consists of two modules, each assisting a different aspect of gait: a tethered cable pelvic module with up to 6 degrees of freedom (DOF), which can modulate the motion of the pelvis in three dimensions, and a two DOF wearable hip module assisting lower limb motion, specifically hip flexion. Both modules are designed to be lightweight and minimally restrictive to the user, and the modules can operate independently or in cooperation with each other, allowing flexible system configuration to provide highly customized and adaptable assistance. Motion tracking performance of approximately 2 mm root mean square (RMS) error for the pelvic module and less than 0.1 mm RMS error for the hip module was achieved. We demonstrate coordinated operation of the two modules on a mannequin test platform with articulated and instrumented lower limbs.

Design and evaluation of an upper extremity wearable robot with payload balancing for human augmentation

Abstract: Electric, pneumatic and hydraulic tools are widely used in industry. Repetitive tasks using heavy tools would often cause fatigue for workers leading to musculoskeletal injuries. The efficiency and manufacturing quality can hardly be guaranteed either. Powered exoskeletons for upper limb [1-3] and lower limb assistance [4-5] have been developed recently.

Evaluation of Force Tracking Controller with Soft Exosuit for Hip Extension Assistance

Abstract: This abstract describes the design and experimental evaluation of a force tracking controller for hip extension assistance utilizing a soft exosuit connected to a tethered off-board actuation system. The new controller aims to improve the force profile tracking capability and demonstrate its advantages over our previously reported work. The controller was evaluated by one healthy participant walking on a treadmill at 1.35 m/s. Results showed that the system can deliver a predefined force profile robustly with a 200 N peak force. The measured peak force value using force controller was 198.7 ± 2.9 N, and the root-mean-squared (RMS) error was 3.4 N (1.7 % of desired peak force). These results indicate that the force control reduces peak force variability and improves force profile tracking capability.