On robust estimation of the location parameter

We present in this work the first end-to-end deep learning based method that predicts both 3D hand shape and pose from RGB images in the wild. Our network consists of the concatenation of a deep convolutional encoder, and a fixed model-based decoder. Given an input image, and optionally 2D joint detections obtained from an independent CNN, the encoder predicts a set of hand and view parameters. The decoder has two components: A pre-computed articulated mesh deformation hand model that generates a 3D mesh from the hand parameters, and a re-projection module controlled by the view parameters that projects the generated hand into the image domain. We show that using the shape and pose prior knowledge encoded in the hand model within a deep learning framework yields state-of-the-art performance in 3D pose prediction from images on standard benchmarks, and produces geometrically valid and plausible 3D reconstructions. Additionally, we show that training with weak supervision in the form of 2D joint annotations on datasets of images in the wild, in conjunction with full supervision in the form of 3D joint annotations on limited available datasets allows for good generalization to 3D shape and pose predictions on images in the wild.

/pdf/3d-hand-shape-and-pose-from-images-in-the-wild-ae9c0wn1ef.pdf

3D Hand Shape and Pose From Images in the Wild

Systems and articles of manufacture for image segmentation are provided herein, and include creating an anatomical model from training data comprising one or more imaging modalities, generating one or more simulated images in a target modality based on the anatomical model and one or more principles of physics pertaining to image contrast generation, and comparing the one or more simulated images to an unlabeled input image of a given imaging modality to determine a simulated image of the one or more simulated images to represent the unlabeled input image.

Image segmentation techniques

We present a novel method for monocular hand shape and pose estimation at unprecedented runtime performance of 100fps and at state-of-the-art accuracy. This is enabled by a new learning based architecture designed such that it can make use of all the sources of available hand training data: image data with either 2D or 3D annotations, as well as stand-alone 3D animations without corresponding image data. It features a 3D hand joint detection module and an inverse kinematics module which regresses not only 3D joint positions but also maps them to joint rotations in a single feed-forward pass. This output makes the method more directly usable for applications in computer vision and graphics compared to only regressing 3D joint positions. We demonstrate that our architectural design leads to a significant quantitative and qualitative improvement over the state of the art on several challenging benchmarks. We will make our code publicly available for future research.

/pdf/monocular-real-time-hand-shape-and-motion-capture-using-3i9jyhp2pm.pdf

Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data

3D hand pose estimation has made significant progress recently, where Convolutional Neural Networks (CNNs) play a critical role. However, most of the existing CNN-based hand pose estimation methods depend much on the training set, while labeling 3D hand pose on training data is laborious and time-consuming. Inspired by the point cloud autoencoder presented in self-organizing network (SO-Net), our proposed SO-HandNet aims at making use of the unannotated data to obtain accurate 3D hand pose estimation in a semi-supervised manner. We exploit hand feature encoder (HFE) to extract multi-level features from hand point cloud and then fuse them to regress 3D hand pose by a hand pose estimator (HPE). We design a hand feature decoder (HFD) to recover the input point cloud from the encoded feature. Since the HFE and the HFD can be trained without 3D hand pose annotation, the proposed method is able to make the best of unannotated data during the training phase. Experiments on four challenging benchmark datasets validate that our proposed SO-HandNet can achieve superior performance for 3D hand pose estimation via semi-supervised learning.

/pdf/so-handnet-self-organizing-network-for-3d-hand-pose-4mqy82gzjh.pdf

SO-HandNet: Self-Organizing Network for 3D Hand Pose Estimation With Semi-Supervised Learning

This paper focuses on the topic of vision based hand pose estimation from single depth map using convolutional neural network (CNN) Our main contributions lie in designing a new pose regression network architecture named CrossInfoNet The proposed CrossInfoNet decomposes hand pose estimation task into palm pose estimation sub-task and finger pose estimation sub-task, and adopts two-branch crossconnection structure to share the beneficial complementary information between the sub-tasks Our work is inspired by multi-task information sharing mechanism, which has been few discussed in hand pose estimation using depth data in previous publications In addition, we propose a heat-map guided feature extraction structure to get better feature maps, and train the complete network end-to-end The effectiveness of the proposed CrossInfoNet is evaluated with extensively self-comparative experiments and in comparison with state-of-the-art methods on four public hand pose datasets The code is available

/pdf/crossinfonet-multi-task-information-sharing-based-hand-pose-29adrdnos6.pdf

CrossInfoNet: Multi-Task Information Sharing Based Hand Pose Estimation

The goal of this paper is to estimate the 3D coordinates of the hand joints from a single depth image. To give consideration to both the accuracy and the real time performance, we design a novel three-branch Convolutional Neural Networks named Hand Branch Ensemble network (HBE), where the three branches correspond to the three parts of a hand: the thumb, the index finger and the other fingers. The structural design inspiration of the HBE network comes from the understanding of the differences in the functional importance of different fingers. In addition, a feature ensemble layer along with a low-dimensional embedding layer ensures the overall hand shape constraints. The experimental results on three public datasets demonstrate that our approach achieves comparable or better performance to state-of-the-art methods with less training data, shorter training time and faster frame rate.

/pdf/hbe-hand-branch-ensemble-network-for-real-time-3d-hand-pose-115q5y8f49.pdf

HBE: Hand Branch Ensemble Network for Real-time 3D Hand Pose Estimation

This article describes a novel framework for low-light colour image enhancement and denoising. To avoid influences from different colour channels, noise reduction and brightness/contrast enhancement are performed in different colour spaces. In the HSI space, the Bilateral filter is used for illumination- and reflection-component separation, and is effective for edge-preservation, halo removal and noise suppression. Brightness/contrast are extrapolated by using a newly designed histogram, where a suppression term based on the statistics of mathematical expectation and standard deviation was added to improve the algorithm's adaptability. Meanwhile, a saturation enhancement function was proposed to ensure more natural colours. In the YCbCr space, based on noise characteristics in low-light images, Gaussian and Median filters were adopted to reduce the noise. Experimental results indicate that the algorithm is effective for low illumination compensation, colour restoration and noise reduction.

A novel framework for low-light colour image enhancement and denoising

In this paper, we present a variation of fuzzy local information c-means (FLICM) algorithm for image segmentation by introducing a novel tradeoff factor and an effective kernel metric. The proposed tradeoff factor utilizes both local spatial and gray level information in a new way, and the Euclidean distance in FLICM algorithm is substituted by Gaussian Radial Basis function. By the novel factor and kernel metric, the new algorithm has edge identification ability and is insensitive to noise. Experiments result on both synthetic and real world images show that the proposed algorithm is effective and efficient, providing higher segmenting accuracy than other competitive algorithms.

An Edge Sensing Fuzzy Local Information C-Means Clustering Algorithm for Image Segmentation

The study of 3D hand pose estimation from a single depth image is regarded as a detection-based or regression-based problem among most of the existing deep learning-based methods, and this approach does not fully exploit the geometry of the hand, such as its structural and physical constraints. To overcome these weaknesses, we design a network with three simple parallel branches that correspond to the three functional parts of the hand. This observation is motivated by the biological viewpoint that each finger plays a different role in performing grasping and manipulation. In each branch, we perform a more detailed regression in two stages – top-down joint location regression followed by bottom-up hand pose regression – which fully exploits both the local and global structure of a hand. Finally, we further make use of the hand structure and physical constraints to refine each joint by its auxiliary points. The proposed network is a unified structure and function model that is more appropriate for hand pose estimation. Our system does not require pose pre-processing or feedback since it can directly perform training and predicting from end-to-end. The experimental results on three public datasets demonstrate that the proposed system achieves performance comparable to state-of-the-art methods.

Xiangbo Lin

Papers

CrossInfoNet: Multi-Task Information Sharing Based Hand Pose Estimation

HBE: Hand Branch Ensemble Network for Real-time 3D Hand Pose Estimation

A novel framework for low-light colour image enhancement and denoising

An Edge Sensing Fuzzy Local Information C-Means Clustering Algorithm for Image Segmentation

A multi-branch hand pose estimation network with joint-wise feature extraction and fusion