Autonomous bin picking poses significant challenges to vision-driven robotic systems given the complexity of the problem, ranging from various sensor modalities, to highly entangled object layouts, to diverse item properties and gripper types. Existing methods often address the problem from one perspective. Diverse items and complex bin scenes require diverse picking strategies together with advanced reasoning. As such, to build robust and effective machine-learning algorithms for solving this complex task requires significant amounts of comprehensive and high quality data. Collecting such data in real world would be too expensive and time prohibitive and therefore intractable from a scalability perspective. To tackle this big, diverse data problem, we take inspiration from the recent rise in the concept of metaverses, and introduce MetaGraspNet, a large-scale photo-realistic bin picking dataset constructed via physics-based metaverse synthesis. The pro-posed dataset contains 217k RGBD images across 82 different article types, with full annotations for object detection, amodal perception, keypoint detection, manipulation order and ambidextrous grasp labels for a parallel-jaw and vacuum gripper. We also provide a real dataset consisting of over 2.3k fully annotated high-quality RGBD images, divided into 5 levels of difficulties and an unseen object set to evaluate different object and layout properties. Finally, we conduct extensive experiments showing that our proposed vacuum seal model and synthetic dataset achieves state-of-the-art performance and generalizes to real world use-cases.

MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis

Trajectory generation for redundant manipulators based on inverse kinematics (IK) still faces some restraints, as it lacks universal IK calculation or specific trajectory generation methods that are suitable for robots with arbitrary degrees of freedom. In this article, the IK-free trajectory generation for robot manipulators is formulated as a Markov decision process and implemented by a general method based on deep reinforcement learning. First, an extensively explored and evaluated actor-critic (E3AC) algorithm that can make diverse action explorations and comprehensive evaluations is designed to solve the trajectory generation problem. Second, a dual-memory structure with expert-guided policy learning strategy is proposed to further enhance the performance of the algorithm in the early training period by additional successful experiences and performing an increasingly unbiased data sampling. Third, a fuzzy feedback reward mechanism that can directly establish a mapping from the abundant state variables to the self-tuning reward is constructed, instead of puzzling out an explicit function to feature the complex relations among the control objects. Finally, the comparative experimental results show that, the proposed approach is more efficient in algorithm convergence and reward calculation, and is more qualified for complex tasks with strong randomness.

Extensively Explored and Evaluated Actor-Critic With Expert-Guided Policy Learning and Fuzzy Feedback Reward for Robotic Trajectory Generation

Deep learning has been widely used for inferring robust grasps. Although human-labeled RGB-D datasets were initially used to learn grasp configurations, preparation of this kind of large dataset is expensive. To address this problem, images were generated by a physical simulator, and a physically inspired model (e.g., a contact model between a suction vacuum cup and object) was used as a grasp quality evaluation metric to annotate the synthesized images. However, this kind of contact model is complicated and requires parameter identification by experiments to ensure real world performance. In addition, previous studies have not considered manipulator reachability such as when a grasp configuration with high grasp quality is unable to reach the target due to collisions or the physical limitations of the robot. In this study, we propose an intuitive geometric analytic-based grasp quality evaluation metric. We further incorporate a reachability evaluation metric. We annotate the pixel-wise grasp quality and reachability by the proposed evaluation metric on synthesized images in a simulator to train an auto-encoder-decoder called suction graspability U-Net++ (SG-U-Net++). Experiment results show that our intuitive grasp quality evaluation metric is competitive with a physically-inspired metric. Learning the reachability helps to reduce motion planning computation time by removing obviously unreachable candidates. The system achieves an overall picking speed of 560 PPH (pieces per hour). 

Learning Suction Graspability Considering Grasp Quality and Robot Reachability for Bin-Picking

https://doi.org/10.1016/j.mechatronics.2023.102970

Machine Learning-based Framework for Optimally Solving the Analytical Inverse Kinematics for Redundant Manipulators

Replicating natural postures of human arms is essential to generate human-like behaviors in robotic applications for humans nearby. However, how to realize this requirement in interactive scenarios remains a challenge due to the kinematic redundancy and unknown postural control strategy of human arms. Inspired by the physiological characteristics that the musculoskeletal system is coordinated to minimize muscle effort in human behaviors, this paper aims to address the issue by solving a muscle effort minimization problem. It adopts a high-fidelity human arm musculoskeletal model (HAMM) and considers the implicit constraint (desired hand pose) and the inequality constraints (range of joint motion). The constrained minimization is in general nonconvex, consequently sensitive to initial guesses in iterative procedures. So, it is impracticable to solve it directly with existing gradient-based deterministic approaches or standard evolutionary algorithms. As the main contribution, a hybrid inverse kinematics algorithm was proposed for the HAMM with 7 independent and 13 mimic joints to obtain the feasible arm postures satisfying the minimization constraints. Using the arm swivel angle that parametrizes the kinematic redundancy of the HAMM, geometrically equidistant initial guess candidates can be generated over the 1-dimension feasible posture manifold. As another contribution, we present a two-phase global minimization algorithm to handle the nonconvexity of the constrained minimization. It consists of a local-search phase on the null-space of the geometric Jacobian matrix and a global-search phase with an initial guess resampling strategy. The proposed approach was validated by replicating the natural arm postures of 5 right-handed subjects in daily tasks.

https://ieeexplore.ieee.org/ielx7/7333/4359219/09855508.pdf

Muscle-Effort-Minimization-Inspired Kinematic Redundancy Resolution for Replicating Natural Posture of Human Arm

Using a closed-form inverse kinematics solution for motion planning has many advantages compared to traditional numerical approaches, most notably much faster computation times and better suitability for real-time applications. Steady progress has been made to develop an analytic inverse kinematics solution for seven degrees of freedom (DoF) manipulators without joint offsets. Redundancy resolutions have been proposed to push the individual joint angles away from their limits. However, no further objectives are considered in these closed-form solutions. We propose a multi-objective real-time optimization approach that is able to minimize joint velocities and accelerations, while still avoiding joint limits. A closed-form solution to the optimization is derived, which only requires a single intuitive tuning parameter. We provide a free and open-source software implementation of the full kinematics algorithm that covers the complete solution space. The null-space motion resulting from our method is evaluated, achieving faster motion times and smoother arm motions along Cartesian paths in comparison to the state-of-the-art approach. Finally, experimental results with a KUKA iiwa robot are reported.

A Real-Time-Capable Closed-Form Multi-Objective Redundancy Resolution Scheme for Seven-DoF Serial Manipulators

This paper presents a novel approach to grasp detection of unknown items with Deep Semantic Segmentation. The approach can detect whether an object is occluded or graspable and offers the possibility to determine grasp vectors for several grippers. This is accomplished with the help of Semantic Segmentation. The Segmentation is tested with Fully Convolutional Networks (FCN) [1] and DeepLab V3+ [2]. The neural networks use RGB images, depth images or both. The size, speed and functionality of the neural networks with different inputs are discussed. A determination for grasp vectors for parallel grippers is explained and experiments are carried out. In contrast to current methods, which only calculate a single grasp vector, our approach can determine all possible grasp vectors.

Jonathan Auberle

Papers

A Real-Time-Capable Closed-Form Multi-Objective Redundancy Resolution Scheme for Seven-DoF Serial Manipulators

Grasp Area Detection of Unknown Objects based on Deep Semantic Segmentation.