The camera provides rich visual information regarding objects and becomes one of the most mainstream sensors in the automation community. However, it is often difficult to be applicable when the objects are not visually distinguished. On the other hand, tactile sensors can be used to capture multiple object properties, such as textures, roughness, spatial features, compliance, and friction, and therefore provide another important modality for the perception. Nevertheless, effective combination of the visual and tactile modalities is still a challenging problem. In this paper, we develop a visual–tactile fusion framework for object recognition tasks. This paper uses the multivariate-time-series model to represent the tactile sequence and the covariance descriptor to characterize the image. Further, we design a joint group kernel sparse coding (JGKSC) method to tackle the intrinsically weak pairing problem in visual–tactile data samples. Finally, we develop a visual–tactile data set, composed of 18 household objects for validation. The experimental results show that considering both visual and tactile inputs is beneficial and the proposed method indeed provides an effective strategy for fusion.

Visual–Tactile Fusion for Object Recognition

Collaborative robots are widely employed in strict hybrid assembly tasks involved in intelligent manufacturing. In this paper, we develop a teaching-learning-collaboration (TLC) model for the collaborative robot to learn from human demonstrations and assist its human partner in shared working situations. The human could program the robot using natural language instructions according to his/her personal working preferences via this approach. Afterward, the robot learns from human assembly demonstrations by taking advantage of the maximum entropy inverse reinforcement learning algorithm and updates its task-based knowledge using the optimal assembly strategy. In the collaboration process, the robot is able to leverage its learned knowledge to actively assist the human in the collaborative assembly task. Experimental results and analysis demonstrate that the proposed approach presents considerable robustness and applicability in human–robot collaborative tasks. Note to Practitioners —This paper is motivated by the human–robot collaborative assembly problem in the context of advanced manufacturing. Collaborative robotics makes a huge shift from the traditional robot-in-a-cage model to robots interacting with people in an open working environment. When the human works with the robot in the shared workspace, it is significant to lessen human programming effort and improve the human–robot collaboration efficiency once the task is updated. We develop a TLC model for the robot to learn from human demonstrations and assist its human partner in collaborative tasks. Once the task is changed, the human may code the robot via natural language instructions according to his/her personal working preferences. The robot can learn from human assembly demonstrations to update its task-based knowledge, which can be leveraged by the robot to actively assist the human to accomplish the collaborative task. We demonstrate the advantages of the proposed approach via a set of experiments in realistic human–robot collaboration contexts.

Facilitating Human–Robot Collaborative Tasks by Teaching-Learning-Collaboration From Human Demonstrations

Objective. Myoelectric activity volitionally generated by the user is often used for controlling hand prostheses in order to replicate the synergistic actions of muscles in healthy humans during grasping. Muscle synergies in healthy humans are based on the integration of visual perception, heuristics and proprioception. Here, we demonstrate how sensor fusion that combines artificial vision and proprioceptive information with the high-level processing characteristics of biological systems can be effectively used in transradial prosthesis control. Approach. We developed a novel context- and user-aware prosthesis (CASP) controller integrating computer vision and inertial sensing with myoelectric activity in order to achieve semi-autonomous and reactive control of a prosthetic hand. The presented method semi-automatically provides simultaneous and proportional control of multiple degrees-of-freedom (DOFs), thus decreasing overall physical effort while retaining full user control. The system was compared against the major commercial state-of-the art myoelectric control system in ten able-bodied and one amputee subject. All subjects used transradial prosthesis with an active wrist to grasp objects typically associated with activities of daily living. Main results. The CASP significantly outperformed the myoelectric interface when controlling all of the prosthesis DOF. However, when tested with less complex prosthetic system (smaller number of DOF), the CASP was slower but resulted with reaching motions that contained less compensatory movements. Another important finding is that the CASP system required minimal user adaptation and training. Significance. The CASP constitutes a substantial improvement for the control of multi-DOF prostheses. The application of the CASP will have a significant impact when translated to real-life scenarious, particularly with respect to improving the usability and acceptance of highly complex systems (e.g., full prosthetic arms) by amputees.

https://iopscience.iop.org/article/10.1088/1741-2560/12/6/066022/pdf

Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis.

Robotic manipulation of highly deformable materials is inherently challenging due to the need to maintain tension and the high dimensionality of the state of the material. Past work in this area mostly focuses on generating a detailed model for the material and its interaction with the robot, then using the model to construct a motion plan. In this paper, we take a different approach by using only sensor feedback to dictate the robot motion. We consider the collaborative manipulation of a deformable sheet between a person and a dual-armed robot (Baxter by Rethink Robotics). The robot is capable of contact sensing via joint torque sensors and is equipped with a head-mounted RGBd sensor. The robot senses contact force to maintain tension of the sheet, and in turn comply to the human motion. This is akin to handling a tablecloth with a partner but with one's eyes closed. To improve the response, we use the RGBd sensor to detect folds, and command the robot to move in an orthogonal direction to smooth them out. This is like handling cloth by looking at the cloth itself. Both controllers are able to follow human motion without excessive crimps in the sheet, but as expected, the hybrid controller combining force and vision outperforms the force controller alone in terms of tension force transient. The ability to quickly detect the state of the deformable material also enables more complex manipulation strategies in the future.

/pdf/collaborative-human-robot-manipulation-of-highly-deformable-38femm7xpw.pdf

Collaborative human-robot manipulation of highly deformable materials

In this letter, we present a general approach to automatically visual servo control the position and shape of a deformable object whose deformation parameters are unknown. The servo control is achieved by online learning a model mapping between the robotic end-effector's movement and the object's deformation measurement. The model is learned using the Gaussian process regression (GPR) to deal with its highly nonlinear property, and once learned, the model is used for predicting the required control at each time step. To overcome GPR's high computational cost while dealing with long manipulation sequences, we implement a fast online GPR by selectively removing uninformative observation data from the regression process. We validate the performance of our controller on a set of deformable object manipulation tasks and demonstrate that our method can achieve effective and accurate servo control for general deformable objects with a wide variety of goal settings. Experiment videos are available at https://sites.google.com/view/mso-fogpr.

Three-Dimensional Deformable Object Manipulation Using Fast Online Gaussian Process Regression

This paper describes an adaptable system which is able to perform manipulation operations (such as Peg-in-Hole or Laying-Down actions) with flexible objects. As such objects easily change their shape significantly during the execution of an action, traditional strategies, e. g., for solve path-planning problems, are often not applicable. It is therefore required to integrate visual tracking and shape reconstruction with a physical modeling of the materials and their deformations as well as action learning techniques. All these different submodules have been integrated into a demonstration platform, operating in real-time. Simulations have been used to bootstrap the learning of optimal actions, which are subsequently improved through real-world executions. To achieve reproducible results, we demonstrate this for casted silicone test objects of regular shape. Note to Practitioners-The aim of this work was to facilitate the setup of robot-based automation of delicate handling of flexible objects consisting of a uniform material. As examples, we have considered how to optimally maneuver flexible objects through a hole without colliding and how to place flexible objects on a flat surface with minimal introduction of internal stresses in the object. Given the material properties of the object, we have demonstrated in these two applications how the system can be programmed with minimal requirements of human intervention. Rather than being an integrated system with the drawbacks in terms of lacking flexibility, our system should be viewed as a library of new technologies that have been proven to work in close to industrial conditions. As a rather basic, but necessary part, we provide a technology for determining the shape of the object when passing on, e. g., a conveyor belt prior to being handled. The main technologies applicable for the manipulated objects are: A method for real-time tracking of the flexible objects during manipulation, a method formodel-based offline prediction of the static deformation of grasped, flexible objects and, finally, a method for optimizing specific tasks based on both simulated and real-world executions.

/pdf/an-adaptable-robot-vision-system-performing-manipulation-1nhmj5wvyx.pdf

An Adaptable Robot Vision System Performing Manipulation Actions With Flexible Objects

We describe the outline of an adaptable system which is able to perform grasping and peg-in-hole actions with flexible objects. The system makes use of visual tracking and shape reconstruction, physical modeling of flexible material and learning based on a kernel density approach. We show results for the different sub-modules in simulation as well as real world data.

Knud Aulkjær Andersen

Papers

An Adaptable Robot Vision System Performing Manipulation Actions With Flexible Objects

An outline for an intelligent system performing peg-in-hole actions with flexible objects