scispace - formally typeset
Open accessJournal ArticleDOI: 10.1109/LRA.2021.3063925

Generation of GelSight Tactile Images for Sim2Real Learning

04 Mar 2021-Vol. 6, Iss: 2, pp 4177-4184
Abstract: Most current works in Sim2Real learning for robotic manipulation tasks leverage camera vision that may be significantly occluded by robot hands during the manipulation. Tactile sensing offers complementary information to vision and can compensate for the information loss caused by the occlusions. However, the use of tactile sensing is restricted in the Sim2Real research due to no simulated tactile sensors being available. To mitigate the gap, we introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator. Similar to the real GelSight sensor, the simulated sensor can produce high-resolution images from depth-maps captured by a simulated optical sensor, and reconstruct the interaction between the touched object and an opaque soft membrane. It can indirectly sense forces, geometry, texture and other properties of the object and enables Sim2Real learning with tactile sensing. Preliminary experimental results have shown that the simulated sensor could generate realistic outputs similar to the ones captured by a real GelSight sensor. All the materials used in this letter are available at .

... read more

Topics: Tactile sensor (68%)

7 results found

Open accessPosted Content
16 Jun 2021-arXiv: Robotics
Abstract: Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually considered despite its direct relation to environment interaction. In this work, we present a suite of simulated environments tailored towards tactile robotics and reinforcement learning. A simple and fast method of simulating optical tactile sensors is provided, where high-resolution contact geometry is represented as depth images. Proximal Policy Optimisation (PPO) is used to learn successful policies across all considered tasks. A data-driven approach enables translation of the current state of a real tactile sensor to corresponding simulated depth images. This policy is implemented within a real-time control loop on a physical robot to demonstrate zero-shot sim-to-real policy transfer on several physically-interactive tasks requiring a sense of touch.

... read more

Topics: Tactile sensor (67%), Reinforcement learning (52%), Image translation (52%) ... read more

2 Citations

Open accessPosted Content
Zilin Si, Wenzhen Yuan1Institutions (1)
09 Sep 2021-arXiv: Robotics
Abstract: Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In this paper, we propose Taxim, a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. A GelSight sensor uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation of the elastomer, which infers the geometry and forces applied at the contact surface. We propose an example-based method for simulating GelSight: we simulate the optical response to the deformation with a polynomial look-up table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. In order to simulate the surface markers' motion that is caused by the surface stretch of the elastomer, we apply the linear elastic deformation theory and the superposition principle. The simulation model is calibrated with less than 100 data points from a real sensor. The example-based approach enables the model to easily migrate to other GelSight sensors or its variations. To the best of our knowledge, our simulation framework is the first to incorporate marker motion field simulation that derives from elastomer deformation together with the optical simulation, creating a comprehensive and computationally efficient tactile simulation framework. Experiments reveal that our optical simulation has the lowest pixel-wise intensity errors compared to prior work and can run online with CPU computing.

... read more

Topics: Tactile sensor (62%), Motion field (53%)

2 Citations

Open accessJournal ArticleDOI: 10.1016/J.AUTCON.2021.104006
Boris Belousov1, Bastian Wibranek2, Jan Schneider1, Tim Schneider1  +3 moreInstitutions (3)
Abstract: Construction is an industry that could benefit significantly from automation, yet still relies heavily on manual human labor. Thus, we investigate how a robotic arm can be used to assemble a structure from predefined discrete building blocks autonomously. Since assembling structures is a challenging task that involves complex contact dynamics, we propose to use a combination of reinforcement learning and planning for this task. In this work, we take a first step towards autonomous construction by training a controller to place a single building block in simulation. Our evaluations show that trial-and-error algorithms that have minimal prior knowledge about the problem to be solved, so called model-free deep reinforcement learning algorithms, can be successfully employed. We conclude that the achieved results, albeit imperfect, serve as a proof of concept and indicate the directions for further research to enable more complex assemblies involving multiple building elements.

... read more

Topics: Reinforcement learning (56%), Robotic arm (55%), Task (project management) (51%) ... read more

Open accessJournal ArticleDOI: 10.3390/S21113818
Li Qin1, Hongyu Wang1, Yazhou Yuan1, Shufan Qin1Institutions (1)
31 May 2021-Sensors
Abstract: The peg-in-hole task with object feature uncertain is a typical case of robotic operation in the real-world unstructured environment It is nontrivial to realize object perception and operational decisions autonomously, under the usual visual occlusion and real-time constraints of such tasks In this paper, a Bayesian networks-based strategy is presented in order to seamlessly combine multiple heterogeneous senses data like humans In the proposed strategy, an interactive exploration method implemented by hybrid Monte Carlo sampling algorithms and particle filtering is designed to identify the features' estimated starting value, and the memory adjustment method and the inertial thinking method are introduced to correct the target position and shape features of the object respectively Based on the Dempster-Shafer evidence theory (D-S theory), a fusion decision strategy is designed using probabilistic models of forces and positions, which guided the robot motion after each acquisition of the estimated features of the object It also enables the robot to judge whether the desired operation target is achieved or the feature estimate needs to be updated Meanwhile, the pliability model is introduced into repeatedly perform exploration, planning and execution steps to reduce interaction forces, the number of exploration The effectiveness of the strategy is validated in simulations and in a physical robot task

... read more

Topics: Particle filter (52%), Feature (computer vision) (52%), Probabilistic logic (51%) ... read more


28 results found

Open accessProceedings ArticleDOI: 10.1109/CVPR.2016.90
Kaiming He1, Xiangyu Zhang1, Shaoqing Ren1, Jian Sun1Institutions (1)
27 Jun 2016-
Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

... read more

Topics: Deep learning (53%), Residual (53%), Convolutional neural network (53%) ... read more

93,356 Citations

Proceedings ArticleDOI: 10.1109/CVPR.2009.5206848
Jia Deng1, Wei Dong1, Richard Socher1, Li-Jia Li1  +2 moreInstitutions (1)
20 Jun 2009-
Abstract: The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond.

... read more

Topics: WordNet (57%), Image retrieval (54%)

31,274 Citations

Journal ArticleDOI: 10.1109/TIP.2003.819861
Abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at sim/lcv/ssim/.

... read more

Topics: Image quality (61%), Subjective video quality (56%), Human visual system model (56%) ... read more

30,333 Citations

Open accessProceedings Article
Morgan Quigley1Institutions (1)
01 Jan 2009-
Abstract: This paper gives an overview of ROS, an opensource robot operating system. ROS is not an operating system in the traditional sense of process management and scheduling; rather, it provides a structured communications layer above the host operating systems of a heterogenous compute cluster. In this paper, we discuss how ROS relates to existing robot software frameworks, and briefly overview some of the available application software which uses ROS.

... read more

Topics: Robot software (55%), Open-source robotics (54%)

7,367 Citations

Open accessJournal ArticleDOI: 10.1145/360825.360839
Bui Tuong Phong1Institutions (1)
Abstract: The quality of computer generated images of three-dimensional scenes depends on the shading technique used to paint the objects on the cathode-ray tube screen. The shading algorithm itself depends in part on the method for modeling the object, which also determines the hidden surface algorithm. The various methods of object modeling, shading, and hidden surface removal are thus strongly interconnected. Several shading techniques corresponding to different methods of object modeling and the related hidden surface algorithms are presented here. Human visual perception and the fundamental laws of optics are considered in the development of a shading rule that provides better quality and increased realism in generated images.

... read more

Topics: Gouraud shading (64%), Blinn–Phong shading model (63%), Phong reflection model (60%) ... read more

3,166 Citations

No. of citations received by the Paper in previous years
Network Information
Related Papers (5)
Generation of GelSight Tactile Images for Sim2Real Learning18 Jan 2021, arXiv: Robotics

Daniel Fernandes Gomes, Paolo Paoletti +1 more

Simulation of Tactile Sensing Arrays for Physical Interaction Tasks06 Jul 2020

Zhanat Kappassov, Juan-Antonio Corrales-Ramon +1 more

A robotic tactile perception system concept13 Feb 1990

Emil M. Petriu, M.A. Greenspan +2 more

Learning robot tactile sensing for object manipulation01 Sep 2014

Yevgen Chebotar, Oliver Kroemer +1 more