Generation of GelSight Tactile Images for Sim2Real Learning
Daniel Fernandes Gomes,Paolo Paoletti,Shan Luo +2 more
- Vol. 6, Iss: 2, pp 4177-4184
Reads0
Chats0
TLDR
In this paper, the authors introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator, which can produce high-resolution images from depth-maps captured by a simulated optical sensor, and reconstruct the interaction between the touched object and an opaque soft membrane.Abstract:
Most current works in Sim2Real learning for robotic manipulation tasks leverage camera vision that may be significantly occluded by robot hands during the manipulation. Tactile sensing offers complementary information to vision and can compensate for the information loss caused by the occlusions. However, the use of tactile sensing is restricted in the Sim2Real research due to no simulated tactile sensors being available. To mitigate the gap, we introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator. Similar to the real GelSight sensor, the simulated sensor can produce high-resolution images from depth-maps captured by a simulated optical sensor, and reconstruct the interaction between the touched object and an opaque soft membrane. It can indirectly sense forces, geometry, texture and other properties of the object and enables Sim2Real learning with tactile sensing. Preliminary experimental results have shown that the simulated sensor could generate realistic outputs similar to the ones captured by a real GelSight sensor. All the materials used in this letter are available at https://danfergo.github.io/gelsight-simulation .read more
Citations
More filters
Journal ArticleDOI
Robotic architectural assembly with tactile skills: Simulation and optimization
TL;DR: In this article , a robotic arm can be used to assemble a structure from predefined discrete building blocks autonomously by using a combination of reinforcement learning and planning for this task, and the achieved results serve as a proof of concept and indicate the directions for further research to enable more complex assemblies involving multiple building elements.
Posted Content
Taxim: An Example-based Simulation Model for GelSight Tactile Sensors.
Zilin Si,Wenzhen Yuan +1 more
TL;DR: Taxim as mentioned in this paper is a high-speed simulation model for a vision-based tactile sensor, which uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation.
Journal ArticleDOI
Robotic architectural assembly with tactile skills: Simulation and optimization
Boris Belousov,Bastian Wibranek,Jan Schneider,Tim Schneider,Georgia Chalvatzaki,Jan Peters,Oliver Tessmann +6 more
TL;DR: In this paper, a combination of reinforcement learning and planning is used to train a controller to place a single building block in simulation and demonstrate that the achieved results serve as a proof-of-concept for more complex assemblies involving multiple building elements.
Journal ArticleDOI
Taxim: An Example-Based Simulation Model for GelSight Tactile Sensors
TL;DR: Taxim as mentioned in this paper is a high-speed simulation model for a vision-based tactile sensor, which is based on the linear displacement relationship and the superposition principle to simulate the deformation of the elastomer.
Peer ReviewDOI
Hardware Technology of Vision-Based Tactile Sensor: A Review
Shixin Zhang,Zixi Chen,Yuan Gao,Weiwei Wan,Jianhua Shan,Hongxiang Xue,Fuchun Sun,Yiyong Yang,Bin Fang +8 more
TL;DR: In this paper , the authors focus on hardware technologies and review the literature over the past five years to provide a valuable guideline for developers to improve the performance of vision-based tactile sensors.
References
More filters
Proceedings ArticleDOI
Deep Residual Learning for Image Recognition
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
Proceedings ArticleDOI
ImageNet: A large-scale hierarchical image database
TL;DR: A new database called “ImageNet” is introduced, a large-scale ontology of images built upon the backbone of the WordNet structure, much larger in scale and diversity and much more accurate than the current image datasets.
Journal ArticleDOI
Image quality assessment: from error visibility to structural similarity
TL;DR: In this article, a structural similarity index is proposed for image quality assessment based on the degradation of structural information, which can be applied to both subjective ratings and objective methods on a database of images compressed with JPEG and JPEG2000.
Proceedings Article
ROS: an open-source Robot Operating System
TL;DR: This paper discusses how ROS relates to existing robot software frameworks, and briefly overview some of the available application software which uses ROS.
Journal ArticleDOI
Illumination for computer generated pictures
TL;DR: Human visual perception and the fundamental laws of optics are considered in the development of a shading rule that provides better quality and increased realism in generated images.