Kyeong-Beom Park

Bio: Kyeong-Beom Park is an academic researcher from Chonnam National University. The author has contributed to research in topics: Augmented reality & Computer science. The author has an hindex of 5, co-authored 11 publications receiving 97 citations.

M-GAN: Retinal Blood Vessel Segmentation by Balancing Losses Through Stacked Deep Fully Convolutional Networks

Abstract: Until now, the human expert segments retinal blood vessels manually in fundus images to inspect human retinal-related diseases, such as diabetic retinopathy and vascular occlusion. Recently, many studies were conducted for automatic retinal vessel segmentation from fundus images through supervised and unsupervised methods to minimize user intervention. However, most of them lack in segmentation robustness and cannot optimize loss functions so that results of the segmentation have made lots of fake or thin branches. This article proposes a new conditional generative adversarial network called M-GAN to conduct accurate and precise retinal vessel segmentation by balancing losses through stacked deep fully convolutional networks. It consists of a newly designed M-generator with deep residual blocks for more robust segmentation and an M-discriminator with a deeper network for more efficient training of the adversarial model. In particular, a multi-kernel pooling block is added between the stacked layers to support the scale-invariance of vessel segmentations of different sizes. The M-generator has down-sampling layers to extract features and up-sampling layers to make segmented retinal blood vessel images from the extracted features. The M-discriminator also has a deeper network similar to the down-sampling of the M-generator, but the final layer is constructed as a fully connected layer for decision making. We conduct pre-processing of the input image using automatic color equalization (ACE) to make the retinal vessels of the input fundus image more clear and perform post-processing that makes the vessel branches smooth and reduces false-negatives using a Lanczos resampling method. To verify the proposed method, we used DRIVE, STARE, HRF, and CHASE-DB1datasets and compared the proposed M-GAN with other studies. We measured accuracy, the intersection of union (IoU), F1 score, and Matthews correlation coefficient (MCC) for comparative analysis. Results of comparison proved that the proposed M-GAN derived superior performance than other studies.

Deep learning-based smart task assistance in wearable augmented reality

Abstract: Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.

An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation

Abstract: For human-robot collaboration (HRC), one of the most practical methods to ensure human safety with a vision-based system is establishing a minimum safe distance. This study proposes a novel integrated mixed reality (MR) system for safety-aware HRC using deep learning and digital twin generation. The proposed approach can accurately measure the minimum safe distance in real-time and provide MR-based task assistance to the human operator. The approach integrates MR with safety-related monitoring by tracking the shared workplace and providing user-centric visualization through smart MR glasses for safe and effective HRC. Two RGB-D sensors are used to reconstruct and track the working environment. One sensor scans one area of the physical environment through 3D point cloud data. The other also scans another area of the environment and tracks the user's 3D skeletal information. In addition, the two partially scanned environments are registered together by applying a fast global registration method to two sets of the 3D point cloud. Furthermore, deep learning-based instance segmentation is applied to the target object's 3D point cloud to increase the registration between the real robot and its virtual robot, the digital twin of the real robot. While only 3D point cloud data are widely used in previous studies, this study proposes a simple yet effective 3D offset-based safety distance calculation method based on the robot's digital twin and the human skeleton. The 3D offset-based method allows for real-time applicability without sacrificing the accuracy of safety distance calculation for HRI. In addition, two comparative evaluations were conducted to confirm the originality and advantage of the proposed MR-based HRC.

Hands-Free Human–Robot Interaction Using Multimodal Gestures and Deep Learning in Wearable Mixed Reality

Abstract: This study proposes a novel hands-free interaction method using multimodal gestures such as eye gazing and head gestures and deep learning for human-robot interaction (HRI) in mixed reality (MR) environments. Since human operators hold some objects for conducting tasks, there are many constrained situations where they cannot use their hands for HRI interactions. To provide more effective and intuitive task assistance, the proposed hands-free method supports coarse-to-fine interactions. Eye gazing-based interaction is used for coarse interactions such as searching and previewing of target objects, and head gesture interactions are used for fine interactions such as selection and 3D manipulation. In addition, deep learning-based object detection is applied to estimate the initial positioning of physical objects to be manipulated by the robot. The result of object detection is then combined with 3D spatial mapping in the MR environment for supporting accurate initial object positioning. Furthermore, virtual object-based indirect manipulation is proposed to support more intuitive and efficient control of the robot, compared with traditional direct manipulation (e.g., joint-based and end effector-based manipulations). In particular, a digital twin, the synchronized virtual robot of the real robot, is used to provide a preview and simulation of the real robot to manipulate it more effectively and accurately. Two case studies were conducted to confirm the originality and advantages of the proposed hands-free HRI: (1) performance evaluation of initial object positioning and (2) comparative analysis with traditional direct robot manipulations. The deep learning-based initial positioning reduces much effort for robot manipulation using eye gazing and head gestures. The object-based indirect manipulation also supports more effective HRI than previous direct interaction methods.

Systematic literature review of hand gestures used in human computer interaction interfaces

Abstract: Gestures, widely accepted as a humans’ natural mode of interaction with their surroundings, have been considered for use in human-computer based interfaces since the early 1980s. They have been explored and implemented, with a range of success and maturity levels, in a variety of fields, facilitated by a multitude of technologies. Underpinning gesture theory however focuses on gestures performed simultaneously with speech, and majority of gesture based interfaces are supported by other modes of interaction. This article reports the results of a systematic review undertaken to identify characteristics of touchless/in-air hand gestures used in interaction interfaces. 148 articles were reviewed reporting on gesture-based interaction interfaces, identified through searching engineering and science databases (Engineering Village, Pro Quest, Science Direct, Scopus and Web of Science). The goal of the review was to map the field of gesture-based interfaces, investigate the patterns in gesture use, and identify common combinations of gestures for different combinations of applications and technologies. From the review, the community seems disparate with little evidence of building upon prior work and a fundamental framework of gesture-based interaction is not evident. However, the findings can help inform future developments and provide valuable information about the benefits and drawbacks of different approaches. It was further found that the nature and appropriateness of gestures used was not a primary factor in gesture elicitation when designing gesture based systems, and that ease of technology implementation often took precedence.

Artificial intelligence (AI) in augmented reality (AR)-assisted manufacturing applications: a review

Abstract: Augmented reality (AR) has proven to be an invaluable interactive medium to reduce cognitive load by bridging the gap between the task-at-hand and relevant information by displaying information wit...

Skin Electronics: Next-Generation Device Platform for Virtual and Augmented Reality

Abstract: Virtual reality (VR) and augmented reality (AR) are overcoming the physical limits of real‐life using advances in devices and software. In particular, the recent restrictions in transportation from the coronavirus disease 2019 (COVID‐19) pandemic are making people more interested in these virtual experiences. However, to minimize the differences between artificial and natural perception, more human‐interactive and human‐like devices are necessary. The skin is the largest organ of the human body and interacts with the environment as the site of interfacing and sensing. Recent progress in skin electronics has enabled the use of the skin as the mounting object of functional devices and the signal pathway bridging humans and computers, with opening its potential in future VR and AR applications. In this review, the current skin electronics are summarized as one of the most promising device solutions for future VR/AR devices, especially focusing on the recent materials and structures. After defining and explaining VR/AR systems and the components, the advantages of skin electronics for VR/AR applications are emphasized. Next, the detailed functionalities of skin electronic devices, including the input, output, energy devices, and integrated systems, are reviewed for future VR/AR applications. [ABSTRACT FROM AUTHOR] Copyright of Advanced Functional Materials is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)