The main goal of this work is to provide an efficient multimodal training platform for improving technicians’ maintenance and assembly skills in order to accelerate their acquisition process of new maintenance and assembly tasks. A concept for multimodal Augmented Reality based training of maintenance skills has been developed which fulfils the requirements of efficient training. The work presents results of interdisciplinary research based on the fusion of cognitive science and computer science in the field of maintenance and assembly training. To overcome limitations of existing approaches, the work addresses the combined training of both sensorimotor and cognitive skills by exploiting the potential of Augmented Reality technologies and multimodal interfaces. Novel concepts and techniques for visual augmentation have been developed in order to improve the visual feedback for the user and to cope with problems caused by tracking inaccuracies. This work also provides key elements (guidelines) of maintenance skill training, called Accelerators, which can be transferred to any procedural training task. It is shown how such a multimodal Augmented Reality environment providing on-the-job training can be implemented based on a standard for real-time 3D communication (X3D ISO standard). In the focus of this work is the presentation component of the training system, that is, the multimodal presentation of feedback to the trainee, whereby amount and type of feedback can depend on captured input data.
The developed concept includes the possibility to easily adapt the training system to different training conditions in order to support various phases of a training program and trainees at various skill levels. Furthermore, the system is able to reuse existing reliable training material. In addition, the generation of new training task scenarios is simplified in comparison to existing training systems.

Multimodal Training of Maintenance and Assembly Skills Based on Augmented Reality

Application of immersive Virtual Reality (VR) system has been proven to be practical and effective in offering the sense of presence and engagement for users to explore inside a virtual environment (VE) creation. However, numerous user experience problems still occur within the application design and development phase. An attempt to categorized the issues from presented studies by the researchers within the scope of papers between years of 2007 to 2017. There is no empirical evaluation conducted in this reviewing process but the aim is to bridge the gap between user experience and application system through understanding and learning from the challenges in the system itself. This can promote a more highly acceptance level of this complex technology among the users. This paper also proposes the future research directions and predicts the development trends of VR system.

/pdf/challenges-in-virtual-reality-system-a-review-1ffbajntc7.pdf

Challenges in virtual reality system: A review

This paper describes our ongoing research work on deviceless interaction using hand gesture recognition with a calibrated stereo system. Video-based interaction is one of the most intuitive kinds of Human-Computer-Interaction with Virtual-Reality applications due to the fact that users are not wired to a computer. If interaction with three-dimensional environments is considered, pointing, grabbing and releasing are the most intuitive gestures used by humans. This paper describes our video-based gesture recognition system that observes the user in front of a large displaying screen, identifying three different hand gestures in real time using 2D feature classification and determines 3D information like the 3D position of the user's hand or the pointing direction if performed. Different scenario applications like a virtual chess game against the computer or an industrial scenario have been developed and tested. To estimate the possible count of distinguishable gestures a sign language recognition application has been developed and tested using a single uncalibrated camera only.

/pdf/gestural-interaction-using-feature-classification-3siwam05ti.pdf

Cornelius Malerczyk

Papers

Gestural Interaction Using Feature Classification