The current study evaluated the use of virtual reality (VR) and augmented reality (AR) platforms, developed within the scope of the SKILLS Integrated Project, for industrial maintenance and assembly (IMA) tasks training. VR and AR systems are now widely regarded as promising training platforms for complex and highly demanding IMA tasks. However, there is a need to empirically evaluate their efficiency and effectiveness compared to traditional training methods. Forty expert technicians were randomly assigned to four training groups in an electronic actuator assembly task: VR (training with the VR platform twice), Control-VR (watching a filmed demonstration twice), AR (training with the AR platform once), and Control-AR (training with the real actuator and the aid of a filmed demonstration once). A post-training test evaluated performance in the real task. Results demonstrate that, in general, the VR and AR training groups required longer training time compared to the Control-VR and Control-AR groups, respe...

Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks

Integrating Augmented Reality with Building Information Modeling: Onsite construction process controlling for liquefied natural gas industry

Metaverses embedded in our lives create virtual experiences inside of the physical world. Moving towards metaverses in aircraft maintenance, mixed reality (MR) creates enormous opportunities for the interaction with virtual airplanes (digital twin) that deliver a near-real experience, keeping physical distancing during pandemics. 3D twins of modern machines exported to MR can be easily manipulated, shared, and updated, which creates colossal benefits for aviation colleges who still exploit retired models for practicing. Therefore, we propose mixed reality education and training of aircraft maintenance for Boeing 737 in smart glasses, enhanced with a deep learning speech interaction module for trainee engineers to control virtual assets and workflow using speech commands, enabling them to operate with both hands. With the use of the convolutional neural network (CNN) architecture for audio features and learning and classification parts for commands and language identification, the speech module handles intermixed requests in English and Korean languages, giving corresponding feedback. Evaluation with test data showed high accuracy of prediction, having on average 95.7% and 99.6% on the F1-Score metric for command and language prediction, respectively. The proposed speech interaction module in the aircraft maintenance metaverse further improved education and training, giving intuitive and efficient control over the operation, enhancing interaction with virtual objects in mixed reality.

Towards Aircraft Maintenance Metaverse Using Speech Interactions with Virtual Objects in Mixed Reality

About the Contributors.Preface.Acknowledgments.Chapter 1. Knowledge Creation and Dialectics (H. Takeuchi & I. Nonaka).Chapter 2. The Knowledge-creating Company (I. Nonaka).Chapter 3. Theory of Organizational Knowledge Creation (I. Nonaka & H. Takeuchi).Chapter 4. Knowledge Creation as Synthesizing Process (I. Nonaka & R. Toyama).Chapter 5. From Managing to Enabling Knowledge (K. Ichijo).Chapter 6. Value Differentiation: Organizing "Know-what" for Product Concept Innovation (K. Kusunoki).Chapter 7. Knowledge Management and Global Competition: Olympus's Approach to Global Knowledge Management in the Digital Still Camera Industry (Y. Ishikura).Chapter 8. Inter-organizational Knowledge Creation: Knowledge and Networks (C. Ahmadjian).Chapter 9. The Strategy-making Process as Dialogue (E. Osomo).Chapter 10. Branding Capabilities: A Look at Sony's Capabilities in Brand Knowledge Creation (S. Akutsu & I. Nonaka). Chapter 11. Synthesizing Modular and Integral Knowledge: Business Architecture Innovation in the IT Era (K. Kusunoki).Chapter 12. Knowledge Creation within a Dialectic Organization (H. Takeuchi).Index.

Hitotsubashi on knowledge management

Large volumes of information in the OR are ignored by surgeons when the amount outpaces human mental processing abilities. We developed an augmented reality (AR) system for dental implant surgery that acts as an automatic information filter, selectively displaying only relevant information. The purpose is to reduce information overflow and offer intuitive image guidance. The system was evaluated in a pig cadaver experiment. Information filtering is implemented via rule-based situation interpretation with description logics. The interpretation is based on intraoperative distances measurement between anatomical structures and the dental drill with optical tracking. For AR, a head-mounted display is used, which was calibrated with a novel method based on SPAAM. To adapt to surgeon specific preferences, we offer two alternative display formats: one with static and another with contact analog AR. The system made the surgery easier and showed ergonomical benefits, as assessed by a questionnaire. All relevant phases were recognized reliably. The new calibration showed significant improvements, while the deviation of the realized implants was $$<$$
 2.5 mm. The system allowed the surgeon to fully concentrate on the surgery itself. It offered greater flexibility since the surgeon received all relevant information, but was free to deviate from it. Accuracy of the realized implants remains an open issue and part of future work.

A system for context-aware intraoperative augmented reality in dental implant surgery

Augmented and Mixed Reality technology provides to the medical field the possibility for seamless visualization of text-based physiological data and various graphical 3D data onto the patient's body. This allows improvements in diagnosis and treatment of patients. For the patient, this technology offers benefits and further potential in therapy, rehabilitation and diagnosis, and explanation. Applications across the whole range of functions that affect the health sector from the physician, the medical student, to the patients are possible. However, the quality of the work of medical professionals is considerably influenced by both usefulness and usability of technology. Consequently, key issues in developing such applications are the tracking methodology, the display technology and most of all ensuring good usability. There have been several research groups who extended the state of the art in employing these technologies in the medical domain. However, only a few are addressing issues of Human-Computer Interaction, Interaction Design, and Usability Engineering. This paper provides a brief overview over the history and the most recent developments in this domain with a special focus on issues of user-centered development.

/pdf/some-usability-issues-of-augmented-and-mixed-reality-for-e-3kbxi7h5qt.pdf

Some usability issues of augmented and mixed reality for e-health applications in the medical domain

There is evidence that recent developments in Augmented Reality (AR) technology has the potential to be applied as pervasive media on multiple devices in different ways and contexts, especially with low-cost devices including Mobile Augmented Reality (MAR) applications on smart phones or Pocket-PCs. In this paper we present a framework in order to combine the pervasive e-education concept with augmented reality content for e-training. We analyze current research, discuss some examples from ultralight / light sport aircraft maintenance and show how to apply this framework generically. We present a learning engine to deliver this special type of content and provide a further outlook of future research. A user-centered approach must ensure that the developments can stimulate motivation and enhance performance of the end users in different training sessions. The main benefit is, that the end users are enabled to better perceive complex, technical facts, systems and components.

/pdf/virtual-and-mixed-reality-interfaces-for-e-training-examples-3o8hi994k9.pdf

Virtual and mixed reality interfaces for e-training: examples of applications in light aircraft maintenance

"Augmented reality (AR),Mixed Reality (MR) and their mix and combination with other disruptive technologies offer an enormous potential for supporting instructors and trainees in modern education and working environments such as of aircraft maintenance technicians or automotive service technicians. In this paper we investigate and show some examples on how the performance and training of such instructors and trainees can be actively supported. Furthermore we will discuss the new challenges for training designers. The augmentation of the physical world with interactive, context-aware information (e.g. 2D and 3D content) provides multifaceted possibilities, on various ubiquitous and pervasive computing environments. While there is still the broad opinion that these concepts are just situated in the world of science fiction (SciFi) and SciFi movies, we will relate these techniques to existing technologies and prototypes in research. Terms like outernet, print + or 2.0, augmented goggles, wearable technology are not just remaining pure buzzwords anymore. We will demonstrate how different prototypes applying low cost rapid prototyping methods can be applied as powerful performance assistance and training support instruments, whereby discussing the requirements and user-needs analysis phases as well as human–computer interaction and interaction design issues, user modelling, usability engineering, prototyping and evaluation issues. Different scenarios are possible and provide the basis to generate storyboards. One of the key factors is hereby to analyse typical tasks and activities of users and utilize familiar user interaction paradigms for accessing information, such as using a book or assisting the work with task sheets. For example by augmenting the material that is printed in the book with additional graphical 3D interactive information which can be viewed and manipulated by the instructor and/or trainee, one can provide a link between traditional learning and technology-enhanced learning. Basing on theoretical and empirical research it is possible then to design via first moodboards and scribbles relevant prototypes. A qualitative and quantitative analysis can be used to define a basic design process for such new environments and settings. Moreover, MR and AR along with Mobile Tagging (MT) combined with Pervasive Computing provide the possibility to realize a Physical World Connection (PWC) between Reality and Virtuality. In the field of aviation and automotive industry, this offers manifold possibilities for maintenance and service personnel to get access to assistive technologies in a very intuitive way to enhance their operation, work, training, and knowledge. Assistance for the large variety of job tasks can be provided to a certain extent by offering augmentation of the different senses like vision and audition, providing a media-rich interface. Although the roots of Mixed Reality and Augmented Reality are based on prototype applications in the aircraft industry in the early 1990s, the impact of these emerging technologies on special target groups has not yet been investigated and validated by many research groups. With a specific focus on these user communities, applications are considerably more influenced by both usefulness and usability of technology. Consequently, it is argued that key issues in developing such applications are the tracking methodology, the display technology, interaction (devices and framework) and most of all ensuring good usability. In this paper, a concrete example in a aviation and automotive environment will be presented as a case study for investigating and validating these key issues. Preliminary results of semi-structured interviews and observations in real training and work settings indicate a lack of information concerning existence of such technologies and environments, but show big interest and potential for such educational and workplace innovations, while concrete visions or user requirements for future augmented education environments remain open and are subject of our further research steps.

Interaction Design of Augmented Education Environments - Augmented and Mixed Reality for performance and training support of Aviation / Automotive Technicians.

Johannes Christian

Papers

Some usability issues of augmented and mixed reality for e-health applications in the medical domain

Virtual and mixed reality interfaces for e-training: examples of applications in light aircraft maintenance

Interaction Design of Augmented Education Environments - Augmented and Mixed Reality for performance and training support of Aviation / Automotive Technicians.