scispace - formally typeset

Journal ArticleDOI

A Comparison of Virtual and Physical Training Transfer of Bimanual Assembly Tasks

01 Apr 2018-IEEE Transactions on Visualization and Computer Graphics (IEEE)-Vol. 24, Iss: 4, pp 1574-1583

TL;DR: A study that compares the effectiveness of virtual training and physical training for teaching a bimanual assembly task and highlights the validity of virtual reality systems in training.

AbstractAs we explore the use of consumer virtual reality technology for training applications, there is a need to evaluate its validity compared to more traditional training formats. In this paper, we present a study that compares the effectiveness of virtual training and physical training for teaching a bimanual assembly task. In a between-subjects experiment, 60 participants were trained to solve three 3D burr puzzles in one of six conditions comprised of virtual and physical training elements. In the four physical conditions, training was delivered via paper- and video-based instructions, with or without the physical puzzles to practice with. In the two virtual conditions, participants learnt to assemble the puzzles in an interactive virtual environment, with or without 3D animations showing the assembly process. After training, we conducted immediate tests in which participants were asked to solve a physical version of the puzzles. We measured performance through success rates and assembly completion testing times. We also measured training times as well as subjective ratings on several aspects of the experience. Our results show that the performance of virtually trained participants was promising. A statistically significant difference was not found between virtual training with animated instructions and the best performing physical condition (in which physical blocks were available during training) for the last and most complex puzzle in terms of success rates and testing times. Performance in retention tests two weeks after training was generally not as good as expected for all experimental conditions. We discuss the implications of the results and highlight the validity of virtual reality systems in training.

Topics: Virtual training (69%), Transfer of training (62%)

Summary (3 min read)

1 INTRODUCTION

  • Section 3 presents the experimental design and hypotheses.
  • Section 6 discusses the results, limitations and future work.

4.2 Materials

  • A virtual replica of the laboratory was modeled for the virtual enviornment used in the virtual experimental conditions.
  • An Oculus Rift Consumer Version 1, two Oculus Touch controllers and two Oculus sensors were used for the virtual experimental conditions.
  • Preassembled blocks for the first and second puzzles were glued together.

4.3 Physical training environment

  • For those experimental conditions in which the physical blocks were available during training (PB and PV I B) these were initially placed on the table following the same configuration as the paper instructions.
  • Preassembled puzzles were placed behind the blocks.

4.4 Virtual training environment

  • All interactions in the virtual training environment could be equally carried out using either hand and participants could concurrently complete one interaction with each hand.
  • A participant could grab and rotate the assembled pieces with one hand and grab the next block to attach with the other hand.
  • The green highlight indicates on the block is colliding with its preview block and within twenty degrees from the correct orientation.
  • By releasing the trigger button of the Oculus Touch controller the virtual block would snap into its correct location.

4.5 Procedure

  • After a waiting period of two weeks, participants returned to the lab for the second session.
  • In this session participants were asked to complete a paper version of the Vandenberg and Kuse Mental Rotations Test [22] .
  • They then completed the retention test for each of the three puzzles, in which they were asked to solve the three burr puzzles from the first session without a training phase, in the same order and in a maximum of three minutes.
  • They completed the same questionnaire from the first session at the end of each retention trial (see Table 3 ).
  • After completing all retention trials they were interviewed regarding strategies used throughout the session.

5.1 Types of errors

  • Unsuccessful puzzle completions during immediate and retention testing were due to one of two reasons.
  • In most cases, participants did not complete the 3D puzzles within the given maximum time (180s).
  • On the other hand, a low number of participants decided to stop the time before the upper limit thinking that they had successfully solved the puzzle.
  • Close inspection showed that they had not correctly assembled the pieces.
  • Completion time values for both immediate and retention testing were corrected by assigning the upper time limit (180s) to all unsuccessful attempts.

5.2 First session 5.2.1 Training times

  • The post hoc analysis revealed statistically significant differences in training times for the first puzzle.
  • There was a statistically significant difference between P (mean rank = 15.
  • The post hoc analysis revealed statistically significant differences in training times for the second puzzle.

5.2.2 Immediate testing success rates

  • The model suggested that participants in the P experimental condition were 0.074 times as likely to successfully assemble the third puzzle than participants in the reference category (PV I B).
  • The model suggested that participants in the PV I experimental condition were 0.028 times as likely to successfully assemble the third puzzle than participants in the reference category (PV I B).

Dimension Question

  • Likert scale extremes Difficulty Please rate the difficulty of the task you just completed.
  • It is important to note that all participants in this condition successfully completed the third puzzle.
  • The model suggested that participants who succeeded at correctly assembling the second puzzle were 9.687 times as likely to successfully assemble the third puzzle than participants in the reference category (PV I B).
  • For the third puzzle, the binomial logistic regression model with the highest percentage of correctly classified observations was the one that ascertained the effect of both experimental condition and successful completion of the previous puzzle.
  • Test statistics using Dunn's procedure [4] for immediate testing times between the different experimental conditions.

5.2.3 Immediate testing completion times

  • The post hoc analysis revealed statistically significant differences in immediate testing times for the third puzzle.
  • The analysis of immediate testing completion times shows some support for H2 and H3.

5.2.4 Subjective questionnaire ratings

  • There was a statistically significant difference in ease of use of the training environment (F(5,54) = 3.044, p = 0.017) between groups as determined by one-way ANOVA for the third puzzle.
  • No other significant interactions were found for the third puzzle.

5.3.1 Participants

  • A total of 56 participants that completed the first part session returned to complete the second session two weeks later (average number of days between training session and retention session: 14.16, SD = 0.918).
  • Overall, retention testing performance was lower than expected for all conditions both in terms of success rates and completion times.

5.3.4 Subjective questionnaire ratings

  • There was no statistically significant difference in rated difficulty and seriousness between groups as determined by one-way ANOVA for any of the three puzzles.
  • Tukey post hoc tests showed no significant interactions.

6 DISCUSSION

  • One of the limitations in their design was the high complexity of the puzzles.
  • Overall, retention testing resulted in lower performance than the authors had expected and they believe this is due to the difficulty associated with remembering the process to solve the three puzzles two weeks after the training.
  • This was further validated by verbal feedback from their participants during the second session.
  • The authors previous piloting of the task had not shown this effect.
  • Future studies should further evaluate the suitability of the task for retention.

7 CONCLUSION

  • The authors analysed performance in terms of success rates as well as immediate testing times and retention testing times.
  • The authors results show that the performance of virtually trained participants was promising.
  • A statistically significant difference wasn't found between condition V E A and the best performing physical condition (PV I B, in which physical blocks and animated instructions were available during training) for the last and most complex puzzle in terms of success rates and immediate testing times.
  • Retention testing performance was unexpectedly low due to the high complexity of the task.
  • The authors believe that the results of this study further validate the effectiveness of virtual training for bimanual assembly tasks.

Did you find this useful? Give us your feedback

...read more

Content maybe subject to copyright    Report

1574 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 24, NO. 4, APRIL 2018
Manuscript received 11 Sept. 2017; accepted 8 Jan. 2018.
Date of publication 19 Jan. 2018; date of current version 18 Mar. 2018.
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TVCG.2018.2793638
This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/
A Comparison of Virtual and Physical Training Transfer
of Bimanual Assembly Tasks
Mar
´
ıa Murcia-L
´
opez and Anthony Steed
Abstract
—As we explore the use of consumer virtual reality technology for training applications, there is a need to evaluate its validity
compared to more traditional training formats. In this paper, we present a study that compares the effectiveness of virtual training and
physical training for teaching a bimanual assembly task. In a between-subjects experiment, 60 participants were trained to solve three
3D burr puzzles in one of six conditions comprised of virtual and physical training elements. In the four physical conditions, training was
delivered via paper- and video-based instructions, with or without the physical puzzles to practice with. In the two virtual conditions,
participants learnt to assemble the puzzles in an interactive virtual environment, with or without 3D animations showing the assembly
process. After training, we conducted immediate tests in which participants were asked to solve a physical version of the puzzles.
We measured performance through success rates and assembly completion testing times. We also measured training times as well
as subjective ratings on several aspects of the experience. Our results show that the performance of virtually trained participants
was promising. A statistically significant difference was not found between virtual training with animated instructions and the best
performing physical condition (in which physical blocks were available during training) for the last and most complex puzzle in terms of
success rates and testing times. Performance in retention tests two weeks after training was generally not as good as expected for all
experimental conditions. We discuss the implications of the results and highlight the validity of virtual reality systems in training.
Index Terms—Learning transfer, virtual reality, assembly, training
1INTRODUCTION
The availability of consumer virtual reality technology has raised the
manufacturing industry’s interest in virtual training for manual assem-
bly tasks. Virtual environments could deliver cost-efficient, safe and
potentially effective training. If proven adequate, virtual training would
also allow for the completion of operator instruction prior to the in-
stallation of physical workstations, tools and components. This would
accelerate the end-to-end manufacturing process and, consequently,
increase efficiency of production. However, more evidence is needed
to ascertain the effectiveness of virtual environments for training as
opposed to more traditional forms of training.
In this paper, we present a study that compares the effectiveness
of virtual and traditional paper- and video-based training transfer of
a bimanual assembly task, motivated by previous research [3, 10]. In
a between-subjects experimental design, participants were trained to
solve three six-piece burr puzzles in a virtual training environment or a
physical training environment. The conditions were designed to account
for situations in which the physical puzzle blocks are available or not
during training. The conditions were also devised to include static
instructions (paper) or combinations of static and animated instructions
(video or 3D animations). Table 1 introduces the experimental condition
types, acronyms and definitions. Table 2 shows a classification of
the experimental conditions according to instruction type and block
availability during training.
Following training, participants were asked to solve physical ver-
sions of the puzzles (referred to as immediate testing). Participants then
completed a retention session, two weeks after the training (referred
to as retention testing). During the course of the study, participants
answered mental rotations tests and questionnaires measuring several
aspects of the experience.
We tested three hypotheses about the effectiveness of training in each
of the conditions being compared in the study. The first hypothesis
Mar
´
ıa Murcia-L
´
opez is with University College London. E-mail:
maria.murcia.13@ucl.ac.uk.
Anthony Steed is with University College London. E-mail:
a.steed@ucl.ac.uk.
(H1) was that conditions in which the physical blocks were available
during training (PB and PV
I
B) would yield a higher number of success-
ful puzzle completions during immediate and retention testing. The
second hypothesis (H2) was that the conditions in which both static
and animated instructions were available during training (PV
I
, PV
I
B
and
V
E
A
) would result in lower assembly times during immediate and
retention testing. The third hypothesis (H3) was that condition PV
I
B,
with animated instructions (video) and physical blocks, would yield the
highest performance as measured by immediate and retention success
rates and assembly testing times. Although we expected some condi-
tions to deliver worse or better performance, we had no hypothesis on
the full order so all the analysis presented in this paper is two-tailed.
Immediate testing results showed some support for the first hypothe-
sis, some support for the second hypothesis and some support for the
third hypothesis. Retention performance was lower than expected for
all conditions both in terms of success rates and completion times and
did not provide evidence to support any of the three hypotheses.
The remainder of this paper is organised as follows. In Section 2
we review related work on learning transfer in immersive mixed reality
systems. Section 3 presents the experimental design and hypotheses.
In Section 4 we introduce the methodology and experimental setup. In
Section 5 we report the results of the study. Section 6 discusses the
results, limitations and future work. Section 7 concludes.
2R
ELATED WORK
Previous research has highlighted the effectiveness of immersive mixed
reality training in different disciplines, including military training, medi-
cal training and vehicle driving simulators [17,21], as well as navigation
and spatial knowledge training [8, 23], amongst others. Despite the
recognised success in the aforementioned fields, studies on immersive
virtual training transfer of procedural and assembly tasks have reported
contrasting results.
Hall and Horwitz compared retention of procedural knowledge of
equipment operation in an immersive virtual environment and in a 2D
computer environment and found no significant differences [7]. They
claimed that virtual reality training may not be superior to conventional
electronic media for training certain skills. Gavish et al. evaluated the
use of virtual reality and augmented reality technology for industrial
maintenance and assembly task training [5]. They concluded that an
augmented reality platform was more suitable for training of this type of
tasks and encouraged further evaluation of virtual reality based training.
Fig. 1. One of the three 3D printed burr puzzles used in the study.
In a more recent study Gonzalez-Franco et al. compared collabo-
rative conventional face-to-face training with a mixed reality training
setup for a manufacturing procedure of an aircraft door [6]. Their re-
sults indicated that performance levels yielded by the immersive mixed
reality training system were not significantly different from the conven-
tional face-to-face training format. Rose et al. evaluated the transfer
from a virtual environment to the real world of a simple sensorimotor
task [16]. Overall, virtual training resulted in equivalent or even bet-
ter real world performance than real or physical training for the task.
However, they advise that their findings may not apply to other types
of training tasks.
Sowndararajan et al. found an effect of level of immersion in memo-
rising a complex procedure [20]. In their study, participants trained in
the system with the higher level of immersion (a large L-shaped projec-
tion display) completed tasks significantly faster and with fewer errors
than participants trained in the system with lower level of immersion
(using a typical laptop display).
Other studies have shown effective learning transfer in virtual en-
vironments with the addition of haptic force-feedback devices. For
instance, Adams et al. conducted a study to explore the benefits of
haptic feedback for virtual training of a manual task [1]. They reported
that force-feedback was a requirement for higher learning transfer in
virtual environments.
Our study is inspired by the work of Carlson et al. in 2015 [3], it-
self motivated by previous work [10, 13, 19]. In a between-subjects
experimental design, Carlson et al. compared the effectiveness of vir-
tual bimanual haptic training versus traditional physical training of
an assembly task consisting of a six-piece burr puzzle. Their results
indicated that physically trained participants initially outperformed
virtually trained participants. However, virtually trained participants
improved their testing times after two weeks. Results also showed that
virtual training was enhanced by using coloured blocks as they helped
participants remember the assembly process. We run a similar task
comparing paper- and video-based training with virtual training in the
absence of a haptic force-feedback device.
We agree with Carlson et al. in that 3D burr puzzles are suitable
proxy tasks or abstractions of context-specific manual assembly tasks,
such as engine assembly operations at vehicle manufacturing plants. We
therefore decided to use the same type of task in our study. Following
their reported methods, we complemented the training task with a series
of mental rotation tests to distribute participants amongst the condition
groups in our between-subjects experimental design [2, 14, 22]. We
also decided to colour-code the puzzle blocks and instructions as well
as to use a semi-transparent virtual representation of the hands in the
virtual environment [11, 12], amongst other recommendations made by
the authors which are further explained in Section 3.
Our study extends and builds on previous work by comparing a
number of virtual and physical training formats, the latter representing
the most common formats (video and paper instructions) in current
assembly process training programmes. The main aim of this research
is to verify whether exposure to a virtual training environment is suffi-
cient for effective training. We are specifically interested in situations
in which haptic devices are not available and when the physical compo-
nents and tools used in the process are not accessible during training.
3E
XPERIMENTAL DESIGN AND HYPOTHESES
Inspired by previous research [3], in our study we used three different
colour-coded versions of a six-piece burr puzzle for the assembly task
(see Figure 1). Burr puzzles have been commonly used for assembly
task training studies in the past because they provide a recognisable and
adequately complex model in which participants must follow a specific
procedure in order to solve them [3, 10]. However, our study differs
from previous work in that no haptic devices were used. In addition, we
are interested in whether consumer virtual reality systems are sufficient
for effective training.
In our study, participants were trained and tested in assembling three
versions of a six-piece burr puzzle. To provide increasing difficulty, the
first three blocks had been preassembled for the first puzzle, the first
two for the second and none for the third. This meant that participants
had to remember a higher number of steps in the assembly process over
the course of the experimental task for each puzzle.
Following a between-subjects experimental design, participants were
trained to solve each puzzle by adding the corresponding unassembled
blocks in one of six experimental conditions (see Table 1). Experimen-
tal conditions were designed to account for scenarios in which blocks
are not available (P and PV
I
), physical blocks are available (PB and
PV
I
B) or virtual blocks are available (
V
E
and
V
E
A
) during training (see
Table 2 for a classification of the experimental conditions). The physi-
cal experimental conditions (P, PB, PV
I
and PV
I
B) were designed to
encompass combinations of paper- and video- based instructions. The
virtual experimental conditions (
V
E
and
V
E
A
) involved a virtual ver-
sion of the paper instructions, with or without 3D animations showing
how to correctly assemble the puzzle, and always with virtual blocks
to practice during training. All instructions (static and animated) were
colour-coded to match the physical puzzle blocks.
Following training and after a short break, participants were asked
to assemble a 3D printed physical version of the corresponding puzzle
within a given time. Participants were asked to attend a retention
session, two weeks after the training, in which they were asked to
solve the same puzzles in the same order and within the same time
constraints. We measured success rates as well as training and testing
times. Sessions were complemented by a series of mental rotations
tests as well as questionnaires and debrief interviews.
As part of their recommendations for future work, Carlson et al. sug-
gested adding a snap-to-fit function or constraint system [18] to alleviate
the time that virtually trained participants spent attempting to fit and
assemble the virtual blocks [3]. We followed this recommendation
and added such functionality in the virtual training environment. We
also followed their recommendation to make the selection of a block
in the virtual environment to cause a change of colour instead of just
causing a change in transparency, as participants in their study reported
that it was difficult to discern transparent pieces against the transparent
virtual representation of the glove. In their discussion they mentioned
individual differences for interaction between the two hands, as some
participants showed a preference for the haptic device or the glove for
predominant use. We therefore decided to make interaction ambidex-
trous, meaning all operations were designed to be performed equally
by the left hand and the right hand.
We made the following hypotheses:
H1:
The conditions in which the physical blocks were available during
training (PB and PV
I
B) would yield a higher number of successful
puzzle completions during immediate and retention testing. This
relates to the experience (or lack of) built around manipulating
and assembling the physical blocks during training.
H2:
The conditions in which static and animated instructions (video
or 3D animations) were available during training (PV
I
, PV
I
B and
V
E
A
) would result in lower assembly times during immediate
and retention testing, as participants would have received richer
visualisation on how to assemble the blocks during training.
H3:
Condition PV
I
B, with physical blocks and animated instructions
(video), would yield the best performance as measured by imme-
diate and retention success rates and assembly testing times. This
hypothesis is based on H1 and H2.

MURCIA-L´OPEZ AND STEED: A COMPARISON OF VIRTUAL AND PHYSICAL TRAINING TRANSFER OF BIMANUAL ASSEMBLY TASKS 1575
A Comparison of Virtual and Physical Training Transfer
of Bimanual Assembly Tasks
Mar
´
ıa Murcia-L
´
opez and Anthony Steed
Abstract
—As we explore the use of consumer virtual reality technology for training applications, there is a need to evaluate its validity
compared to more traditional training formats. In this paper, we present a study that compares the effectiveness of virtual training and
physical training for teaching a bimanual assembly task. In a between-subjects experiment, 60 participants were trained to solve three
3D burr puzzles in one of six conditions comprised of virtual and physical training elements. In the four physical conditions, training was
delivered via paper- and video-based instructions, with or without the physical puzzles to practice with. In the two virtual conditions,
participants learnt to assemble the puzzles in an interactive virtual environment, with or without 3D animations showing the assembly
process. After training, we conducted immediate tests in which participants were asked to solve a physical version of the puzzles.
We measured performance through success rates and assembly completion testing times. We also measured training times as well
as subjective ratings on several aspects of the experience. Our results show that the performance of virtually trained participants
was promising. A statistically significant difference was not found between virtual training with animated instructions and the best
performing physical condition (in which physical blocks were available during training) for the last and most complex puzzle in terms of
success rates and testing times. Performance in retention tests two weeks after training was generally not as good as expected for all
experimental conditions. We discuss the implications of the results and highlight the validity of virtual reality systems in training.
Index Terms—Learning transfer, virtual reality, assembly, training
1INTRODUCTION
The availability of consumer virtual reality technology has raised the
manufacturing industry’s interest in virtual training for manual assem-
bly tasks. Virtual environments could deliver cost-efficient, safe and
potentially effective training. If proven adequate, virtual training would
also allow for the completion of operator instruction prior to the in-
stallation of physical workstations, tools and components. This would
accelerate the end-to-end manufacturing process and, consequently,
increase efficiency of production. However, more evidence is needed
to ascertain the effectiveness of virtual environments for training as
opposed to more traditional forms of training.
In this paper, we present a study that compares the effectiveness
of virtual and traditional paper- and video-based training transfer of
a bimanual assembly task, motivated by previous research [3, 10]. In
a between-subjects experimental design, participants were trained to
solve three six-piece burr puzzles in a virtual training environment or a
physical training environment. The conditions were designed to account
for situations in which the physical puzzle blocks are available or not
during training. The conditions were also devised to include static
instructions (paper) or combinations of static and animated instructions
(video or 3D animations). Table 1 introduces the experimental condition
types, acronyms and definitions. Table 2 shows a classification of
the experimental conditions according to instruction type and block
availability during training.
Following training, participants were asked to solve physical ver-
sions of the puzzles (referred to as immediate testing). Participants then
completed a retention session, two weeks after the training (referred
to as retention testing). During the course of the study, participants
answered mental rotations tests and questionnaires measuring several
aspects of the experience.
We tested three hypotheses about the effectiveness of training in each
of the conditions being compared in the study. The first hypothesis
Mar
´
ıa Murcia-L
´
opez is with University College London. E-mail:
maria.murcia.13@ucl.ac.uk.
Anthony Steed is with University College London. E-mail:
a.steed@ucl.ac.uk.
(H1) was that conditions in which the physical blocks were available
during training (PB and PV
I
B) would yield a higher number of success-
ful puzzle completions during immediate and retention testing. The
second hypothesis (H2) was that the conditions in which both static
and animated instructions were available during training (PV
I
, PV
I
B
and
V
E
A
) would result in lower assembly times during immediate and
retention testing. The third hypothesis (H3) was that condition PV
I
B,
with animated instructions (video) and physical blocks, would yield the
highest performance as measured by immediate and retention success
rates and assembly testing times. Although we expected some condi-
tions to deliver worse or better performance, we had no hypothesis on
the full order so all the analysis presented in this paper is two-tailed.
Immediate testing results showed some support for the first hypothe-
sis, some support for the second hypothesis and some support for the
third hypothesis. Retention performance was lower than expected for
all conditions both in terms of success rates and completion times and
did not provide evidence to support any of the three hypotheses.
The remainder of this paper is organised as follows. In Section 2
we review related work on learning transfer in immersive mixed reality
systems. Section 3 presents the experimental design and hypotheses.
In Section 4 we introduce the methodology and experimental setup. In
Section 5 we report the results of the study. Section 6 discusses the
results, limitations and future work. Section 7 concludes.
2R
ELATED WORK
Previous research has highlighted the effectiveness of immersive mixed
reality training in different disciplines, including military training, medi-
cal training and vehicle driving simulators [17,21], as well as navigation
and spatial knowledge training [8, 23], amongst others. Despite the
recognised success in the aforementioned fields, studies on immersive
virtual training transfer of procedural and assembly tasks have reported
contrasting results.
Hall and Horwitz compared retention of procedural knowledge of
equipment operation in an immersive virtual environment and in a 2D
computer environment and found no significant differences [7]. They
claimed that virtual reality training may not be superior to conventional
electronic media for training certain skills. Gavish et al. evaluated the
use of virtual reality and augmented reality technology for industrial
maintenance and assembly task training [5]. They concluded that an
augmented reality platform was more suitable for training of this type of
tasks and encouraged further evaluation of virtual reality based training.
Fig. 1. One of the three 3D printed burr puzzles used in the study.
In a more recent study Gonzalez-Franco et al. compared collabo-
rative conventional face-to-face training with a mixed reality training
setup for a manufacturing procedure of an aircraft door [6]. Their re-
sults indicated that performance levels yielded by the immersive mixed
reality training system were not significantly different from the conven-
tional face-to-face training format. Rose et al. evaluated the transfer
from a virtual environment to the real world of a simple sensorimotor
task [16]. Overall, virtual training resulted in equivalent or even bet-
ter real world performance than real or physical training for the task.
However, they advise that their findings may not apply to other types
of training tasks.
Sowndararajan et al. found an effect of level of immersion in memo-
rising a complex procedure [20]. In their study, participants trained in
the system with the higher level of immersion (a large L-shaped projec-
tion display) completed tasks significantly faster and with fewer errors
than participants trained in the system with lower level of immersion
(using a typical laptop display).
Other studies have shown effective learning transfer in virtual en-
vironments with the addition of haptic force-feedback devices. For
instance, Adams et al. conducted a study to explore the benefits of
haptic feedback for virtual training of a manual task [1]. They reported
that force-feedback was a requirement for higher learning transfer in
virtual environments.
Our study is inspired by the work of Carlson et al. in 2015 [3], it-
self motivated by previous work [10, 13, 19]. In a between-subjects
experimental design, Carlson et al. compared the effectiveness of vir-
tual bimanual haptic training versus traditional physical training of
an assembly task consisting of a six-piece burr puzzle. Their results
indicated that physically trained participants initially outperformed
virtually trained participants. However, virtually trained participants
improved their testing times after two weeks. Results also showed that
virtual training was enhanced by using coloured blocks as they helped
participants remember the assembly process. We run a similar task
comparing paper- and video-based training with virtual training in the
absence of a haptic force-feedback device.
We agree with Carlson et al. in that 3D burr puzzles are suitable
proxy tasks or abstractions of context-specific manual assembly tasks,
such as engine assembly operations at vehicle manufacturing plants. We
therefore decided to use the same type of task in our study. Following
their reported methods, we complemented the training task with a series
of mental rotation tests to distribute participants amongst the condition
groups in our between-subjects experimental design [2, 14, 22]. We
also decided to colour-code the puzzle blocks and instructions as well
as to use a semi-transparent virtual representation of the hands in the
virtual environment [11, 12], amongst other recommendations made by
the authors which are further explained in Section 3.
Our study extends and builds on previous work by comparing a
number of virtual and physical training formats, the latter representing
the most common formats (video and paper instructions) in current
assembly process training programmes. The main aim of this research
is to verify whether exposure to a virtual training environment is suffi-
cient for effective training. We are specifically interested in situations
in which haptic devices are not available and when the physical compo-
nents and tools used in the process are not accessible during training.
3E
XPERIMENTAL DESIGN AND HYPOTHESES
Inspired by previous research [3], in our study we used three different
colour-coded versions of a six-piece burr puzzle for the assembly task
(see Figure 1). Burr puzzles have been commonly used for assembly
task training studies in the past because they provide a recognisable and
adequately complex model in which participants must follow a specific
procedure in order to solve them [3, 10]. However, our study differs
from previous work in that no haptic devices were used. In addition, we
are interested in whether consumer virtual reality systems are sufficient
for effective training.
In our study, participants were trained and tested in assembling three
versions of a six-piece burr puzzle. To provide increasing difficulty, the
first three blocks had been preassembled for the first puzzle, the first
two for the second and none for the third. This meant that participants
had to remember a higher number of steps in the assembly process over
the course of the experimental task for each puzzle.
Following a between-subjects experimental design, participants were
trained to solve each puzzle by adding the corresponding unassembled
blocks in one of six experimental conditions (see Table 1). Experimen-
tal conditions were designed to account for scenarios in which blocks
are not available (P and PV
I
), physical blocks are available (PB and
PV
I
B) or virtual blocks are available (
V
E
and
V
E
A
) during training (see
Table 2 for a classification of the experimental conditions). The physi-
cal experimental conditions (P, PB, PV
I
and PV
I
B) were designed to
encompass combinations of paper- and video- based instructions. The
virtual experimental conditions (
V
E
and
V
E
A
) involved a virtual ver-
sion of the paper instructions, with or without 3D animations showing
how to correctly assemble the puzzle, and always with virtual blocks
to practice during training. All instructions (static and animated) were
colour-coded to match the physical puzzle blocks.
Following training and after a short break, participants were asked
to assemble a 3D printed physical version of the corresponding puzzle
within a given time. Participants were asked to attend a retention
session, two weeks after the training, in which they were asked to
solve the same puzzles in the same order and within the same time
constraints. We measured success rates as well as training and testing
times. Sessions were complemented by a series of mental rotations
tests as well as questionnaires and debrief interviews.
As part of their recommendations for future work, Carlson et al. sug-
gested adding a snap-to-fit function or constraint system [18] to alleviate
the time that virtually trained participants spent attempting to fit and
assemble the virtual blocks [3]. We followed this recommendation
and added such functionality in the virtual training environment. We
also followed their recommendation to make the selection of a block
in the virtual environment to cause a change of colour instead of just
causing a change in transparency, as participants in their study reported
that it was difficult to discern transparent pieces against the transparent
virtual representation of the glove. In their discussion they mentioned
individual differences for interaction between the two hands, as some
participants showed a preference for the haptic device or the glove for
predominant use. We therefore decided to make interaction ambidex-
trous, meaning all operations were designed to be performed equally
by the left hand and the right hand.
We made the following hypotheses:
H1:
The conditions in which the physical blocks were available during
training (PB and PV
I
B) would yield a higher number of successful
puzzle completions during immediate and retention testing. This
relates to the experience (or lack of) built around manipulating
and assembling the physical blocks during training.
H2:
The conditions in which static and animated instructions (video
or 3D animations) were available during training (PV
I
, PV
I
B and
V
E
A
) would result in lower assembly times during immediate
and retention testing, as participants would have received richer
visualisation on how to assemble the blocks during training.
H3:
Condition PV
I
B, with physical blocks and animated instructions
(video), would yield the best performance as measured by imme-
diate and retention success rates and assembly testing times. This
hypothesis is based on H1 and H2.

1576 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 24, NO. 4, APRIL 2018
Table 1. Experimental condition types, acronyms and definitions. See
Table 2 for a classification of the experimental conditions according to
instruction type and block availability during training. Please note the
choice of acronym V
I
to represent video and V
E
to represent virtual envi-
ronment to avoid any confusion in making reference to the experimental
conditions throughout the paper.
Type Acronym Definition
Physical
P Paper instructions
PB Paper instructions and physical blocks
PV
I
Paper instructions and assembly process video
PV
I
B Paper instructions, assembly process video and physical blocks
Virtual
V
E
Virtual paper instructions and virtual blocks
V
E
A Virtual paper instructions and virtual blocks, with assembly process animations
Table 2. Classification of the experimental conditions according to in-
struction type (static or static and animated) and block availability (no
blocks, physical blocks or virtual blocks) during training. See Table 1 for
experimental condition types, acronyms and definitions.
Physical Virtual
No blocks Physical blocks Virtual blocks
Static instructions P PB V
E
Static and animated instructions PV
I
PV
I
BV
E
A
Fig. 2. Phyiscal lab where the experiment took place (left) and analogous
virtual environment (right).
4METHOD
4.1 Participants
A total of 60 participants (30 female, 30 male; average age 26.51 years,
SD = 6.47
) were recruited from the student and staff population at
University College London (
UCL
). All participants signed a consent
form and the study was approved by the
UCL
Research Ethics Commit-
tee (Project ID: 6708/004). Participants were paid
£
15 for participation.
A screener questionnaire was used to filter out potential participants
who enjoy solving 3D puzzles or who have any type of colour-blindness.
Eligible participants were assigned to the different experimental condi-
tions based on individual results for Purdue’s Visualisation of Rotations
Test [2] to avoid any possible bias between groups, ensuring a similar
mean score for the test in each of the experimental condition groups.
Likewise, an equal number of females and males were assigned to each
group.
4.2 Materials
The user study was conducted in a lab at
UCL
. The room consisted of a
3.1 meters long by 2.7 meters wide by 4.0 meters high room. A virtual
replica of the laboratory was modeled for the virtual enviornment used
in the virtual experimental conditions. Figure 2 contains images of
the physical room and analogous virtual environment. An Oculus Rift
Consumer Version 1, two Oculus Touch controllers and two Oculus
sensors were used for the virtual experimental conditions. The virtual
environment was rendered at scale 1:1 in Unity 5.6.0 without VSync at
90FPS in each eye on an Intel Core i7-4770K CPU @ 3.50GHz, with
16GB
RAM and Nvidia GeForce GTX 1080 GPU running Windows
8.1 Pro. The Oculus Avatar SDK 1.15.0 [9] was used to include hand
presence and interaction for the Oculus Touch controllers. The Burr
Tools 0.6.3 software was used to digitally create and solve the three
versions of the six-piece burr puzzles as well as to generate the paper
instructions and assembly process videos [15]. The physical puzzle
blocks were 3D printed using a Ultimaker 2+ 3D printer with a 0.4mm
nozzle and standard settings, with PLA 3D printing material. Preassem-
bled blocks for the first and second puzzles were glued together. Paper
instructions were printed on A3 paper and attached to 5mm A3 foam-
boards. Assembly videos were presented using VLC 2.2.3 on a 13-inch
mid 2014 MacBook Pro laptop running macOS 10.12.2.
4.3 Physical training environment
Participants assigned to the physical experimental conditions (P, PB,
PV
I
and PV
I
B) were seated on a stool in front of the table in the lab
on which the blocks had been placed in the correct initial configuration
for each puzzle. Participants were seated facing the table and were told
that they could adjust the distance to it if they wished to.
Paper instructions were designed to show the initial configuration of
the blocks at the top and the assembly process steps at the bottom (see
Figure 3). For the first two puzzles, blocks that had been preassembled
and the corresponding steps in the assembly process were faded out.
The orientation of the images of the blocks in the instructions was
randomly selected for each puzzle. For those experimental conditions
involving paper instructions, these were placed against the wall on
the table in front of the participant. Assembly process videos were
generated using Burr Tools [15] and showed a step-by-step animation
of the assembly process from the perspective matching the one in the
paper instructions. The laptop was placed on the table in front of the
participant. Participants could interact with the video (play, pause, stop,
rewind, and fast forward) using the VLC user interface.
For those experimental conditions in which the physical blocks were
available during training (PB and PV
I
B) these were initially placed on
the table following the same configuration as the paper instructions.
Preassembled puzzles were placed behind the blocks.
4.4 Virtual training environment
Participants assigned to the virtual experimental conditions (
V
E
and
V
E
A
) were seated on a stool in the center of the lab. They were
then asked to put on the Oculus Rift and hold the two Oculus Touch
controllers with the experimenter’s help. The virtual environment
showed the virtual replica of the room and table used in the physical
environment in front of them, with the blocks for the corresponding
puzzle arranged in the correct configuration. Participants were seated
facing the virtual table and were told that they could adjust the distance
to it if they wished to. For the first two puzzles (in which two or
three of the blocks had been preassembled) participants could see the
preassembled puzzle hovering over the table in front of them. Virtual
paper instructions were presented against the wall on the table in the
same location as the physical paper instructions were presented in the
physical training environment.
Using the Oculus Avatar SDK 1.15.0 [9], virtual hands were ren-
dered using the default shader (see Figure 4). Participants could then
manipulate the 3D environment by grabbing the virtual puzzle blocks.
They could hold the trigger button to grab unassembled puzzle blocks
and the grip button to move and rotate assembled blocks as a single
unit. Participants could grab any block at any given time, but only the
correct block in the assembly process could be attached to the puzzle.
No physics constraints were added to the blocks meaning they could
be moved through each other and through the virtual hands and table.
Visual feedback was provided to aid participants in learning the as-
sembly process during training. When participants grabbed the correct
block in the assembly process, a blue transparent preview block was
shown in the puzzle, indicating where the block had to be assembled.
Participants had the option to deactivate the block preview. A blue
highlight was used to indicate what the next block in the assembly
process was. This highlight would then turn to red when the block
collided with the preview block, indicating that the piece was near its
correct location but in the wrong orientation. The highlight would turn
green when the block was within an angle of twenty degrees from the
Fig. 3. Assembly instruction sheet for each of the three burr puzzles
used in the study. Each instruction sheet contains a diagram of the six
pieces and five ordered steps needed to solve the puzzle. Preassembled
pieces and steps for Puzzles 1 and 2 were faded out.
correct orientation. If the participant released the trigger when the
block showed a green highlight, it would snap into the correct location
and the participant could move on to assemble the next piece or reset
the puzzle. No audio or vibration feedback was used in the experience.
A user interface with virtual buttons was added on the right-hand
side of the virtual table. Buttons were represented by blue spheres
which the participant could interact with by touching them, after which
they would turn to grey and back to blue to indicate that the interaction
was successful. For participants in the
V
E
and
V
E
A
conditions, two
buttons were available: RESET and HELP ON/OFF. Interacting with
the RESET button would immediately relocate all blocks in their initial
positions so participants could restart the assembly process whenever
they wished. The HELP ON/OFF button acted as a toggle to activate
and deactivate the blue transparent preview of the block in the puzzle
so participants could practice assembling the puzzle with and without
the visual aid.
For participants in the
V
E
A
condition, two more buttons were added:
NEXT STEP and REPLAY LAST STEP. The NEXT STEP button
would trigger the animation of the assembly of the next block in the
process. The REPLAY LAST STEP would reposition the last block
assembled in its original location on the table and animate its assembly
onto the puzzle.
All interactions in the virtual training environment could be equally
carried out using either hand and participants could concurrently com-
plete one interaction with each hand. For example, a participant could
grab and rotate the assembled pieces with one hand and grab the next
block to attach with the other hand.
Fig. 4. Screenshot of a participant grabbing a virtual block and assem-
bling it onto the 3D puzzle. The green highlight indicates on the block is
colliding with its preview block and within twenty degrees from the correct
orientation. By releasing the trigger button of the Oculus Touch controller
the virtual block would snap into its correct location.
4.5 Procedure
The experimental task consisted of two lab sessions. The first session
comprised training and immediate testing. The second session, two
weeks after the first, comprised retention testing. Figure 5 shows an
outline of the experimental task. Before the first lab session, participants
were asked to read and sign an online informed consent form and answer
a digital version of Purdue’s Visualisation of Rotations Test [2] used to
pre-allocate participants to the experimental conditions. Participants
also answered a background questionnaire with a specific focus on
prior experience with videogames, 3D modelling software and virtual
environments.
During the first lab session, participants were asked to sign a paper
copy of the consent form and asked to read an information sheet with
written instructions describing the experimental task. In this session,
participants completed a familiarisation task and three trials, each with
a training and a testing stage. The three trials corresponded with each
of the three burr puzzles in increasing order of difculty. During the
familiarisation task participants were introduced to the physical or
virtual training environment depending on the experimental condition
they had been assigned to. A sample assembly task involving piling up
rectangular blocks was used and participants were able to familiarise
themselves with the paper instruction format, the video player and the
interactive virtual environment, accordingly.
For each of the trials, the training stage involved learning to assemble
the corresponding puzzle in one of the six experimental conditions in a
maximum time of eight minutes. During the testing stage participants
were asked to assemble the physical 3D puzzle in a maximum of three
minutes. Time limits for training and testing were defined through
piloting of the experimental task. In each trial participants completed
the training stage and, after a thirty second break, the testing stage. They
then completed a questionnaire at the end of each trial (see Table 3).
For both training and testing, participants were told what the limit
times were and were advised that they could end the stage before the
time expired if they wished to. Participants were also told that the
initial configuration of the blocks on the table during training would
match the initial configuration of the blocks during testing and the
paper instructions.
Participants were asked to try their best if they were in doubt as
to how to assemble the puzzles during testing. An experimenter was
present at all times during the experimental task to manage cables for
those particpiants in the virtual experimental conditions and provide
guidance on the different phases of the experimental task. After com-
pleting all trials, participants were interviewed regarding the strategies

MURCIA-L´OPEZ AND STEED: A COMPARISON OF VIRTUAL AND PHYSICAL TRAINING TRANSFER OF BIMANUAL ASSEMBLY TASKS 1577
Table 1. Experimental condition types, acronyms and definitions. See
Table 2 for a classification of the experimental conditions according to
instruction type and block availability during training. Please note the
choice of acronym V
I
to represent video and V
E
to represent virtual envi-
ronment to avoid any confusion in making reference to the experimental
conditions throughout the paper.
Type Acronym Definition
Physical
P Paper instructions
PB Paper instructions and physical blocks
PV
I
Paper instructions and assembly process video
PV
I
B Paper instructions, assembly process video and physical blocks
Virtual
V
E
Virtual paper instructions and virtual blocks
V
E
A Virtual paper instructions and virtual blocks, with assembly process animations
Table 2. Classification of the experimental conditions according to in-
struction type (static or static and animated) and block availability (no
blocks, physical blocks or virtual blocks) during training. See Table 1 for
experimental condition types, acronyms and definitions.
Physical Virtual
No blocks Physical blocks Virtual blocks
Static instructions P PB V
E
Static and animated instructions PV
I
PV
I
BV
E
A
Fig. 2. Phyiscal lab where the experiment took place (left) and analogous
virtual environment (right).
4METHOD
4.1 Participants
A total of 60 participants (30 female, 30 male; average age 26.51 years,
SD = 6.47
) were recruited from the student and staff population at
University College London (
UCL
). All participants signed a consent
form and the study was approved by the
UCL
Research Ethics Commit-
tee (Project ID: 6708/004). Participants were paid
£
15 for participation.
A screener questionnaire was used to filter out potential participants
who enjoy solving 3D puzzles or who have any type of colour-blindness.
Eligible participants were assigned to the different experimental condi-
tions based on individual results for Purdue’s Visualisation of Rotations
Test [2] to avoid any possible bias between groups, ensuring a similar
mean score for the test in each of the experimental condition groups.
Likewise, an equal number of females and males were assigned to each
group.
4.2 Materials
The user study was conducted in a lab at
UCL
. The room consisted of a
3.1 meters long by 2.7 meters wide by 4.0 meters high room. A virtual
replica of the laboratory was modeled for the virtual enviornment used
in the virtual experimental conditions. Figure 2 contains images of
the physical room and analogous virtual environment. An Oculus Rift
Consumer Version 1, two Oculus Touch controllers and two Oculus
sensors were used for the virtual experimental conditions. The virtual
environment was rendered at scale 1:1 in Unity 5.6.0 without VSync at
90FPS in each eye on an Intel Core i7-4770K CPU @ 3.50GHz, with
16GB
RAM and Nvidia GeForce GTX 1080 GPU running Windows
8.1 Pro. The Oculus Avatar SDK 1.15.0 [9] was used to include hand
presence and interaction for the Oculus Touch controllers. The Burr
Tools 0.6.3 software was used to digitally create and solve the three
versions of the six-piece burr puzzles as well as to generate the paper
instructions and assembly process videos [15]. The physical puzzle
blocks were 3D printed using a Ultimaker 2+ 3D printer with a 0.4mm
nozzle and standard settings, with PLA 3D printing material. Preassem-
bled blocks for the first and second puzzles were glued together. Paper
instructions were printed on A3 paper and attached to 5mm A3 foam-
boards. Assembly videos were presented using VLC 2.2.3 on a 13-inch
mid 2014 MacBook Pro laptop running macOS 10.12.2.
4.3 Physical training environment
Participants assigned to the physical experimental conditions (P, PB,
PV
I
and PV
I
B) were seated on a stool in front of the table in the lab
on which the blocks had been placed in the correct initial configuration
for each puzzle. Participants were seated facing the table and were told
that they could adjust the distance to it if they wished to.
Paper instructions were designed to show the initial configuration of
the blocks at the top and the assembly process steps at the bottom (see
Figure 3). For the first two puzzles, blocks that had been preassembled
and the corresponding steps in the assembly process were faded out.
The orientation of the images of the blocks in the instructions was
randomly selected for each puzzle. For those experimental conditions
involving paper instructions, these were placed against the wall on
the table in front of the participant. Assembly process videos were
generated using Burr Tools [15] and showed a step-by-step animation
of the assembly process from the perspective matching the one in the
paper instructions. The laptop was placed on the table in front of the
participant. Participants could interact with the video (play, pause, stop,
rewind, and fast forward) using the VLC user interface.
For those experimental conditions in which the physical blocks were
available during training (PB and PV
I
B) these were initially placed on
the table following the same configuration as the paper instructions.
Preassembled puzzles were placed behind the blocks.
4.4 Virtual training environment
Participants assigned to the virtual experimental conditions (
V
E
and
V
E
A
) were seated on a stool in the center of the lab. They were
then asked to put on the Oculus Rift and hold the two Oculus Touch
controllers with the experimenter’s help. The virtual environment
showed the virtual replica of the room and table used in the physical
environment in front of them, with the blocks for the corresponding
puzzle arranged in the correct configuration. Participants were seated
facing the virtual table and were told that they could adjust the distance
to it if they wished to. For the first two puzzles (in which two or
three of the blocks had been preassembled) participants could see the
preassembled puzzle hovering over the table in front of them. Virtual
paper instructions were presented against the wall on the table in the
same location as the physical paper instructions were presented in the
physical training environment.
Using the Oculus Avatar SDK 1.15.0 [9], virtual hands were ren-
dered using the default shader (see Figure 4). Participants could then
manipulate the 3D environment by grabbing the virtual puzzle blocks.
They could hold the trigger button to grab unassembled puzzle blocks
and the grip button to move and rotate assembled blocks as a single
unit. Participants could grab any block at any given time, but only the
correct block in the assembly process could be attached to the puzzle.
No physics constraints were added to the blocks meaning they could
be moved through each other and through the virtual hands and table.
Visual feedback was provided to aid participants in learning the as-
sembly process during training. When participants grabbed the correct
block in the assembly process, a blue transparent preview block was
shown in the puzzle, indicating where the block had to be assembled.
Participants had the option to deactivate the block preview. A blue
highlight was used to indicate what the next block in the assembly
process was. This highlight would then turn to red when the block
collided with the preview block, indicating that the piece was near its
correct location but in the wrong orientation. The highlight would turn
green when the block was within an angle of twenty degrees from the
Fig. 3. Assembly instruction sheet for each of the three burr puzzles
used in the study. Each instruction sheet contains a diagram of the six
pieces and five ordered steps needed to solve the puzzle. Preassembled
pieces and steps for Puzzles 1 and 2 were faded out.
correct orientation. If the participant released the trigger when the
block showed a green highlight, it would snap into the correct location
and the participant could move on to assemble the next piece or reset
the puzzle. No audio or vibration feedback was used in the experience.
A user interface with virtual buttons was added on the right-hand
side of the virtual table. Buttons were represented by blue spheres
which the participant could interact with by touching them, after which
they would turn to grey and back to blue to indicate that the interaction
was successful. For participants in the
V
E
and
V
E
A
conditions, two
buttons were available: RESET and HELP ON/OFF. Interacting with
the RESET button would immediately relocate all blocks in their initial
positions so participants could restart the assembly process whenever
they wished. The HELP ON/OFF button acted as a toggle to activate
and deactivate the blue transparent preview of the block in the puzzle
so participants could practice assembling the puzzle with and without
the visual aid.
For participants in the
V
E
A
condition, two more buttons were added:
NEXT STEP and REPLAY LAST STEP. The NEXT STEP button
would trigger the animation of the assembly of the next block in the
process. The REPLAY LAST STEP would reposition the last block
assembled in its original location on the table and animate its assembly
onto the puzzle.
All interactions in the virtual training environment could be equally
carried out using either hand and participants could concurrently com-
plete one interaction with each hand. For example, a participant could
grab and rotate the assembled pieces with one hand and grab the next
block to attach with the other hand.
Fig. 4. Screenshot of a participant grabbing a virtual block and assem-
bling it onto the 3D puzzle. The green highlight indicates on the block is
colliding with its preview block and within twenty degrees from the correct
orientation. By releasing the trigger button of the Oculus Touch controller
the virtual block would snap into its correct location.
4.5 Procedure
The experimental task consisted of two lab sessions. The first session
comprised training and immediate testing. The second session, two
weeks after the first, comprised retention testing. Figure 5 shows an
outline of the experimental task. Before the first lab session, participants
were asked to read and sign an online informed consent form and answer
a digital version of Purdue’s Visualisation of Rotations Test [2] used to
pre-allocate participants to the experimental conditions. Participants
also answered a background questionnaire with a specific focus on
prior experience with videogames, 3D modelling software and virtual
environments.
During the first lab session, participants were asked to sign a paper
copy of the consent form and asked to read an information sheet with
written instructions describing the experimental task. In this session,
participants completed a familiarisation task and three trials, each with
a training and a testing stage. The three trials corresponded with each
of the three burr puzzles in increasing order of difculty. During the
familiarisation task participants were introduced to the physical or
virtual training environment depending on the experimental condition
they had been assigned to. A sample assembly task involving piling up
rectangular blocks was used and participants were able to familiarise
themselves with the paper instruction format, the video player and the
interactive virtual environment, accordingly.
For each of the trials, the training stage involved learning to assemble
the corresponding puzzle in one of the six experimental conditions in a
maximum time of eight minutes. During the testing stage participants
were asked to assemble the physical 3D puzzle in a maximum of three
minutes. Time limits for training and testing were defined through
piloting of the experimental task. In each trial participants completed
the training stage and, after a thirty second break, the testing stage. They
then completed a questionnaire at the end of each trial (see Table 3).
For both training and testing, participants were told what the limit
times were and were advised that they could end the stage before the
time expired if they wished to. Participants were also told that the
initial configuration of the blocks on the table during training would
match the initial configuration of the blocks during testing and the
paper instructions.
Participants were asked to try their best if they were in doubt as
to how to assemble the puzzles during testing. An experimenter was
present at all times during the experimental task to manage cables for
those particpiants in the virtual experimental conditions and provide
guidance on the different phases of the experimental task. After com-
pleting all trials, participants were interviewed regarding the strategies

1578 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 24, NO. 4, APRIL 2018
Fig. 5. Overview of the experimental procedure.
used throughout the sessions.
After a waiting period of two weeks, participants returned to the
lab for the second session. In this session participants were asked to
complete a paper version of the Vandenberg and Kuse Mental Rotations
Test [22]. They then completed the retention test for each of the three
puzzles, in which they were asked to solve the three burr puzzles from
the first session without a training phase, in the same order and in a
maximum of three minutes. They completed the same questionnaire
from the first session at the end of each retention trial (see Table 3).
After completing all retention trials they were interviewed regarding
strategies used throughout the session.
5R
ESULTS
5.1 Types of errors
Unsuccessful puzzle completions during immediate and retention test-
ing were due to one of two reasons. In most cases, participants did not
complete the 3D puzzles within the given maximum time (180s). On
the other hand, a low number of participants decided to stop the time
before the upper limit thinking that they had successfully solved the
puzzle. However, close inspection showed that they had not correctly
assembled the pieces. Completion time values for both immediate and
retention testing were corrected by assigning the upper time limit (180s)
to all unsuccessful attempts.
5.2 First session
5.2.1 Training times
Boxplots with training times for each of the puzzles are shown in
Figure 6. Non-parametric statistical analysis was performed for training
times because our data was not normally distributed as shown by a
Shapiro-Wilk test.
A Kruskal-Wallis H test showed that there was an overall statistically
significant difference in training times for the first puzzle between the
different experimental conditions,
χ
2
(5)=25.648, p < 0.001
, with a
mean rank score of 15.35 for P, 38.85 for PB, 13.85 for PV
I
, 40.15 for
PV
I
B, 36.25 for V
E
and 38.55 for V
E
A.
A Kruskal-Wallis H test showed that there was an overall statistically
significant difference in training times for the second puzzle between
the different experimental conditions,
χ
2
(5)=22.764, p < 0.001
, with
a mean rank score of 22.50 for P, 40.00 for PB, 10.60 for PV
I
, 36.70
for PV
I
B, 37.45 for V
E
and 35.75 for V
E
A.
A Kruskal-Wallis H test showed that there was no overall statistically
significant difference in training times for the third puzzle between the
different experimental conditions,
χ
2
(5)=10.701, p = 0.058
, with a
mean rank score of 21.95 for P, 33.50 for PB, 18.90 for PV
I
, 35.70 for
PV
I
B, 37.15 for V
E
and 35.80 for V
E
A.
Pairwise comparisons were performed using Dunn’s procedure [4]
with a Bonferroni correction for multiple comparisons with adjusted p-
values. These are displayed in Table 4. Note that pairwise comparisons
for puzzles in which the Kruskal-Wallis H test showed no overall
statistically significant difference have not been included.
The post hoc analysis revealed statistically significant differences in
training times for the first puzzle. There was a statistically significant
difference between P (mean rank = 15.35) and PB (mean rank = 38.85)
(
p = 0.036
), PV
I
B (mean rank = 40.15) (
p = 0.020
) and V
E
A (mean
rank = 38.55) (
p = 0.041
). There was also a statistically significant
difference between PV
I
(mean rank = 13.85) and PB (mean rank =
38.85) (
p = 0.018
), PV
I
B (mean rank = 40.15) (
p = 0.010
) and V
E
A
(mean rank = 38.55) (p = 0.021).
The post hoc analysis revealed statistically significant differences in
training times for the second puzzle. There was a statistically significant
difference between PV
I
(mean rank = 10.60) and PB (mean rank =
40.00) (
p = 0.002
), PV
I
B (mean rank = 36.70) (
p = 0.010
), V
E
(mean
rank = 37.45) (p = 0.007) and V
E
A (mean rank = 35.75) (p = 0.015).
5.2.2 Immediate testing success rates
A binomial logistic regression was performed to ascertain the effects
of experimental condition on the likelihood that participants succeed
at assembling each puzzle during the immediate testing phase. Fig-
ure 7 shows the number of successful and unsuccessful completions of
each puzzle for all experimental conditions. PV
I
B was chosen as the
reference category as this was the condition that produced the highest
number of successful puzzle completions, overall.
The binomial logistic regression model was not statistically sig-
nificant,
χ
2
(5)=8.809, p = 0.117
for the first puzzle. The model
explained 18.3% (Nagelkerke R
2
) of the variance in success rate and
correctly classified 61.7% of cases. The Wald criterion demonstrated
that only condition P made a significant contribution to prediction
(p = 0.016). The model suggested that participants in this condition
were 0.05 times as likely to successfully assemble the first puzzle than
participants in the reference category (PV
I
B).
The binomial logistic regression model was statistically significant,
χ
2
(5)=12.016, p = 0.035
for the second puzzle. The model explained
24.7% (Nagelkerke R
2
) of the variance in success rate and correctly
classified 71.7% of cases. The Wald criterion demonstrated that P
and PV
I
made a significant contribution to prediction (
p = 0.016
and
p = 0.035, respectively). The model suggested that participants in the
P experimental condition were 0.048 times as likely to successfully
assemble the second puzzle than participants in the reference category
(PV
I
B). The model suggested that participants in the PV
I
experimen-
tal condition were 0.074 times as likely to successfully assemble the
second puzzle than participants in the reference category (PV
I
B).
The binomial logistic regression model was statistically significant,
χ
2
(5)=24.255, p < 0.001
for the third puzzle. The model explained
45.8% (Nagelkerke R
2
) of the variance in success rate and correctly
classified 78.3% of cases. The Wald criterion demonstrated that P and
PV
I
made a significant contribution to prediction (p = 0.035 and p
= 0.007, respectively). The model suggested that participants in the
P experimental condition were 0.074 times as likely to successfully
assemble the third puzzle than participants in the reference category
(PV
I
B). The model suggested that participants in the PV
I
experimental
condition were 0.028 times as likely to successfully assemble the third
puzzle than participants in the reference category (PV
I
B). Condition
Table 3. Trial questionnaire.
Dimension Question Likert scale extremes
Difficulty Please rate the difficulty of the task you just completed. 1: Very difficult - 5: Very easy
Ease of use Please rate the ease of use in assembling parts in the training environment. 1: Very difficult - 5: Very easy
Seriousness Please rate how seriously you took the task. 1: Very unseriously - 5: Very seriously
Fig. 6. Boxplot containing training times for each of the puzzles. Medians are shown as dark horizontal lines. Boxes represent the interquartile ranges
(IQR). Whiskers represent either the extreme data points or extend to 1.5 x IQR. Outliers (data points outside the whiskers) are shown by circles. A
value, X, is an outlier if X < lower quartile 1.5 x interquartile range or if X < upper quartile + 1.5 x IQR. See Table 4 for pairwise interactions.
Fig. 7. Number of successful (green) and failed (red) attempts at solving the three puzzles in the immediate testing phase for each of the experimental
conditions.
V
E
A
did not contribute to this model (Wald = .000). However, it is
important to note that all participants in this condition successfully
completed the third puzzle.
A binomial logistic regression was then performed to ascertain the
effects of successful completion of the first puzzle on the likelihood
that participants succeed at assembling the second puzzle during the
immediate testing phase. The logistic regression model was statistically
significant,
χ
2
(1)=12.993, p < 0.001
. The model explained 26.5%
(Nagelkerke R
2
) of the variance in success rate and correctly classified
73.3% of cases. The model suggested that participants who succeeded
at correctly assempling the first puzzle were 7.65 times as likely to suc-
cessfully assemble the second puzzle than participants in the reference
category (PV
I
B).
A binomial logistic regression was also performed to ascertain the
effects of successful completion of the second puzzle on the likelihood
that participants succeed at assembling the third puzzle during the
immediate testing phase. The logistic regression model was statistically
significant,
χ
2
(1)=15.174, p < 0.001
. The model explained 30.8%

Citations
More filters

Journal ArticleDOI
Abstract: Background: This review focused on how immersive head-mounted display virtual reality (VR) was used in post-secondary level education and skill training, with the aim to better understand its state of the art as found from the literature. While numerous studies describe the use of immersive VR within a specific educational setting, they are often standalone events not fully detailed regarding their curricular integration. This review aims to analyse these events, with a focus on immersive VR’s incorporation into post-secondary education. Objectives: O1) Review the existing literature on the use of immersive VR in post-secondary settings, determining where and how it has been used within each educational discipline. This criterion focused on literature featuring the use of immersive VR, due to its influence on a user’s perceived levels of presence and imagination. O2) Identify favourable outcomes from the use of immersive VR when compared to other learning methods. O3) Determine the conceptual rationale (purpose) for each implementation of immersive VR as found throughout the literature. O4) Identify learning theories and recommendations for the utilization of immersive VR in post-secondary education. Methods: A literature review was undertaken with searches of Education Research Complete, ERIC, MEDLINE, EMBASE, IEEE Xplore, Scopus and Web of Science: Core Collection to locate reports on the use of immersive VR in post-secondary curricula. Results: 119 articles were identified, featuring disciplines across Arts and Humanities, Health Sciences, Military and Aerospace, Science and Technology. 35 out of 38 experiments reported to have found a positive outcome for immersive VR, after being compared with a non-immersive platform. Each simulation’s purpose included one or more of the following designations: skill training, convenience, engagement, safety, highlighting, interactivity, team building and suggestion. Recommendations for immersive VR in post-secondary education emphasize experiential learning and social constructivist approaches, including student-created virtual environments that are mainly led by the students themselves under team collaboration. Conclusion: Immersive VR brings convenient, engaging and interactive alternatives to traditional classroom settings as well as offers additional capability over traditional methods. There is a diverse assortment of educational disciplines that have each attempted to harness the power of this technological medium.

19 citations


Journal ArticleDOI
TL;DR: Developing MR:emin, an immersive MR music learning environment for the theremin, an electronic music instrument that is controlled without physical contact, shows that performance error during training is significantly smaller in the immersive MR environment.
Abstract: Learning music is a challenging process that requires years of practice to master, either with lessons from a professional teacher or through self-teaching. While practicing, students are expected to self-evaluate their performance which may be difficult without timely feedback from a professional. Research into computer-assisted music instrument tutoring (CAMIT) attempts to address this through the use of emerging technologies. In this paper, we study CAMIT for mixed reality (MR) by developing MR:emin, an immersive MR music learning environment for the theremin, an electronic music instrument that is controlled without physical contact. MR:emin integrates a physical theremin with the immersive learning environment. To better understand the effectiveness of such environments, we perform a user study with MR:emin comparing traditional music learning with two virtual learning environments, an immersive one and a non-immersive one. In a between-groups study, 30 participants were trained to play a sequence of notes on the theremin using one of the three training environments. Results of our statistical analysis show that performance error during training is significantly smaller in the immersive MR environment. This does not necessarily lead to improved performance after training; analysis of post-training improvement indicates that immersive training results in the smallest amount of improvement. Participants, however, indicate that the MR:emin environment is more engaging and increases confidence during practice. We discuss potential factors leading to the decrease in learning and provide some environment guidelines to aid in the design of engaging immersive music learning environments.

11 citations


Cites background from "A Comparison of Virtual and Physica..."

  • ...2016; Murcia-Lopez and Steed 2018; Werrlich et al....

    [...]

  • ...Also using burr puzzles, Murcia-Lopez and Steed (2018) sought to understand the effects of XR training when haptic devices or physical objects are not available during the training process....

    [...]


Journal ArticleDOI
TL;DR: The combination of virtual reality training and conventional physical training with COMB appears to have the potential to speed up the recovery of motor control and rehabilitation of women with MS.
Abstract: As their illness progresses, patients with Multiple Sclerosis (MS) may suffer from motor impairments. In the present study, we examined the effectiveness of three interventions for learning a bimanual coordination task: Virtual reality training (VRT), conventional physical training (CPT), and the combination of VRT and CPT (COMB). A total of 45 women with MS were randomly assigned to one of the following study conditions: VRT, CPT or COMB. Bimanual coordination was assessed at baseline, eight weeks later at study completion, and 4 weeks after that at follow-up. Bimanual coordination improved over time from baseline to study completion and to follow-up. Compared to the VRT and CPT conditions, the COMB condition led to higher coordination accuracy and consistency. The combination thus appears to have the potential to speed up the recovery of motor control and rehabilitation of women with MS.

9 citations


Cites background or methods or result from "A Comparison of Virtual and Physica..."

  • ...Given the background described above and based on previous research (Grealy et al., 1999; Lozano-Quilis et al., 2013; Murcia-López & Steed, 2018; Sampson et al., 2016), the following hypothesis was formulated: we expected that, compared to VRT and CPT, combining these two methods would have a more…...

    [...]

  • ...…training are the activation of pathways, the increase in the number of synapses, and the increase in the sensory area that is seen in plasticization (Ashton-Miller, Wojtys, Huston, & Fry-Welch, 2001; Baram & Miller, 2006; Grealy et al., 1999; Lozano-Quilis et al., 2013; Murcia-López & Steed, 2018)....

    [...]

  • ...Psychological problems have been reported for women with MS in addition to their motor and sensory problems (Lozano-Quilis et al., 2013; Murcia-López & Steed, 2018)....

    [...]

  • ...Moreover, a facilitative sense of control and confidence can be observed following VRT (Massetti et al., 2016; Murcia-López & Steed, 2018)....

    [...]

  • ...These results are consistent with those reported in previous studies (Grealy et al., 1999; LozanoQuilis et al., 2013; Massetti et al., 2016; Murcia-López & Steed, 2018; Sampson et al., 2016; Song et al., 2010)....

    [...]


Journal ArticleDOI
10 Dec 2020
TL;DR: A review of the current state-of-the-art of use of XR technologies in training personnel in the field of manufacturing and presents several key application domains where XR is being currently applied, notably in maintenance training and in performing assembly task.
Abstract: Recently, the use of extended reality (XR) systems has been on the rise, to tackle various domains such as training, education, safety, etc. With the recent advances in augmented reality (AR), virtual reality (VR) and mixed reality (MR) technologies and ease of availability of high-end, commercially available hardware, the manufacturing industry has seen a rise in the use of advanced XR technologies to train its workforce. While several research publications exist on applications of XR in manufacturing training, a comprehensive review of recent works and applications is lacking to present a clear progress in using such advance technologies. To this end, we present a review of the current state-of-the-art of use of XR technologies in training personnel in the field of manufacturing. First, we put forth the need of XR in manufacturing. We then present several key application domains where XR is being currently applied, notably in maintenance training and in performing assembly task. We also reviewed the applications of XR in other vocational domains and how they can be leveraged in the manufacturing industry. We finally present some current barriers to XR adoption in manufacturing training and highlight the current limitations that should be considered when looking to develop and apply practical applications of XR.

7 citations


Proceedings ArticleDOI
23 Mar 2019
TL;DR: This work replicated in VR an experimental paradigm employed to investigate grasping behavior in the real world, and tracked participants' forefinger and thumb as they picked up unfamiliar objects presented at different orientations, and exhibiting the same physics behavior of their real counterparts.
Abstract: Grasping is one of the fundamental actions we perform to interact with objects in real environments, and in the real world we rarely experience difficulty picking up objects. Grasping plays a fundamental role for interactive virtual reality (VR) systems that are increasingly employed not only for recreational purposes, but also for training in industrial contexts, in medical tasks, and for rehabilitation protocols. To ensure the effectiveness of such VR applications, we must understand whether the same grasping behaviors and strategies employed in the real world are adopted when interacting with objects in VR. To this aim, we replicated in VR an experimental paradigm employed to investigate grasping behavior in the real world. We tracked participants' forefinger and thumb as they picked up, in a VR environment, unfamiliar objects presented at different orientations, and exhibiting the same physics behavior of their real counterparts. We compared grasping behavior within and across participants, in VR and in the corresponding real world situation. Our findings highlight the similarities and differences in grasping behavior in real and virtual environments.

7 citations


Cites background from "A Comparison of Virtual and Physica..."

  • ...Understanding how to wield AR/VR tools to reshape how we perceive the world also has incredible potential for societal and clinical applications: in particular, for training in industrial tasks [6, 12, 28], in medical and surgical applications [8],or for rehabilitation protocols [17, 32]....

    [...]


References
More filters

Journal ArticleDOI
Abstract: This paper considers the use of rank sums from a combined ranking of k independent samples in order to decide which populations differ. Such a procedure is suggested as a convenient alternative to making separate rankings for each pair of samples, and the two methods are compared. Asymptotic use of the normal tables is given and the treatment of ties is discussed. A numerical example is given.

2,822 citations


Journal ArticleDOI
Abstract: A new paper-and-pencil test of spatial visualization was constructed from the figures used in the chronometric study of Shepard and Metzler (1971). In large samples, the new test displayed substantial internal consistency (Kuder-Richardson 20 = .88), a test-retest reliability (.83), and consistent sex differences over the entire range of ages investigated. Correlations with other measures indicated strong association with tests of spatial visualization and virtually no association with tests of verbal ability.

2,167 citations


Journal ArticleDOI
TL;DR: The challenges each of these factors present to the effective design of virtual environments and systematic approaches to the resolution of each of them are discussed.
Abstract: Virtual environments are envisioned as being systems that will enhance the communication between humans and computers. If virtual systems are to be effective and well received by their users, considerable human-factors research needs to be accomplished. This paper provides an overview of many of these human-factors issues, including human performance efficiency in virtual worlds (which is likely influenced by task characteristics, user characteristics, human sensory and motor physiology, multimodal interaction, and the potential need for new design metaphors); health and safety issues (of which cybersickness and deleterious physiological aftereffects may pose the most concern); and the social impact of the technology. The challenges each of these factors present to the effective design of virtual environments and systematic approaches to the resolution of each of these issues are discussed.

510 citations


"A Comparison of Virtual and Physica..." refers background in this paper

  • ...Previous research has highlighted the effectiveness of immersive mixed reality training in different disciplines, including military training, medical training and vehicle driving simulators [17,21], as well as navigation and spatial knowledge training [8, 23], amongst others....

    [...]