Int J Adv Manuf Technol (2017) 89:1279–1293
DOI 10.1007/s00170-016-9164-5
ORIGINAL ARTICLE
Use of projector based augmented reality to improve
manual spot-welding precision and accuracy for automotive
manufacturing
Ashish Doshi
1
· Ross T. Smith
1
· Bruce H. Thomas
1
· Con Bouras
2
Received: 18 March 2016 / Accepted: 11 July 2016 / Published online: 25 July 2016
© The Author(s) 2016. This article is published with open access at Springerlink.com
Abstract This paper presents the use of a projector-based
spatial augmented reality system in an industrial quality
assurance setting to highlight spot-weld locations on vehi-
cle panels for manual welding operators. The aim of this
work is to improve the precision and accuracy of manual
spot-weld placements with the aid of visual cues as a proac-
tive step by the automotive manufacturer to enhance product
quality. The prototype system was deployed at General
Motors (GM) Holden plant in Elizabeth, Australia on the
production line building Holden Cruze vehicles. Production
trials were conducted and techniques developed to analyse
and validate the precision and accuracy of spot-welds both
with and without the visual cues. A reduction of 52 % of
the standard deviation of manual spot-weld placement was
observed when using augmented reality visual cues. The
average standard deviation with-AR assistance (19 panels
and 114 spot-welds) was calculated at 1.94 mm compared
to without-AR (45 panels and 270 spot-welds) at 4.08 mm.
All welds were within the required specification and panels
evaluated in this study were used as the final product made
available to consumers. The visual cues enabled operators
to spot-weld at a higher degree of precision and accuracy.
Grant ID: AutoCRC C1-53 AR Projection System for Work Cells 2
Ashish Doshi
ashishd@ieee.org
1
Wearable Computer Laboratory, University of South
Australia, Mawson Lakes Campus, Adelaide, Australia
2
GM Holden Vehicle Operations, Elizabeth, South Australia,
Australia
Keywords Automotive manufacturing · Spot-weld
validation · Spatial augmented reality · Visualisation
1 Introduction
General Motors (GM) Holden have a long history of pro-
ducing safe and reliable road vehicles.
1
As per their charter,
they are actively seeking to improve existing manufactur-
ing practices and vehicle designs using new technology to
meet the challenges of future automotive requirements. This
is an evolving process that the automotive industry as a
whole are involved in since the Model T assembly line
2
was
revolutionised in 1913 to gain a production advantage in
the automotive market. This work on integrating augmented
reality technology on the production line for manual spot-
welding was conducted in collaboration with GM Holden
Vehicle Operations at their facility in Elizabeth, Australia.
The vehicle body assembly plant in Elizabeth, Australia
opened in 1962 and has continuously undergone revisions
and modernisations to achieve production efficiency. Due
to the expense of robotic dexterity, not all of the assembly
processes have been automated. This meant spot-welding
and adhesive applications of small assembly panels are
performed manually. There are two types of testing con-
ducted on the production line. Destructive testing is per-
formed periodically by taking a completed vehicle body out
of production [
1]. Non-destructive testing of weld quality
assurance is performed more frequently by the use of a sin-
gle probe ultrasonic tester to measure the thickness of the
spot-weld [
2]. Visual inspection checks are undertaken to
ascertain whether spot-welds are in the appropriate location
1
http://www.holden.com.au/about/innovation/safety.
2
http://corporate.ford.com/company/history.html.
1280 Int J Adv Manuf Technol (2017) 89:1279–1293
and to ensure that the overall pattern conforms to the work
instruction [
3]. In the absence of suitable stencils or physical
guides, visual inspection is a viable method.
Existing manual production stations rely on job specific
instructions which operators commit to memory. Physi-
cal guides are placed on the welding part to help welders
locate a subset of the required welds (these are mainly for
stud-welds), but due to size restrictions there are no phys-
ical guides for every weld. Therefore, quality assurance
inspectors are limited to prioritising critical stud-welds, test-
ing of spot-weld thickness, and visual inspection of weld
placements.
The engineering requirements in a production envi-
ronment is such that the developed system has to work
alongside existing plant infrastructure, no interference with
production cycle (expensive if there was any), operators
should not wear or use additional wearable device without
specific approval and any prototype has to adhere to plant
safety regulations. Unlike the vision-based monitoring sys-
tem introduced by [
4] which was limited to detection and
evaluation of spot-welds, the goal of our work is to explore
and investigate the use of augmented reality in the produc-
tion environment. Our solution is to develop a projector
based Spatial Augmented Reality (SAR) system that can
be integrated at production stations as required and works
alongside existing plant infrastructure. The projection sys-
tem becomes an active part of the welding process, i.e.
specifying spot-weld placements. SAR superimposes com-
puter generated virtual imagery (e.g. virtual cues) directly
onto the physical objects’ surfaces. The virtual cues act as
visual aids to help the operators weld in the correct location
on the panel. The system is unobtrusive to the welders and
they do not require to hold or wear additional equipment or
personal protective equipment (PPE), which would interfere
with their spot-welding tasks.
The SAR system will not only enhance the operator’s
experience but also aid and improve the manufacturing pro-
cess as a whole, without hindering the user. Unlike [
5]
that proposed to setup a virtual welder trainer in a training
centre, the SAR visualisation system can also be used for
on-the-job training of operators with higher precision and
accuracy. This investigation details an industrial proof-of-
concept of the SAR projector system validated on an opera-
tional production line, which is safe and reliable within the
manufacturing environment [
6].
This paper is organised as follows. Section
2 describes
the prior work of using augmented reality (AR) and visu-
alisation methods in the manufacturing industry. Section
3
details existing processes within the plant whilst Section 4
describes the production line and challenges faced when
commissioned as the test bed for the SAR visualisation
system. The design of suitable virtual cues is presented in
Section
5. The SAR prototype system was developed and
installed on the production line at the GM Holden plant in
Elizabeth, Australia. An offline pilot study was conducted,
and this is presented in Section
6. Section 7 describes the
production trials conducted and presents the outcome of the
precision and accuracy tests. Discussion of the results is also
included in this section. This paper is concluded with our
summary and recommendations for further work in this area
in Section
8.
2 Prior work
A large portion of weld quality research has focussed to
date on weld gun technology, such as resistive [7, 8], gas
metal arc [
9]andlaser[10] for manual operators, automated
systems as used by manufacturing robots [
11]andweld
dynamics, such as spot-weld formations [
12] and weld heat
distribution [10, 13]. However, not much research have been
directed towards weld placements. Echtler et al. [
14]did
propose an augmented reality-based method to aid in weld
placements on the production line. This was subsequently
incorporated within the weld gun design.
Augmented reality (AR) is increasingly becoming a pop-
ular method for a user to interact with technology either
to obtain more information or to enhance their experience
for a specific task/application. This is especially true for
single user interactions in mobile augmented reality. Auto-
motive industries are slowly but surely adopting AR for
user interaction. They are using it for marketing products as
well as for Head-Up Displays (HUD) to project information
onto vehicle windscreen. BMW [
14] and Volkswagen [15]
have used AR on their production line, but AR is still an
uncommon practice among the wider automotive industry.
The automotive manufacturing industry started integrat-
ing new technologies such as AR in the late-1990s. For
example, Reiners et al. [
16] explored a Head Worn Display
(HWD)-based AR system on a doorlock assembly. The user
would wear the HWD with transparent glasses. The optical
tracker would track the head pose, the camera would track
the pose of the door and identify AR markers. The virtual
object of the doorlock is thus rendered onto the eyewear
with audio instructions on the assembly task. Though the
system received positive feedback of its potential, one of its
major drawbacks was the optical tracker used caused lag in
the tracking phase. This was a major issue as they could not
sufficiently compensate for head movements reliably.
Although using HWDs have been a common approach
to deploying AR technology for manufacturing, findings
reported in [
17] show that virtual-to-real space percep-
tion can lead to significant assembly inaccuracies. Hence,
the requirement of better plant integration systems. The
ARVIKA [
18] project emphasised a similar conclusion.
Over a 4-year period, the project was aimed at fostering
Int J Adv Manuf Technol (2017) 89:1279–1293 1281
research and development partnerships between automo-
tive industries and academics in Germany on AR. They
observed that though AR research had progressed signifi-
cantly in an academic environment, the biggest drawback
to AR integration in manufacturing was operator usabil-
ity (for instance interactions and workflow interruptions),
safety regulations, and environmental conditions in the
plant. Therefore, for a widespread acceptance of AR on the
production line, these issues have to be addressed, which
is the foundation of our work. This conclusion was further
supported by [
19] through various AR projects at Daimler
Chrysler (in Germany), ranging from servicing and main-
tenance of vehicles through to prototype design and plant
organisation.
Reference [
20] along with [21] proposed a desktop-
based AR system for manual assembly and planning pro-
cess. Their systems reposition virtual models over existing
plant layout to facilitate easier plant process planning. This
helped engineers to visualise production layout changes and
optimise the workflow based on production requirements.
A similar system was developed in [
22] for use on mobile
devices.
AIRBUS Military [23] trialled mobile AR work instruc-
tions for electrical harness routing on the Airbus Mili-
tary aircraft frame. Though there was limited testing, the
reported user satisfaction was promising and showed that
using AR as a virtual aid can significantly improve perfor-
mance.
Zhou et al. [
24] investigated the feasibility of using SAR
with laser projectors to display spot weld positions and its
information onto a vehicle panel. Though this was success-
fully tested in a laboratory environment, the system could
not be deployed on the production line due to safety con-
cerns [
25]. In an open and multi-user production station,
the risk associated with the hazard of working with reflec-
tive aluminium panels is quite high, hence, the use of such
laser projectors was not advisable. Their proposal however
was only operative for a single work station and has limited
visualisation field of view due to plant physical constraints.
Manufacturers such as BMW [14] and Volkswagen [15]
have trialled an in-situ welding facilitator using laser pro-
jection. Their system is based on AR markers [26], where
markers are placed at specific positions on the assembly line
to help with position and orientation identification thereby
offsetting it to place the virtual graphic in the correct loca-
tion. Two ways have been proposed to do this. One is with
a specialised welding gun [
14] and the other is with laser
projectors [
24, 27]. In both the cases, the AR interaction is
limited to single user, is quite expensive and can pose as a
safety hazard in the plant. In contrast, our system addresses
the current assembly AR limitations, with multi-user inter-
action, full projective coverage of the object/panel surface,
increase of manual production efficiency (accuracy and
precision) and to minimize unnecessary burden on operators
when integrating new AR technology [
28].
3 The production line
This section describes the existing process in the GM
Holden plant in Elizabeth, South Australia, Australia and
motivation behind the development of the SAR prototype
for the production line. The production station that was
chosen is one among many manual work stations of the
GM Holden Cruze production pipeline. The chosen sta-
tion was identified as having a large work-area for ease of
SAR prototype development and consists of a vehicle panel
sitting firmly on a static fixture. This provided the ideal sur-
face area to display the AR visualisations. Furthermore, the
vehicle panel’s positioning could be replicated within the
visualisation laboratory for testing purposes.
3.1 Existing physical environment
There are two variants of the GM Holden Cruze that spot-
welding are performed at this station. The majority of
welding locations for these two variants are identical, but
there are a number of spot-welds locations that differ. The
surface area of the vehicle panel is approximately 1.5m
(width) and 1.5m (length) and is placed on a static fixture at
1m above ground. Figure
1 provides a depiction of the space
and physical constraints around the fixture.
Fig. 1 Manual welding work station at GM Holden Vehicle Opera-
tions, Elizabeth, Australia
1282 Int J Adv Manuf Technol (2017) 89:1279–1293
3.2 Workflow
Two operators are assigned to the station where the welding
is carried out. Between them, they weld 62 spot-welds and
six stud-welds within the allocated 180 s. The time includes
hoisting the panel onto the fixture, securing it, welding, ver-
ification of the welds, unfasten the clamps and hoisting the
work piece onto a rotatable fixture for moving the panel
into the robotic welding station. As per our observations, the
workflow for this production station is as follows:
1. If fixture is clear, hoist new panel onto fixture.
2. Press palm button to clamp the fixture.
3. With reference to Fig.
2b, weld guns on the left and
front (bottom of layout) are taken.
4. One operator spot-welds on the left-side of the panel
and the other on the front.
5. Weld guns are returned to position.
6. With reference to Fig. 2b, weld guns on the right and
front right (bottom of layout) are taken.
7. Whilst one operator is handling the stud-weld gun,
the other puts the hand-held fixtures for stud-weld
positioning in place.
8. One operator spot-welds on the right-side of the panel
and the other places stud-welds on the front.
9. Weld guns are returned to position and stencil is
removed.
Fig. 2 Proposed schematic with a side view and b top view
10. The panel is unclamped from the fixture.
11. Hoist completed panel onto rotating fixture for entry
into automated welding station.
12. Repeat process.
The observed workflow has been optimised for the oper-
ators’ time to handle weld guns, perform welding, and
quick verification of the weld positions. To manage the
quality assurances within the production cycle, inspectors
perform ultrasonic thickness test of the spot-welds whilst
the operators are welding at other spot-weld positions.
3.3 Human requirements
During hoisting of the panel between fixtures, there is a
large amount of movement around all the stations with
different panels being hoisted into positions at other sta-
tions as well, hence, an important issue is to maintain an
unobstructed floor area for operators’ safety. Although a
wearable device such as used in [
27] would have a number
of advantages, a wearable system was impractical for use
on this production line (especially with large surface area of
the panel) and could lead to slower production turnover.
4 SAR enhancements
Due to the layout of the station and the close proximity
to other stations, the placement of projectors could not be
mounted from the floor, as this would disrupt the natu-
ral foot flow of the operators. Hence, an I-beam mounted
solution was considered. The schematic for the layout is
presented in Fig.
2.
4.1 Physical layout
With reference to Fig.
2, our solution incorporates two pro-
jectors in the system. With two operators at this station, two
projectors are required to cover for occlusions during weld-
ing. A single projector is not adequate to handle occlusions.
As can be observed in Fig. 1, the location and position-
ing of the weld gun could potentially cause occlusions. For
example, whilst one is being used on the back of the panel
(flat surface area), the weld gun can occlude the projected
light at the front (outward vertical bend). Hence, some spot-
weld positions might be visible to the second operator and
some might not, thus interfering with overall workflow.
Other occlusion examples are clamping fixtures and human
interference. With two projectors, there is at least 80 % over-
lapping projector field of view coverage between the two
projectors. The front end of the panel is curved in and is
therefore hidden from one projector. This is compensated
for by the second projector, with all of the visible surfaces
Int J Adv Manuf Technol (2017) 89:1279–1293 1283
for the panel within the fields of view of the projectors.
A third projector directly orthogonal and above the panel
would have been preferable, but this was not possible due to
overhead monorail for hoisting panels. Although additional
projectors were considered, this was not followed through
due to the physical constraints of this station layout.
Fig. 3 SAR network diagram