Human-Robot Collaboration: Task Sharing Through Virtual Reality

TLDR: This work proposes to extend collaborative robot solutions with including Virtual Realty as a sensor and to provide comfort features to the operator and to create cooperation between human and industrial robot in experiments.

Abstract: Collaborative Robots provide many possibilities, when it comes to Human-Robot Collaboration. Until now, these approaches are usually custom made, sensor-integrated solutions, where the robot's safety controller ensures the safety of the human worker. These solutions are according to today's rules and standards. We propose to extend these solutions with including Virtual Realty as a sensor and to provide comfort features to the operator. In order to create cooperation between human and industrial robot in our experiments, we propose to have a simple nut screwing operation as an example, where the industrial robot does the hard part. With sharing the task in such manner, we will ensure that the robot is doing the hard and monotonous work, while the worker benefits from the task sharing. Results are demonstrated through simulation and in reality also.

Figures

Fig. 5. State flow.

Fig. 1. An example of a dynamic safety system which uses multiple sensors and one potential lidar sensor [4] which is already in use in GIM Ltd robotic solutions [6].

Fig. 2. Human-robot collaboration is changing and there are many standards which are related to it [7].

Fig. 4. Four levels of HRI [10].

Fig. 3. Collaborative and manual modes applied in collaborative workspace [9].

TABLE 1. COMPARISON BETWEEN VR AND NON VR SCENARIOS

Full text

Human-Robot Collaboration: Task sharing through
Virtual Reality
1
st
Beibei Shu
Department of Industrial Engineering
UiT The Arctic University of Norway
Narvik, Norway
beibei.shu@uit.no
2
nd
Gabor Sziebig
Department of Industrial Engineering
UiT The Arctic University of Norway
Narvik, Norway
gabor.sziebig@uit.no
3
rd
Sakari Pieskä
Centria University of Applied Sciences
Ylivieska, Finland
Sakari.Pieska@centria.fi
Abstract — Collaborative Robots provide many possibilities,
when it comes to Human-Robot Collaboration. Until now, these
approaches are usually custom made, sensor-integrated solutions,
where the robot’s safety controller ensures the safety of the human
worker. These solutions are according to today’s rules and
standards. We propose to extend these solutions with including
Virtual Realty as a sensor and to provide comfort features to the
operator. In order to create cooperation between human and
industrial robot in our experiments, we propose to have a simple
nut screwing operation as an example, where the industrial robot
does the hard part. With sharing the task in such manner, we will
ensure that the robot is doing the hard and monotonous work,
while the worker benefits from the task sharing. Results are
demonstrated through simulation and in reality also.
Keywords—Human-Robot Collaboration, industrial robot,
simulation, virtual reality
I. INTRODUCTION
The paradigm related to Human-Robot Collaboration (HRC)
is changing from the separate human from robot (past) to
improved human access to robot (present) and close human-
robot interaction (future). One major problem for the
introduction of robots especially in unstructured environment is
the possibility to rely on dependable sensors. Sensor data are
needed for reactive planning, motion/force control, visual
servoing, fault diagnosis, and monitoring of safety levels. If the
HRC system is planned for unstructured environments with
unpredictable movements of persons, HRC should be equipped
with a versatile sensor system, including: range, proximity,
touch, vision, sound, temperature, and so on. The selection, the
arrangement, the number of sensors and their reliability
contribute to the measure of dependability of a manipulator for
interaction tasks [1].
There are lot of sensors available for HRC systems. Close
Human-Robot Collaboration with advanced safety sensors may
support speed & separation monitoring and safety rated
monitored stop modes. These might include close proximity
sensors, such as pan/tilt/zoom cameras, stereo cameras, depth
cameras, projection based-systems [2], and audio/video
feedback systems. A better fit for traditional robots (large, high
speed, high payload) can be achieved with compliment power &
force limiting functions (PFL robots) [3]. Sensors for distance
interaction include pan/tilt/zoom cameras, stereo cameras,
projection based-systems, 3D Lidar [4], audio/video feedback
systems, certified safety sensors. HRC sensors can also include
force/torque sensors or proximity sensors to be used integrated
to grippers. Fig. 1 shows an example configuration of the
dynamic safety system with sensors [4] together with an
example of advanced lidar sensors.
Fig. 1. An example of a dynamic safety system which uses multiple sensors
and one potential lidar sensor [4] which is already in use in GIM Ltd
robotic solutions [6].
However, the sensors and other devices may vary depending
on the complexity of the needed safety system. There are also
many standards which have to take into account when choosing
sensors, like Omron STI presentation shows in Fig. 2.
Fig. 2. Human-robot collaboration is changing and there are many standards
which are related to it [7].
Fusion of the information coming from multiple sensors may
help in providing a coherent and reliable description of the world
surrounding the robot. In general, it is required to integrate
sensor information based only on approximate models of the
environment. Data fusion is particularly important when
monitoring contacts, e.g. for selecting impedance parameters or
for determining the most dangerous “control points” on the robot
to be driven away from a human with higher priority [8].
Unfortunately, there has been little work on achieving the fusion
of contact and visual information.
Collaborative robot is defined in standard ISO 10218-2 as
follows: Robot designed for direct interaction with a human

within a defined collaborative workspace i.e. workspace within
the safeguarded space where the robot and a human can perform
tasks simultaneously during production operation. Basically, the
idea is that robot does not hurt a person and the means to protect
a person are controlled force and speed, separation monitoring,
hand-guiding and safety-rated monitored stop. Fig. 3 shows the
means that can be applied in manual or collaborative operation.
Person enters
collaborative workspace
Manual
high
speed
Manual
reduced
speed
Hand guiding Safety-rated
monitored stop
Control of
speed and
separation
Control of
power and force
Enabling device
(or teach pendant)
For program
verification only,
mode selection,
high speed button,
hold-to-run device
Reduced speed
max. 250 mm/s
Safety distance
according to
EN 13855
(detection devices)
and
EN 13857 (guards)
Max. static and
dynamic force
– see
ISO/TS 15066
Fig. 3. Collaborative and manual modes applied in collaborative workspace
[9].
Different Human-Robot Interaction (HRI) will happen when
human collaborate with robot. A new classification strategy has
been proposed depending on the level of HRI. According to this
approach, most of the possible HRI in industry could be
classified into four levels of interaction [10], see in Fig. 4.
Fig. 4. Four levels of HRI [10].
1) Shared workspace without shared task: The robot and
the human acting in a shared fenceless workplace but working
on their own task each other.
2) Shared workspace, shared task without physical
interaction: The robot and the human have a shared task, but no
direct interaction. The robot can only move to a predefined
position near human for assisting.
3) Shared workspace, shared task ‘‘handing-over’’: The
shared task consists of a direct handing-over between robot and
human, but no physical interaction.
4) Shared workspace, shared task with physical
interaction: The robot and the human working in a task which
physical interaction is necessary.
II. LITERATURE REVIEW ON HRC WITH VIRTUAL REALITY
Nowadays, with the development of industrial 4.0
technologies, more and more network cameras and sensors have
been adopted during human-robot collaboration [11].
Researchers also developed various way to use these cameras
and sensors. Such as in [12], the author showed a method which
using augmented reality (AR) technologies in a mobile platform
to control a real robot. A tablet camera offers a real-time video
stream to a server through wireless communication, and the
tablet feedback 3D graphics with touchscreen interaction to
users. It allows user to effectively communicate with robot.
Due to various reasons, such as human safety, space
limitation or price, people cannot directly conduct experiment
on real robot. Then, more researchers use the cameras and
sensors combining with Virtual Reality (VR) technology to
conduct experiments in simulation. In [13], the author
introduced a simulated hand guiding robot system with a using
of force feedback device. In the VR simulation, it allows user to
move the robot (which with a screw driver mounted) simply and
intuitively on the target place (on the screw) and let the robot out
put the correct torque. In [14], the paper introduced a simulated
robot controller using Unity built-in kinematics, achieved a real-
time controlling of a specific type of ABB robot. The author
using Robot Operating System (ROS) as middleware driving the
robot in VR world, and feedback to user by HTC Vive. In [15],
the paper also introduced a simulated robot controller, but
implement a special designed forward and inverse kinematics
algorithms in MATLAB. The robot controller can manipulate
specific type of KUKA robot in real-time under virtual reality
environment. And the author also explained a possibility to
connect the VR model with real robot. In [16], the paper
introduced a prototype which using proactive and adaptive
techniques to avoid possible collision between robot and human.
After setting up Microsoft Kinect (as an input for skeletal
tracking of the user) and putting on Oculus Rift DK2 (as an
output device for stereoscopic visual display, and as an input
device for head motion tracking), the user can interact with the
prototype in virtual reality environment. In [17], the author using
Oculus Rift DK2 as visual display, and a tracking system that
tracks the user’s head position, pose, and eye-gaze, achieved
controlling a real Baxter robot to pick up a part. And a simulated
robot moving synchronously.
With the help of VR simulation, researchers really make
human-robot collaboration more flexible and intuitive, and more
experiments can be conducted to test different new ideas.
III. COLLABORATION IN VIRTUAL REALITY
With the widespread of automation and industrial robotics,
there are new tasks, which human and an industrial robot could
solve together. One of the tasks, which human solves effectively
is placing a nut on a screw. This is however challenging in cases,
where the screw and the nut is heavy and a worker needs to do
this repetitive task all day long. Our solution provides
ergonomic solution for this problem, as it provides a high level
HRI to relieve human from the tedious work. It is very easy to
construct an HRC scenario that the robot picks up the screw and
put it at a predefined position, then the human put a nut on screw,
rotate the screw. But this is only level 2 in the HRI, we could

make the robot even more helpful with the HRI is level 4. Then
the robot interacts with human directly, helping human put the
nut on the screw and rotate the screw. All the human worker
need to do is just move a nut near to robot (no matter which
direction) and trigger the robot, then let the robot finishing the
rest of the task.
IDLE
Find nut
Not find nut
Start s crewing
Approach
Stopped
Start rotatin g
Slower
Faster
Slower
Faster
Cannot start rotating
STOP
Cannot reach
Fig. 5. State flow.
The process of the whole HRC can be described as different
processes. Human approach to the robot working area triggering
robot from idle state to active. During human-robot interaction,
if any error in the process, robot will pop-out error message to a
screen and communicate with human. The human interaction
process contains:
1) Screw picked: Robot picks up the screw, hold it by
gripper. If system cannot find the screw, output an error.
2) Pointed at center of nut: System find out the coordinate
of nut, then robot move the screw to nut position with pointing
at the center of nut. If robot cannot reach to the target place,
output an error.
3) Screwing: Start the physical interaction.
4) Finished: The screwing task finished, back to idle state
or stop state.
Human operator can trigger the robot processes by different
voice instructions. During the screwing process, there are two
stages, before the screw/nut locked and after screw/nut locked.
If human moved the nut position beyond a certain tolerance
before screw/nut locked, or any other reason causing robot
cannot start rotating, an error will be output. After the screw/nut
locked, human can change the robot rotating speed from a high
speed to stop rotating gradually by voice instructions, also
human can stop the whole screwing process and go to next
process. See in Fig. 5.
If it is achieved, the difficulty of teaching a robot will
dramatically decrease, since there is no specific pre-
programming point needed for a robot. And the human worker
doesn’t need to move nut into specific point to cooperate with
robot either. Just enabling the robot, then the robot will
collaborate with user automatically. Even no robotic
background user can work with robot.
But it is difficult to test the task on real robot directly. Since
this task is categorized as level 4 in interaction-levels between
human and industrial robot, a lot of safety procedure need to be
setup to ensure human worker will not injured before the test,
such as monitoring and controlling robot position, speed,
torques, and near-field vision system for human hand, body and
face [8]. Besides the human safety reason, there are many other
advantages of using VR, see the comparison in TABLE 1. In
order to test and verify the new idea faster, run a simulation test
in Virtual Reality environment will be the best choice.
TABLE 1. COMPARISON BETWEEN VR AND NON VR SCENARIOS
Using VR
Not using VR
Locating target
position
Simulation software
acquire nut position in
real time
Special 3D position
detection device must be
installed, and feedback
the coordinate to robot
system
Recognizing
target
No special recognition
system needed
System need to setup a
digital database in
advance for recognizing
nut, screw, ball and can
Human safety
No special protection
for human
System should trace
human skeleton, and
feedback the 3D
position data to robot
system to avoid hurting
human
Testing area
No special testing area
needed
A special testing area
with barrier is needed
Test calibration
One button to restore
system starting point
Careful measurement is
needed to keep
experiment in the same
starting point
IV. EXPERIMENTING WITH COLLABORATION
According to the proposal, we designed the experiment. The
experiment is running under a simulation environment which the
simulation software is Visual Components. Visual Components
is running in Windows platform which is much user friendly and
avoid a lot of coding process comparing with Linux platform. In
its own library, Visual Components already embedded abundant
robot models, such as ABB, KUKA, FANUC, NACHI, etc. And

the software offered a Python API to user, so we have the
possibility to modify almost every feature of the software.
In Fig. 6, a NACHI MZ07 robot is putted on a work platform,
with a simple gripper mounted on. Since the goal is just put a
nut on screw, where to achieve the goal is unknown before the
system is started. Human worker can move in the nut from any
direction, we define the robot motion not based from any
specific point. So, in the initial state, a nut with a random
coordinate on a human worker’s hand. And a screw with a
random coordinate in a box. Inside the box, there are also a can
and a ball. A camera mounted on a pillar to capture and
recognize different objects with their coordinates. After the
human worker triggered the system by press the switch under
the foot (simulation started), the background Python script can
access to the objects’ coordinates and generate a series of
predefined robot point-to-point (PTP) or linear (LIN) motion
statements.
Fig. 6. Pick the screw, put at the center of nut.
Fig. 7. The robot Tool Centre Point (TCP) represented by each statement
respectively
In Fig 7, we can see the corresponding TCP positions
represented by each robot motion statement. The goal which
putting a nut on screw should also contain a basic method to
teach robot finishing the task. The robot motion statement is
generated by this method. In this method, after the system
acquired the information of nut and screw position (can be input
from camera system or simulation initial state), the TCP position
in each motion statement will be re-calculated and updated.
With this method, our goal can be achieved, and a level 4
interaction could be conducted between human and robot
through the VR environment.
In Fig. 8, we can see a sample execution of the previously
described interaction. The sequence of the movement are
determined by the state flow presented in Fig. 5, but the actual
position coordinates are calculated real-time in the simulation
environment. The data necessary to decide about positions are
gathered from the virtual reality and the reality (physical robot).
The connection between these two are based on a networked
solution previously developed in-house, which can control the
physical robot from an external device on high frequency (83
Hz) [18].
Fig. 8. Movement of the robot
The human can follow the state of the interaction through a
Graphical User Interface (GUI). This GUI describes the current
interaction, information on the next steps and if there any errors,
that needs attention from the human operator. Not only the GUI,
but other sensors are also used in order to detect the human’s
intentions. Two depth cameras are installed to detect human and
nut in real-time. Microsoft Kinects are chosen as these have the
necessary speed and are well known and used in human-
machine interaction scenarios. One is used to detect human
skeleton pose and another one is focused on human hand and nut
for higher accuracy. After the nut and human positions are
detected, the information is updated in the simulation software
and on the physical robot also.
A video recording of the interaction can be viewed at:
http://vizlab.uit.no/iecon2018/

Fig. 9. The proposed HRC in real world.
In Fig. 9, we can see the proposed HRC in real world. A human
holding a nut, waiting the robot to pick the screw from the box
and working together. With the successful of preliminary idea
test in VR environment, we can conduct this real test in next
stage.
V. CONCLUSION
Human-Robot Collaboration is not any more limited to sharing
the space with an industrial robot. In our experiments we show
how given simple task could be shared between the human
operator and the industrial robot. The experiment shows the
feasibility of the approach and the Virtual Reality system helps
the operator to achieve the necessary comfort functions, which
is needed for a level 4 Human-Robot Collaboration.
VI. FUTURE WORK
To achieve this robot movement, future work should be
focused on four main tasks:
• A camera/sensor system which should not only can
recognize the nut and screw but also can calculate their
coordinates in 3D space.
• A server can calculate the robot joint angle based from
the 3D coordinates.
• The robot can move to the desired position by receiving
the joint angle value from server.
• Human safety procedures.
At last, a synchronization between the real world and the
virtual world should be setup.
REFERENCES
[1] R. Alami, A. Albu-Schaeffer, A. Bicchi, R. Bischoff, R. Chatila, A. De
Luca, et al. “Safe and dependable physical human-robot interaction in
anthropic domains: State of the art and challenges,” in IEEE International
Conference on Intelligent Robots and Systems. 2006.
[2] C. Vogel, M. Poggendorf, C. Walter and N. Elkmann. “Towards safe
physical human-robot collaboration: A projection-based safety system,”
in IEEE International Conference on Intelligent Robots and Systems.
2011.
[3] N. Ando, P. T. Szemes, P. Korondi and H. Hashimoto, "Friction
compensation for 6DOF Cartesian coordinate haptic interface," IEEE/RSJ
International Conference on Intelligent Robots and Systems, Lausanne,
Switzerland, 2002, pp. 2893-2898 vol.3.
[4] 3D Lidar http://www.velodynelidar.com/.
[5] T. Salmi, O. Väätäinen, T. Malm, J. Montonen and I. Marstio, “Meeting
New Challenges and Possibilities with Modern Robot Safety
Technologies,” in Enabling Manufacturing Competitiveness and
Economic Sustainability. 2014. pp. 183-188.
[6] GIM Ltd robotic solutions http://gimltd.fi/services.html.
[7] T. Knauer, “Safety Sensor Technologies Enabling Closer Human-Robot
Collaboration,” in AUTOMATE 2017, 2017.
[8] P. Korondi, D. Young and H. Hashimoto, "Sliding mode based
disturbance observer for motion control," Proceedings of the 37th IEEE
Conference on Decision and Control (Cat. No.98CH36171), Tampa, FL,
USA, 1998, pp. 1926-1927 vol.2.
[9] T. Malm, T. Salmi, I. Marstio and J. Montonen, “Safe collaboration of
operators and industrial robots,” in Automaatiopäivät22 2017 Seminar,
2017.
[10] M. Bdiwi, M. Pfeifer, and A. Sterzing, “A new strategy for ensuring
human safety during various levels of interaction with industrial robots,”
CIRP Annals, 2017, vol. 66(1), pp. 453-456.
[11] Dániel, B., Korondi, P., Sziebig, G. and Thomessen, T., "Evaluation of
flexible graphical user interface for intuitive human robot interactions,"
Acta Polytechnica Hungarica, 11(1), 2014, pp.135-151.
[12] J.A. Frank, M. Moorhead, and V. Kapila. “Realizing mixed-reality
environments with tablets for intuitive human-robot collaboration for
object manipulation tasks,” in 2016 25th IEEE International Symposium
on Robot and Human Interactive Communication (RO-MAN). 2016.
[13] U. Dombrowski, T. Stefanak, and J. Perret, “Interactive Simulation of
Human-robot Collaboration Using a Force Feedback Device,” Procedia
Manufacturing, 2017, vol. 11, pp. 124–131.
[14] A. de Giorgio, M. Romero, M. Onori, and L. Wang, “Human-machine
Collaboration in Virtual Reality for Adaptive Production Engineering,”
Procedia Manufacturing, 2017, vol. 11, pp. 1279-1287.
[15] L. Gammieri, M. Schumann, L. Pelliccia, G. Di Gironimo, and P.
Klimant, “Coupling of a Redundant Manipulator with a Virtual Reality
Environment to Enhance Human-robot Cooperation,” Procedia CIRP,
2017, vol. 62, pp. 618-623.
[16] E. Matsas, G.-C. Vosniakos, and D. Batras, “Prototyping proactive and
adaptive techniques for human-robot collaboration in manufacturing
using virtual reality,” Robotics and Computer-Integrated Manufacturing,
2018, vol. 50, pp. 168-180.
[17] M.M. Moniri, F.A.E. Valcarcel, D. Merkel, and D. Sonntag, “Human
Gaze and Focus-of-Attention in Dual Reality Human-Robot
Collaboration,” in 2016 12th International Conference on Intelligent
Environments (IE), 2016.
[18] P. Korondi and J. Gyeviki, "Robust Position Control for a Pneumatic
Cylinder," 2006 12th International Power Electronics and Motion Control
Conference, Portoroz, 2006, pp. 513-518.

Frequently asked questions

Q1. What have the authors contributed in "Human-robot collaboration: task sharing through virtual reality" ?

The authors propose to extend these solutions with including Virtual Realty as a sensor and to provide comfort features to the operator. In order to create cooperation between human and industrial robot in their experiments, the authors propose to have a simple nut screwing operation as an example, where the industrial robot does the hard part.