King’s Research Portal
DOI:
10.1109/COMST.2018.2851452
Document Version
Peer reviewed version
Link to publication record in King's Research Portal
Citation for published version (APA):
Antonakoglou, K., Xu, X., Steinbach, E., Mahmoodi, T., & Dohler, M. (2018). Towards Haptic Communications
over the 5G Tactile Internet. Ieee Communications Surveys And Tutorials, 1-27.
https://doi.org/10.1109/COMST.2018.2851452
Citing this paper
Please note that where the full-text provided on King's Research Portal is the Author Accepted Manuscript or Post-Print version this may
differ from the final Published version. If citing, it is advised that you check and use the publisher's definitive version for pagination,
volume/issue, and date of publication details. And where the final published version is provided on the Research Portal, if citing you are
again advised to check the publisher's website for any subsequent corrections.
General rights
Copyright and moral rights for the publications made accessible in the Research Portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights.
•Users may download and print one copy of any publication from the Research Portal for the purpose of private study or research.
•You may not further distribute the material or use it for any profit-making activity or commercial gain
•You may freely distribute the URL identifying the publication in the Research Portal
Take down policy
If you believe that this document breaches copyright please contact librarypure@kcl.ac.uk providing details, and we will remove access to
the work immediately and investigate your claim.
Download date: 09. Aug. 2022
1
Towards Haptic Communications over the 5G Tactile Internet
Konstantinos Antonakoglou
1
, Xiao Xu
2
, Eckehard Steinbach
2
, Toktam Mahmoodi
1
, Mischa Dohler
1
1
Centre for Telecommunications Research, King’s College London
2
Chair of Media Technology, Technical University of Munich
Touch is currently seen as the modality that will complement audition and vision as a third media stream over the Internet in a
variety of future haptic applications which will allow full immersion and that will, in many ways, impact society. Nevertheless, the
high requirements of these applications demand networks which allow ultra-reliable and low-latency communication (URLLC) for
the challenging task of applying the required Quality of Service (QoS) for maintaining the user’s Quality of Experience (QoE) at
optimum levels. In this survey, we enlist, discuss and evaluate methodologies and technologies of the necessary infrastructure for
haptic communication. Furthermore, we focus on how the fifth generation (5G) of mobile networks will allow haptic applications
to take life, in combination with the haptic data communication protocols, bilateral teleoperation control schemes and haptic data
processing needed. Finally, we state the lessons learned throughout the surveyed research material along with the future challenges
and infer our conclusions.
Index Terms—Tactile Internet, 5G, haptic communication, bilateral teleoperation, haptic data reduction, multi-modal media
networks
I. INTRODUCTION
G
REAT part of ongoing research on the fifth generation
of mobile networks (5G) is focused on meeting the
requirements of the Tactile Internet [1]–[3]. A major design
challenge here is to provide ultra-low delay communication
over the network which would enable real-time interactions
across wireless networks. This, in turn, will empower people
to wirelessly control both real and virtual objects. It will
undoubtedly add a new dimension to human-machine inter-
action and lead to an unprecedented revolution in almost
every segment of society with applications and use cases
like mobile augmented video content, road traffic/autonomous
driving, healthcare, smart grid, remote education, and remote
immersion/interaction among others [1].
One specific application domain of the Tactile Internet is
teleoperation which allows for remote immersion, including
remote touch. Traditional remote interaction solutions such
as voice or video conferencing, remote teaching, etc., have
reached a high level of sophistication and widespread use
thanks to the growth and progress of audio-visual commu-
nications.
With the benefits of this technology, users experience an
improved virtual presence, immersing in a remote environ-
ment. With current advances in communication infrastructure,
it has been foreseen that in the near future, a complete remote
immersion can be realized with the ability of physical inter-
action with the remote environment. This is achieved by the
exchange of multi-modal information, such as the combination
of audio, video and haptic information, over the Internet.
Such immersion will be feasible for commercially acceptable
use, with real-time applications such as teleoperation with
haptic feedback (referred to as teleoperation) or haptic data
broadcasting in virtual environments [4].
Haptics refer to both kinesthetic perception (information of
forces, torques, position, velocity, etc. sensed by the muscles,
joints, and tendons of the body) and tactile perception (in-
formation of surface texture, friction, etc. sensed by different
types of mechanoreceptors in the skin) [5]. It must be noted
that the previously mentioned term ”tactile” refers to its literal
meaning, i.e. the human perception of touch. When used in
the term ”Tactile Internet”, it signifies the feature of ultra-low
delay communication over the Internet which is a necessity
for many 5G use cases including haptic communication. As
one of the applications of the Tactile Internet, haptic commu-
nication using networked teleoperation systems has specific
requirements, the most demanding being the efficient and
timely exchange of kinesthetic or tactile information while
synchronously providing the user with auditory and visual
information.
Different from the communication of audio and video
signals, haptic signals in bilateral teleoperation systems are
bidirectionally exchanged over the network. It involves human
users and closes a global control loop between the human
users and the actuators/teleoperators. Thus, system stability
and teleoperation quality are very sensitive to communication
delay [6].
Use cases of the Tactile Internet, which highlight its impor-
tance, can be found in the medical, industrial, education and
entertainment sectors. These include remote medical exami-
nation or surgery, industrial teleoperation in e.g. construction
sites, mines or factories, tele-mentoring and gaming to name a
few. The benefits of the realization of the Tactile Internet will
revolutionize our way of living and increase the safety and
efficiency of various tasks. Nonetheless, there are hindrances
to be overcome and the previously mentioned requirements to
be met.
Concepts and technologies around the Internet of Things
(IoT), 5G and the Tactile internet overlap each other, as
indicated in [7], requiring very low latency and high reliabil-
ity communication channels, high-bandwidth low-latency and
secure infrastructure as well as bringing the intelligence of the
network closer to the edge of the network.
As described in [8], one of the challenges in 5G mobile
2
networks development is the provision of low-latency com-
munications with acceptable Quality of Experince (QoE) for
the users. Since evaluating QoE in haptic-based applications
with force feedback over the Internet is a process that has only
recently taken its first steps, the way to resolve this open issue
is still under investigation.
The delay requirements of haptic communication for net-
worked teleoperation systems are heavily dependent on the
application scenarios. Taking into account the latest achieve-
ments on haptic communication, as illustrated in Figure 1, the
less dynamic the remote environment, the more the interaction
between a user and the remote environment is increased. Con-
sequently, different application scenarios arise in accordance
with each level of dynamics and the corresponding range
of time delay that is considered as acceptable for feasible
interaction.
Applications which can tolerate delays over 1ms are within
the scope of teleoperation (the blue circle of Figure 1); a
broad range of applications that can be divided into three
categories of teleoperation, wherein each scenario is associated
to a level of dynamics of the remote environment the user is
interacting with and the corresponding delay tolerance. This
leads to teleoperation applications with different degrees of
immersive perception that range from space teleoperation to
remote steering of automobiles, demonstrating different levels
of abstraction between the user and the remote environment.
The case of highly dynamic environments, where a latency
of under 1ms is needed, is out of the scope of teleoperation
as only control systems can undertake the completion of
tasks with such latency requirements because humans are
underqualified for this kind of interaction. Specifically, for
completing such tasks high Quality of Control (QoC) is
needed. Examples would be a magnetic levitation system that
keeps a running train floating in midair, a fully automatic
driving system that precisely platoons vehicles and zips the
vehicles through intersections without traffic lights, or a real-
time simultaneous localization and mapping (SLAM) with
autonomous-controlled cameras [9]. As a result, these cases
will not be examined in this survey.
On the other hand, in this survey we focus on the efforts for
haptic communication in networked teleoperation systems over
the Tactile Internet and examine in detail the advancements
in teleoperation over long distances. Three main domains
for enabling teleoperation over global connectivity are stud-
ied here, including: (i) the communication network from
the perspective of providing reliable (guaranteed) low-latency
communications, (ii) intelligent data processing to compensate
for the communication latency and for reducing bandwidth
usage, and finally (iii) stability control schemes implemented
at the teleoperation devices to reduce the impact of potential
latency. The focus of this study is on remote environments of
low and intermediate dynamics (red text in Figure 1). Within
this range of dynamics there is a variety of applications such
as remote surgery (low dynamics) or collaboration of users in
virtual or real environments (intermediate dynamics).
The structure of this survey is as follows: In Section
II we describe teleoperation systems in detail, classify and
describe the challenges behind bilateral teleoperation systems
and elaborate on a number of commercially popular haptic
devices. Section III is concerned with how network-based
teleoperation systems communicate over the Internet, data
stream management for the audio, video and haptic data
streams, with special interest to network protocols of the trans-
port and application layer. It also includes common network
performance parameters and elaborates on provisioning of
QoS in the network. Additionally, we briefly discuss network
security. Next, Section IV covers a range of methodologies
and frameworks for reducing the quantity of haptic data to
be transmitted through a communication channel. In Section
V we refer to the main and also most recent teleoperation
bilateral control approaches, mainly focusing on passivity-
based approaches but also mentioning other approaches not
based on passivity. The approaches we will focus on can be
combined with haptic data reduction methods to provide high
QoE to the user. Moreover, Section VI presents the latest
developments on 5G mobile infrastructure and technologies
with focus on ultra-reliable low-latency communication as
well as the main KPIs of various 5G use cases. Section VII
discusses the lessons learned from this survey and outlines
future research directions as well as the current challenges of
haptic communication over the 5G networking infrastructure.
Finally, in Section VIII we infer the conclusions.
Environment
dynamics (level)
10
0
10
1
10
2
10
5
Human in the loop
(QoE requirements)
Delay tolerance (ms)
Teleoperation
Stability & transparency
Teleoperation accross planets,
supervised control
QoC requirements
Very
Low
Low
Medium
High
Very
High
Figure 1. Delay requirements on different applications of immersive percep-
tion
II. TELEOPERATION
Multi-modal telepresence and teleaction (TPTA) systems for
haptic telemanipulation, also known as telehaptic systems [10],
usually consist of one human operator using a haptic interface
(master device) on one end, a communication channel and
one teleoperator (slave actuator) on the other end (Figure 2).
For short distance applications the communication channel can
be a direct wired or wireless communication channel without
the need for network infrastructure. On the other hand, long
distance applications benefit from packet-switched network
infrastructures, and transmit their data as packets. The scale of
long distance applications can range from teleoperation over
local area networks to teleoperation over the Internet.
The goal of TPTA systems, as implied by their name, is
to provide to the user the feeling of presence in the remote
3
environment where the teleoperator exists. It is a goal which
can be achieved due to the ongoing improvement of the
relevant hardware and software for providing the human users
with multi-modal (visual, auditory, and haptic) feedback.
Comm.
Channel
Video/Audio
Position/Velocity
Force/Position
Surface texture
Force/Position
Surface texture
Video/Audio
Position/Velocity
Figure 2. An example of a haptic communication system. In this case, the
master device (user) sends position and/or velocity data while the slave device
(robot) transmits the haptic feedback data, audio and video data streams.
A. Classification of teleoperation systems
Nowadays, extensive research has been made in bilateral
and multilateral telehaptic systems [11], [12]. An approach
for classifying teleoperation systems can be based on the
different communication delays and interaction levels a user
may experience and results in two main categories, Direct
control systems and Supervisory control systems as described
in [13]. As shown in Figure 3, we subdivide each of these
categories further into subcategories:
1) Direct control: The human operator interacts in real-time
with the environment while the master and slave devices
communicate using position/force signals.
a) Closed-loop with negligible delay: In this case, the
communication channel presents minimum delay and
therefore the user is restricted to be in close proximity
to the slave device.
b) Time-delayed closed loop: The most common form of
teleoperation for digital closed-loop control systems.
Similarly to the previous subcategory, the master de-
vice controls the slave actuator but the user is less
restricted in terms of distance from the device (e.g.
transatlantic teleoperation). The remote side is not
autonomous, however, an internal control loop which
processes the command signals from the master device
is included in the teleoperator. In this case, the com-
munication channel (e.g. the Internet), may introduce
variable delays [6].
2) Supervisory control: The teleoperator is a) autonomously
or b) semi-autonomously controlled and receives high-
level commands from the master. It is also referred to
as task-based teleoperation. Examples are teleoperation
across planets or teleoperated robots with autonomous
functionalities [14].
B. Master and slave subsystems
Typically, at the master subsystem of a haptic bilateral
communication system, a human operator interacts with a
haptic interface which uses sensors and transmits motion data
(position or velocity data which are previously packetized)
over a communication channel, to the slave subsystem. In
Teleoperation
systems
Direct
control
Closed-loop
(negligible
delay)
Time-delayed
closed-loop
Supervisory
control
Semi-
autonomous
Fully
autonomous
Figure 3. A classification of teleoperation systems
return, the latter will respond with the force reflection/feedback
of the remote environment, in the form of kinesthetic or
vibrotactile force feedback data [12] while in some cases, such
as in the concept of virtual fixture, position data may also be
transmitted.
Haptic devices which are used as master teleoperation
interfaces, also called haptic manipulators, are comprised of
actuators and sensors which form the kinesthetic and tactile de-
vice subsystems. Such haptic devices may be able to reproduce
and process kinesthetic (kinesthetic interfaces), tactile (tactile
interfaces) or both types of haptic data (haptic interfaces). Such
devices have been created either as commercially available
products or prototypes for academic research.
In [15] the authors discuss the topic of haptic devices and
haptic actuators in relation to haptic communication over the
Tactile Internet, making the important point that there is a need
for ungrounded haptic devices with which the user does not
need to stay in a specific area, contrary to the current state
of haptic devices which are grounded. A list of hand-held
kinesthetic devices as well as a performance evaluation was
presented in [16]. As stated the most popular haptic interface
is the Geomagic Touch (formerly known as Phantom Omni).
These devices present specific technical characteristics such as
the Degrees-of-Freedom (DoF) they support (either for sensing
position or exerting force), the maximum force or torque they
can output, the usable space they can operate in and their
rotation capabilities (if their DoF specification allows them).
Many haptic interfaces, such as CyberGrasp [17] (an ex-
oskeleton device), may also be entirely wearable or have
wearable components in order to provide tactile feedback more
effectively. It is possible to use more than one actuator for
each finger. A variety of such interfaces are called tactile
displays and make use of tactile actuator arrays using various
technologies. Examples of such tactile devices are TPad [18],
which is applied to the screen of mobile phones and Gloveone
[19], a glove that provides tactile feedback to the fingers and
palm.
The hardware design parameters of haptic devices (e.g.
sampling frequency) and the number and type of sensors and
actuators determine the amount of data the device will output
or needs as input. They also determine the limitations of the
interaction between a user or an object and the device. A recent
detailed review of tactile sensors has been made in [20].
The slave haptic subsystem can be either a physical device
4
which interacts with a physical remote environment or a virtual
pointer of any form (e.g. a virtual hand) that operates in
a virtual environment. A key difference between physical
and virtual environments is that the control laws that govern
a physical environment are of continuous nature whereas a
virtual is of discrete nature. Virtual environments, even though
it is not feasible to perfectly replicate a physical environment,
have the advantage of allowing, in some cases, the interaction
among multiple users to interact with each other in a virtual
space over a local network or the Internet. By employing
the tactile or kinesthetic modalities these systems are called
Collaborative-Haptic Virtual Environments (C-HAVE) [21].
C. Challenges of teleoperation systems
Communication of haptic information for teleoperation sys-
tems imposes strong demands on the communication network.
This presents two main challenges for designing a reliable
teleoperation system.
First, haptic sensor readings from kinesthetic devices are
typically sampled, packetized and transmitted at a rate of
1 kHz or even higher [6], [22], [23] to maintain stability and
transparency of the system (further discussed in V-B). It must
be noted that this is not a strict requirement, however, accord-
ing to the stability analysis in [24]–[26] there is a relationship
between the sampling rate, the maximum displayed stiffness
and the system damping for ensuring system stability. A
teleoperation system operating with lower values of sampling
rate may still work and the user may be able to complete
a task. Nonetheless, the maximum displayed stiffness, while
guaranteeing system stability, is smaller than that of a higher
sampling rate and therefore the system may require larger
damping for stabilizing a hard contact.
Communication of kinesthetic information for teleoperation
systems, hence, requires a thousand or more haptic data
packets per second to be transmitted between the master
and the slave devices. Such a high packet rate may lead to
the consumption of a large amount of network resources in
combination with the transmission of audio and video data
and leads to inefficient data communication (see Section IV).
Therefore, haptic data reduction, or packet rate reduction, is
required in teleoperation systems. Moreover, tactile informa-
tion, especially in the form of complicated texture surfaces,
requires data compression.
Second, teleoperation systems are very sensitive to data loss
and latency [6]. Concerning the latter, a haptic communication
system device usually needs to transmit and receive a packet
every millisecond, otherwise stability cannot be guaranteed.
Consequently, an important question can be raised concerning
the amount of latency compared to the amount of data loss
that a system can tolerate. As it has been shown in [27] a
90% reduction can be attained, whereas even a small amount
of delay can disrupt the stability of a bilateral teleoperation
system. Even for a small communication delay or packet loss
rate, teleoperation systems may show stability issues making
degradation of teleoperation quality and task performance.
With the introduction of a communication channel such as
the Internet over mobile networks, this issue is inevitable.
Therefore, to guarantee system stability and improve Qual-
ity of Task (QoT) performance is a key objective of telemanip-
ulation systems [28], [29]. Quality of Task, presents the quality
of task performance and is usually quantified by measuring
the task completion time due to simplicity. Additionally, other
performance measures are the sum of squared forces (SOSF),
peak forces, task error/failure rate, the haptic device trajectory,
range of motion and velocity [30].
On the other hand, this also implies that the network
infrastructure itself, if improved to the point of meeting all
requirements, should be able provide adequate resources and
quality of communication for the best possible QoE and
decrease the dependence to altering haptic information.
In addition, haptic communication systems usually need to
provide to the user visual and audio feedback from the slave
subsystem. High packet rate, packet loss and variable delay
can cause the management and synchronization of the data
streams to become a challenging problem. In this case, packet-
switched network frameworks and protocols are needed for
synchronizing the data streams [31], for measuring the network
conditions and managing the Quality of Service [32].
Summing up, we detail three main solution spaces to
improve haptic communication:
• The communication network solution space covering both
aspects of the Internet and the mobile/wireless commu-
nication that enables the Tactile Internet.
• Data processing solutions to reduce data transmission
using perceptual thresholds or prediction methods in
order to compensate the incurred delay by long distance
communications.
• Stability control solutions to reduce the effect of extra
delay and provide stability for the control loop.
Improvements in all solution spaces of haptic communication
are under development and research. Main contributions to
these solution spaces will be presented in the next chapters.
Individual or joint improvement of the communication chan-
nel, control components and signal processing will guarantee
high teleoperation quality, system stability and scalability.
While current research studies address mainly these solution
spaces independently (few studies address two of these spaces
jointly), the ultimate solution for enabling the haptic commu-
nication should be based on joint optimization of these three
solution spaces. A discussion of future challenges on haptic
communication over 5G exists in Section VII.
III. HAPTIC COMMUNICATION OVER THE INTERNET
With the increase of mobile Internet-enabled machines and
devices over the world, mobile networks play an important
role as the medium for the transmission and reception of data.
In comparison to other networks, the more complex mobile
network infrastructure inevitably introduces latency into any
communication system.
Since using the Internet over a mobile network as a com-
munication channel can be responsible for most of the time
the transmitted information will be delayed, finding ways to
reduce this delay is inevitable. In this way, system stability
and transparency will be easier to maintain.
![Figure 5. A bilateral teleoperation system using the time-domain passivity control architecture (from [177]). The Passivity Observer (PO) entities compute the energy flows for both directions and provide input to the Passivity Controller (PC) on each side.](/figures/figure-5-a-bilateral-teleoperation-system-using-the-time-3xwh6bsa.webp)
