King’s Research Portal
DOI:
10.1109/JPROC.2018.2867835
Document Version
Peer reviewed version
Link to publication record in King's Research Portal
Citation for published version (APA):
Steinbach, E., Strese, M., Eid, M., Liu, X., Bhardwaj, A., Liu, Q., Al-Ja'afreh, M., Mahmoodi, T., Hassen, R., El
Saddik, A., & Holland, O. (2019). Haptic Codecs for the Tactile Internet[40pt]. Proceedings of the IEEE, 107(2),
447 - 470. [8470161]. https://doi.org/10.1109/JPROC.2018.2867835
Citing this paper
Please note that where the full-text provided on King's Research Portal is the Author Accepted Manuscript or Post-Print version this may
differ from the final Published version. If citing, it is advised that you check and use the publisher's definitive version for pagination,
volume/issue, and date of publication details. And where the final published version is provided on the Research Portal, if citing you are
again advised to check the publisher's website for any subsequent corrections.
General rights
Copyright and moral rights for the publications made accessible in the Research Portal are retained by the authors and/or other copyright
owners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights.
•Users may download and print one copy of any publication from the Research Portal for the purpose of private study or research.
•You may not further distribute the material or use it for any profit-making activity or commercial gain
•You may freely distribute the URL identifying the publication in the Research Portal
Take down policy
If you believe that this document breaches copyright please contact librarypure@kcl.ac.uk providing details, and we will remove access to
the work immediately and investigate your claim.
Download date: 10. Aug. 2022
PROCEEDINGS OF THE IEEE 1
Haptic Codecs for the Tactile Internet
Eckehard Steinbach, Fellow IEEE, Matti Strese, Student Member IEEE, Mohamad Eid, Xun Liu,
Amit Bhardwaj, Qian Liu, Member IEEE, Mohammad Al-Ja’afreh,
Toktam Mahmoodi, Senior Member IEEE, Rania Hassen, Member IEEE,
Abdulmotaleb El Saddik, Fellow IEEE, and Oliver Holland, Member IEEE
Abstract—The Tactile Internet will enable users to physically explore remote environments and to make their skills available across
distances. An important technological aspect in this context is the acquisition, compression, transmission, and display of haptic
information. In this paper, we present the fundamentals and state-of-the-art in haptic codec design for the Tactile Internet. The
discussion covers both kinesthetic data reduction and tactile signal compression approaches. We put a special focus on how limitations
of the human haptic perception system can be exploited for efficient perceptual coding of kinesthetic and tactile information. Further
aspects addressed in this paper are the multiplexing of audio and video with haptic information and the quality evaluation of haptic
communication solutions. Finally, we describe the current status of the ongoing IEEE standardization activity P1918.1.1 which has the
ambition to standardize the first set of codecs for kinesthetic and tactile information exchange across communication networks.
Index Terms—Tactile Internet, Haptic Codecs, Perceptual Coding, Haptics
F
1 INTRODUCTION
V
ISUAL and auditory information are predominant in modern
multimedia systems. The acquisition, storage, transmission
and display of these modalities have reached a quality level which
is typically referred to as high-definition (HD) and beyond. For ex-
ample, high-end video cameras capture ultra-high-definition con-
tent, highly efficient video codecs such as H.265/HEVC achieve
remarkable compression factors, and high-resolution monitors and
Virtual Reality (VR) Head Mounted Displays (HMDs) enable
high-end virtual experiences. Similar HD technology for audio
is also available. Technical solutions addressing the sense of touch
(also referred to as haptic technology), in contrast, have not yet
reached the same level of sophistication.
In the context of the Tactile Internet [1], these solutions,
however, will significantly gain in relevance. Enabling remote
physical interaction with convincing touch experiences is one
of the key technologies that allows motor skills to be available
across distances and enables fully immersive multi-sensory remote
exploration of real or virtual environments where users can see,
hear and, in particular, feel remote objects. For the latter, haptic
information needs to be captured, compressed, transmitted and
• Eckehard Steinbach, Matti Strese and Rania Hassen and
Amit Bhardwaj are with the Chair of Media Technol-
ogy, Technical University of Munich, Munich, Germany.
{eckehard.steinbach,matti.strese, rania.hassen, amit.bhardwa j}@tum.de
Rania Hassen is also with the Computer Science Department,
Faculty of Computers and Information, Assiut University, Egypt
(rania.khairy@aun.edu.eg)
Abdulmotaleb El Saddik and Mohammad Al-Ja’afreh are with the
Multimedia Communications Research Laboratory (MCRLab), University
of Ottawa, Ottawa, Canada. {elsaddik, jaa f reh}@uottawa.ca
Mohamad Eid is with the Electrical and Computer Engineering in the
Engineering division at the New York University Abu Dhabi (NYUAD),
Abu Dhabi, UAE. {mohamad.eid@nyu.edu}
Toktam Mahmoodi, Xun Liu and Oliver Holland are with the Department
of Informatics at Kings College London, London, United Kingdom.
{toktam.mahmoodi, xun.2.liu, oliver.holland}@kcl.ac.uk
Qian Liu is with the Deptartment of Computer Science and Technology,
Dalian University of Technology, Dalian, China. {qianliu@dlut.edu.cn}
Manuscript received XXXXX; revised XXXXX
displayed with minimum latency. The compression of haptic
information is handled by haptic codecs which is the focus of
this paper.
Haptic data consists of two submodalities, i.e. kinesthetic
and tactile information (see Sections 2.1 and 3.1 for a detailed
description of the characteristics of both of these haptic submodal-
ities). While the compression of kinesthetic information has been
studied extensively in the context of bilateral teleoperation systems
with kinesthetic feedback (see e.g. [2]–[8]), the compression of
tactile information has received comparatively little attention so
far. This is an increasingly active area of research as the focus
in machine and computer haptics during recent years has clearly
shifted toward the realization of tactile touch experiences [9].
This is not surprising as we humans heavily rely on the tactile
modality to interact with objects in our environment. Also, from
a technical perspective, the tactile modality has high relevance in
many applications. In a Virtual Reality application, for example, a
typical intention of a user is to interact physically with the objects
in the virtual scene and to experience their material and surface
properties. Many challenges have to be overcome before tactile
solutions will reach the same level of sophistication as corre-
sponding HD video or audio solutions. With recent advances in
Virtual Reality (VR), Augmented Reality (AR) and Telepresence,
however, the topic is rapidly gaining in relevance and is becoming
an enabling technology for novel fields of application, such as
E-Commerce with tactile feedback (T-Commerce), telepresence
applications like Skype with touch interaction (T-Skype), or touch-
augmented VR systems (T-VR).
The main contributions of this paper can be summarized as
follows:
• We describe selected use cases and application scenarios
for haptic communication. This discussion motivates the
development of haptic codecs for the Tactile Internet.
• We present the state-of-the-art in the area of kinesthetic and
tactile data compression. In order to make this discussion as
accessible as possible, we provide the relevant background in
psychophysics and human haptic perception.
PROCEEDINGS OF THE IEEE 2
• We introduce the kinesthetic codec currently under inves-
tigation by the IEEE standardization group P1918.1.1. In
this context, we present the reference hardware and software
setup used to develop the kinesthetic codec as well as
the provided reference data traces. Furthermore, we present
the recently completed cross-validation experiments which
demonstrate that the selected kinesthetic codec solution
shows remarkable data reduction performance.
• We introduce a novel tactile processing pipeline which covers
the acquisition of surface material properties, the processing
of the acquired sensor signals, the compression of the raw
or processed tactile data as well as the presentation of
corresponding tactile experiences to the user. The latter takes
the interaction pattern of the user into account.
• We present the recently approved hardware and software
reference setup for tactile codec development within IEEE
P1918.1.1 which consists of a sensorized surface material
scanning tool and a voicecoil-based display. In this context
we also show example data traces which can be used to
evaluate tactile codecs.
• We provide an overview of the available objective quality
evaluation measures for kinesthetic information. These ob-
jective measures are experimentally evaluated and compared
with subjective evaluation results.
• Additionally, we discuss several topics which become rele-
vant in the context of the Tactile Internet, such as the mul-
tiplexing of several video, audio, and haptic data streams as
well as handshaking mechanisms for session establishment.
• Finally, we present the requirements for haptic codec design
identified by IEEE P1918.1.1 as well as the current status of
this standardization activity.
This paper is organized as follows. In the remainder of Section 1,
we further discuss the relevance of haptic communication for the
Tactile Internet. Additionally, we present several use cases for the
Tactile Internet which require high-fidelity haptic codec solutions.
Section 2 is then dedicated to tactile information and tactile
codecs. Section 3 provides details about kinesthetic information
and the state-of-the-art data reduction approaches for this type of
data. Section 4 addresses the multiplexing of audiovisual infor-
mation with haptic information. Section 5 discusses objective and
subjective quality evaluation approaches for haptic communication
solutions. Section 6 summarizes the current status of the ongoing
standardization activity IEEE P1918.1.1 Haptic Codecs for the
Tactile Internet. In Section 7 we conclude the paper.
1.1 The relevance of haptic communication for the Tac-
tile Internet
Emergence of the Tactile Internet [1], which aims at provid-
ing ultra-low delay and ultra-high reliability communications,
has enabled a paradigm shift from conventional content-oriented
communication to control-oriented communication. The Tactile
Internet is of particular relevance for the realization of human-in-
the-loop applications which are highly delay sensitive and require
a tight integration of the communication and control mechanisms
[10]. The human-in-the-loop Tactile Internet paves the way for
delivering human skills in addition to the human knowledge,
remotely, giving life to the Internet of skills [11]. Within this
paradigm, human multi-sensory information for interaction and
communication with the remote environment needs to be ex-
changed. To this end, haptic communications, by exchanging
kinesthetic and tactile information, provides the platform for the
human-in-the-loop Tactile Internet, and the possibility of deliver-
ing remote physical experiences globally.
1.2 Use cases and application scenarios
The human haptic perception system processes kinesthetic and
tactile stimuli simultaneously. Different sensing mechanisms are
responsible for perceiving the two haptic submodalities [12].
Depending on the Tactile Internet use case or application scenario
considered, one modality or the other or a combination of both
form the input to the haptic codecs. Please note that in haptic
technology the two modalities are often considered independently,
as different sensing and actuation principles are applied. For a
human user, however, both types of information are fused into a
joint touch experience. In the following, we discuss selected use
cases and application scenarios which rely on either kinesthetic
or tactile information exchange. Finally, we will discuss a virtual
material showroom as an example where the user benefits from a
combination of both modalities.
Fig. 1: Bilateral teleoperation with kinesthetic feedback. The
operator controls the position of the remote robot (teleoperator).
Interaction forces are measured during contact and sent back to the
operator. Additionally, visual and auditory information is streamed
back to the operator.
1.2.1 Bilateral Teleoperation with Kinesthetic Feedback
We start with bilateral teleoperation with kinesthetic feedback
which is the classical use case for kinesthetic information ex-
change. We keep this part relatively short as it has been discussed
in detail in many other works, e.g. as early as in 1967 by [13],
or, later in the context of stabilizing the closed-loop kinesthetic
interaction in the presence of communication delay in, e.g., [14]
and [15], or more recently in overview papers such as [16]
and [17]. Traditional teleoperation scenarios with purely kines-
thetic feedback enable the remote control of robots in, e.g., distant
or dangerous environments. Figure 1 illustrates a typical setup
where the user is connected to a kinesthetic input/output device
and the teleoperator is realized using a robotic arm equipped
with force sensors, a video camera, a microphone and an end-
effector or tool. Possible use cases are tele-maintenance and tele-
surgery. Although most previous works in teleoperation consider
the kinesthetic submodality only, the combination of kinesthetic
and tactile feedback promises improved user experience [18], [19].
Besides low-frequency kinesthetic force feedback, high-frequency
tactile signals and thermal feedback allow, e.g., for the remote
perception of object surface properties [20].
1.2.2 E-Commerce with Tactile Feedback
The presentation of object surface properties on touch screens
enables novel applications for online-shopping, which we denote
as T-Commerce in the following. For example, novel tactile
PROCEEDINGS OF THE IEEE 3
Fig. 2: Example applications which allow users to experience
selected surface properties of offered products on websites using
surface haptics displays (left: Tanvas Touch tablet [21], right: TPad
Phone [22]).
displays [21], [22] as shown in Fig. 2 allow for the display of
fine surface roughness information on glass displays. A high-
fidelity T-Commerce scenario, however, requires additional effort
in the object data acquisition, transmission and display. If we
want to provide the user with a high-quality and comprehen-
sive remote touch experience, a complete object representation
should be available which includes all relevant kinesthetic and
tactile properties. Ideally, the user will not be able to distinguish
between locally touching the real object and the provided online
experience.
1.2.3 Telepresence with Tactile Feedback (T-Skype)
Todays telepresence systems (e.g., video conference or video chat)
exchange high-quality audio and video among two or more par-
ticipants. While these systems at least partially fulfill the promise
of immersing a user into a remote space and to generate a certain
level of presence, they lack the capability to also exchange touch
experiences during interaction. If the users are equipped with
tactile actuators, tactile feedback experiences (e.g. vibrotactile
stimuli) can be provided. This could be for instance useful for
calming a child by touching her or him gently from a distance
during a business trip.
Fig. 3: Material Showroom. A user can interact with different
material probes and receives kinesthetic and tactile feedback.
1.2.4 Virtual Reality with Kinesthetic and Tactile Feedback
Current VR systems only display visual and audible information
to the user. The most intuitive reaction in VR, however, is trying to
grasp and interact with objects. A large number of wearable haptic
devices have been proposed during recent years to address this
issue (see [23] for a extensive survey in this field). As a common
problem, these devices generally lack the ability to combine
kinesthetic and tactile feedback in a lightweight system. The first
non-wearable approaches have been proposed that combine tactile
and kinesthetic feedback. For example, [24] uses the capabilities of
a kinesthetic interface (Phantom Omni device) augmented with a
vibromechanical actuator to create selected tactile stimuli (friction,
stiffness and roughness). The authors in [25] use a solenoid
plunger and a rolling stainless steel ball to render different friction
forces. Figure 3 shows the example of a haptic showroom where
the user can freely move around and interact with material samples
while receiving both kinesthetic and tactile feedback.
2 TACTILE INFORMATION AND TACTILE CODECS
Besides a solid understanding of human psychophysics, the provi-
sion of high-quality tactile experiences in the context of the Tactile
Internet requires, in our opinion, three main components: efficient
acquisition of tactile object properties, analysis and compression
of tactile information, and tactile display technology which ideally
can reproduce all relevant tactile dimensions simultaneously. The
corresponding tactile pipeline is shown in Fig. 4.
2.1 Tactile Perception
This section first describes the human tactile perception of object
properties. It is followed by approaches to collect and display
such tactile information as well as how it can be compressed and
transmitted. Table 1, which is adapted from [26] and reproduced
from [27], shows the mechanoreceptors that are responsible for
human tactile perception of, e.g., fine roughness or friction.
TABLE 1: Function, roles and respective frequency range of
four types of mechanoreceptors in the human skin (reproduced
from [27]).
Merkel
cell
Ruffini
ending
Meissner
corpuscle
Pacinian
corpuscle
Best
stimulus
Pressure
(hardness),
edges,
corner,
points
Stretch
Lateral
motion
High-
frequency
vibration
Example
use cases
Reading
braille
Holding
large
objects
Sensing
slippage
of objects
(friction)
Sensing
haptic texture
Frequency
range (Hz)
0 − 100 / 1 − 300 5 − 1, 000
Most
sensitive
frequency (Hz)
5 / 50 200
2.1.1 Object Identification
The human haptic perception system relies on kinesthetic as well
as tactile sensory information in the interaction with objects.
Humans typically perform six types of exploration patterns, as
described in [28], [29], to identify unknown objects. During the
interaction with objects, enclosure and contour following reveal
spatial content about the object shape and its coarse contour
properties. Humans lift objects to estimate their weight. Static
touch is used to identify the thermal conductance through the
bare finger. Pressing on the material reveals information about
its stiffness. Finally, arbitrary sliding motions allow for the
perception of the fine roughness, also known as haptic texture,
and the friction properties of the object surface.
PROCEEDINGS OF THE IEEE 4
Waveform-based
Compression, Transmission
and Reconstruction
Feature Extraction for
Parametric Representati-
on and Parameter
Transmission
Microphone
Accelerometer
Camera
Infrared Sensor
Additional Sensors
Sensorized
Material
Recording
Device
Real World
Objects
Tactile
Display
Technologies
Interaction Parameters
Material Data-
base
Material Classification
or Retrieval and Material
Identifier Transmission
Database
Lookup
Quality of Experience
Material Data-
base
Tactile
Rendering
Signal
Preparation for
Display
[1] Source: http://guidetooilpainting.com/texture.html
[1]
Optimization
Acquisition of Tactile
Information
Representation, Analysis
and Transmission
Display of Tactile Information
Human Skin
Human
Interaction
Parameters
Fig. 4: Acquisition, analysis, transmission, and display of tactile information.
2.1.2 Tactile Dimensions
Based on the evaluation of the adjectives used to describe tactile
information in previous studies, the authors in [30] have identified
five major tactile dimensions:
1) Friction between a bare finger and a surface forces the
human to apply a specific lateral force during sliding motions.
Components of friction models [31] commonly are the surface-
specific force to break the adhesion with it, or, the required traction
to slide the bare finger [32] [33].
2) Hardness perception results from specific exploration pat-
terns such as tapping on an object surface, pinching an object, or
pressing on the surface [34]–[36]. The authors in [37] compared
the realism of virtual surfaces using a database approach, an
input-output approach, and Hooke’s law. The study showed that
overlaying either the recorded acceleration transients or manually
tuned and velocity-scaled decaying sinusoids on a virtual surface
resulted in a perceived hardness that closely matched that of a
real surface. LaMottes study of tool-based interactions found that
humans were significantly better at discriminating the hardness
of surfaces when tapping rather than when pressing into the sur-
face [34]. This result indicates that the transient vibrations elicited
by tapping largely determine the surfaces perceived hardness and
can be used to change the perceived hardness of a virtual surface.
This dimension further considers, e.g., the compliance, or, the
persistence of the material deformation [32].
3) Warmth conductivity is perceived by the thermal receptors
in the human skin [29]. The combination of the ambient temper-
ature and the warmth conductivity of a material determine how
warm or cold the direct touch is perceived [38]. The response
range of thermal receptors lies in the range of 5
◦
C − 45
◦
C.
The influence of warmth conductivity is underscored in current
research [29], because materials like glass and steel can only
be discriminated by their different warmth conductivities [38]
in the absence of visual surface information. While some object
properties such as the shape or size can be visually determined
by a human without touch, thermal attributes of objects or the
environment can only be sensed through the skin. Basically,
humans are very sensitive to rapid changes in temperature, but
respond slowly to gradual changes [27]. As a result, temperature
scenarios are classified into four types, i.e. cold, cool, warm and
hot. The recent study in [27] presents that the receptors responsible
for thermal sensation include four classes of thermo-receptors:
1) high-threshold cold receptors 2) low-threshold cold receptors
3) high-threshold warm receptors and 4) low-threshold warm
receptors, and two classes of nociceptors, i.e., 1) heat nociceptor
and 2) cold nociceptor. The low-threshold cold receptor is sensitive
to sudden cooling changes, such as a breeze from an open window,
whereas the high-threshold cold receptor is less sensitive to the
temperature change, but functions sensibly when the temperature
is very low, even below 0
◦
C. Low-threshold and high-threshold
warm receptors are classified in a similar way. Nociceptors are
responsible for sensing pain when the skin temperature is beyond
a certain threshold [39].
4) Macroscopic Roughness and 5) Microscopic Roughness
The duplex nature of roughness (introduced by David Katz in
1925 [40]), consisting of microscopic and macroscopic roughness,
has been described and confirmed in different works like [29]
and [30] and is based on the presence of different mechanorecep-
tors in the human skin. The surface material structural threshold
between coarse and fine haptic textures has been determined as
approximately 200 microns [41]. Four types of receptors, namely,
cutaneous and subcutaneous mechanoreceptors, are responsible
for the sense of touch. These mechanoreceptors, including Meiss-
ner corpuscles, Merkel cells, Pacinian corpuscles and Ruffini
endings are described in [26]. Their function and roles are also
shown in Table 1.
Macroscopic Roughness comprises the existence of visible
height profiles and the regularity of the surface of the object. These
spatial cues are responsible for coarse structures and sensations
described as uneven, relief or voluminous. Object surfaces can be
