Cronfa - Swansea University Open Access Repository
_____________________________________________________________
This is an author produced version of a paper published in:
IEEE Transactions on Systems, Man, and Cybernetics: Systems
Cronfa URL for this paper:
http://cronfa.swan.ac.uk/Record/cronfa39979
_____________________________________________________________
Paper:
Yang, C., Wang, X., Li, Z., Li, Y. & Su, C. (2017). Teleoperation Control Based on Combination of Wave Variable and
Neural Networks. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(8), 2125-2136.
http://dx.doi.org/10.1109/TSMC.2016.2615061
_____________________________________________________________
This item is brought to you by Swansea University. Any person downloading material is agreeing to abide by the terms
of the repository licence. Copies of full text items may be used or reproduced in any format or medium, without prior
permission for personal research or study, educational or non-commercial purposes only. The copyright for any work
remains with the original author unless otherwise specified. The full-text must not be sold in any format or medium
without the formal permission of the copyright holder.
Permission for multiple reproductions should be obtained from the original author.
Authors are personally responsible for adhering to copyright and publisher restrictions when uploading content to the
repository.
http://www.swansea.ac.uk/library/researchsupport/ris-support/
Teleoperation Control based on Combination of
Wave Variable and Neural Networks
Chenguang Yang, Xingjian Wang, Zhijun Li, Yanan Li, Chun-Yi Su
Abstract—In this paper, a novel control scheme is developed for
a teleoperation system, combining the radial basis function (RBF)
neural networks (NNs) and wave variable technique to simultane-
ously compensate for the effects caused by communication delays
and dynamics uncertainties. The teleoperation system is set up
with a TouchX joystick as the master device and a simulated
Baxter robot arm as the slave robot. The haptic feedback is
provided to the human operator to sense the interaction force
between the slave robot and the environment when manipulating
the stylus of the joystick. To utilize the workspace of the telerobot
as much as possible, a matching process is carried out between
the master and the slave based on their kinematics models. The
closed loop inverse kinematics (CLIK) method and RBF NN
approximation technique are seamlessly integrated in the control
design. To overcome the potential instability problem in the
presence of delayed communication channels, wave variables and
their corrections are effectively embedded into the control system,
and Lyapunov based analysis is performed to theoretically
establish the closed-loop stability. Comparative experiments have
been conducted for a trajectory tracking task, under the different
conditions of various communication delays. Experimental results
show that in terms of tracking performance and force reflection,
the proposed control approach shows superior performance over
the conventional methods.
Index Terms—Neural Networks; Teleoperation Control; Wave
Variable; Time-Varying Delay
I. INTRODUCTION
In the past decades, robotic technologies have been devel-
oped rapidly in a wide range of engineering fields. The teler-
obot operation as one of the most attractive and challenging
topic in robotics has been used in various applications such
as telesurgery, search and rescue, 3D game development and
so on [1]. A typical teleoperation system usually comprises 5
This work was partially supported by National Nature Science Founda-
tion (NSFC) under Grants 61473120, 61573147 and 91520201, Guangdong
Provincial Natural Science Foundation 2014A030313266 and International
Science and Technology Collaboration Grant 2015A050502017, Science and
Technology Planning Project of Guangzhou 201607010006 and the Funda-
mental Research Funds for the Central Universities under Grant 2015ZM065.
C. Yang, X. Wang, Z. Li and C.-Y. Su are with Key Laboratory of
Autonomous Systems and Networked Control, College of Automation Science
and Engineering, South China University of Technology, Guangzhou, 510640
China. (Email: cyang@ieee.org; xj.wang
scut@qq.com; zjli@ieee.org;chun-
yi.su@concordia.ca). C. Yang is also with Zienkiewicz Centre for Computa-
tional Engineering, Swansea University, SA1 8EN, UK. C.-Y. Su is on leave
from Concordia University, Canada.
Y. Li is with Department of Bioengineering, Imperial College London, SW7
2AZ London, UK. (Email: yanan.li@imperial.ac.uk)
parts: human operator, master device, communication chan-
nels, slave robot and environment [2]. Usually, the human
operator controls the motion of a master device, which is
physically in contact with human. The master device generates
commanding trajectories passed through the communication
channels, which are passed to the slave robot that acts on
the target environment and completes the task. The interaction
force between the slave robot and the environment is fedback
to the master, according to which the human operator could
control the robot more effectively.
In this paper, we use a Geomagic
r
TouchX as the master
device, which is designed by the SensAble Technologies Inc.
TouchX is a haptic feedback device including both hardware
drives and software packages (OpenHaptics
r
toolkit) [1]. The
TouchX arm includes 3 rotational joints, each joint is equipped
with a motor to generate the feedback force. A 3 degrees of
freedom (DOFs) gimbal joint stylus is installed at the end of
the manipulator to provide the orientation motion. As a slave
robot, there are 7 revolute joints in each arm of Baxter, which
make it easy to move in the 3-D space [3]. In order to grasp
and handle the objects, a rotational gripper is installed at the
end-effector of each arm. The MATLAB Robotics Toolbox [4]
is used to establish the kinematics and dynamics models of the
Baxter robot arm, which is used as the slave telerobot to test
the proposed method.
As we known, the communication channels play significant
roles in a teleoperation systems, and time delays in the
channels may cause system unstable in the presence of force
feedback [5]. Much effort has been made to handle the effect
of time delays [6], especially in the bilateral teleoperation
system [7], [8]. The notion of wave variable was proposed
and it has been established in [9] that the wave variable could
guarantee the stability of the communication with time delays.
The influence of time-varying delays to the stability of a
teleoperation system was studied in [10]. In [2], the method
of integrating the corrected wave was investigated to remove
the distortion caused by the transmission of wave variables. A
wave variable based control was proposed in [11] to handle
the problem in the bilateral time-varying system. Based on this
idea, a novel approach was presented for controlling the time-
varying delayed teleoperation system with a PD controller
[12].
Due to the existence of uncertainties in practical application-
s [13], the research on controlling the uncertain robot system
becomes significantly important. The adaptive control method
for robots has been studied in a considerable number of works
[14]–[16], which could be used in the situations of unknown
parameters and time-varying parameters in the robot model
[17], [18]. Time-varying delays and uncertainties of the robot
model have been studied together with adaptive control in
[19], [20]. In [21]–[23], adaptive fuzzy control was used for
identification of the unknown nonlinear control system. Fuzzy
control was used for studying uncertain nonlinear systems in
[24], [25]. In [26], fuzzy control was adopted to improve
the performance of the automobile cruise system. In recent
years, the applications of NN to the robot control system have
become increasingly popular [27]–[31], due to the fact that
the NN has the ability to emulate complicated nonlinearity
and uncertain functions [32]–[34]. The RBF NN is a highly
effective method and has been extensively used for control
design of uncertain robot systems [35]. Adaptive RBF NN
based control has been investigated in [36], [37] to deal
with the deadzone and uncertain robotic model. In [38], an
adaptive NN method was applied to achieve control of the
uncertain marine vessel system. In [39], a controller was
designed for dual-arm coordination of a humanoid robot based
on the adaptive neural control. The tracking performance of
the adaptive NN control for a discrete-time system was studied
in [40]. In [41], the effectiveness of the NN control was firstly
considered in the Prandtl-Ishlinskii (PI) hysteresis system. The
RBF NN was investigated in [42] as a compensator to solve
the non-linearities problem that a standard PD controller could
not handle. In [43], the RBF NN was used to learn the robot
behavior, and in [44] the RBF NN was investigated to improve
the behaviour of the non-linear actuator. In [45], trajectory
control of a fruit and vegetable picking robot was studied,
and in [46] the RBF NN was used to compensate for the
deadzone of the non-linear system. In [47], the RBF NN has
been discussed in detail for compensation for the tracking error
in controlling mobile robots. In this paper, a NN controller
based on the PD control is applied to the slave robot with
7 DOFs, which guarantees more accurate trajectory tracking
than the conventional PD controller.
The reminder of the paper is organized as follows: in Section
II, the computational model of the master-slave teleoperation
system is analysed. In Section III, the PD control on the master
and slave is first discussed, and the nonlinear uncertainties
of the model of the slave robot are then analysed. The
CLIK method is used for avoiding kinematic singularities and
numerical drifts. The RBF NN control is designed for the
slave robot. Finally, the convergence of the tracking error and
the stability of the teleoperation system with the time-varying
delays are established. Comparative experiments are carried
out in Section IV and conclusions and possible future work
are discussed in Section V.
Fig. 1. The system framework
Throughout this paper, the following notions are used
• 0
m×n
stands for an m × n dimensional zero matrix.
• k·k denotes the Euclidean norm of vectors and induced
norm of matrices.
• A := B means that B is defined as A.
• sup means the least upper bound of a partially ordered
set.
• λ(·) stands for eigenvalue of a matrix.
• ℜ(·) stands for real part of a complex number.
• L
2
is a function space and the functions in L
2
are
quadratically integrable. L
∞
is a function space and the
functions in L
∞
are essentially bounded measurable.
• tr stands for trace of a matrix.
II. MATHEMATICAL MODEL OF TELEOPERATION
SYSTEM
A. Illustration of Teleoperation System
The teleoperation system in this paper is shown in Fig. 1.
As we see, the human operator holds the stylus of the haptic
device and drives the motion of the master device, which will
be regarded as position commands. Through processing by the
master computer, the new commands will be generated by the
master computer and then passed to the slave computer and
received by the slave controller. The Baxter robot will move in
accordance with the commands from the slave controller. The
manipulator end-effector interacts with the environment, and
the interaction force is passed to the haptic device to be sensed
by the human operator, which will lead to a new movement
and new control commands. In next five subsections we will
analyze the mathematical models of the components of the
teleoperation system.
B. Kinematics and Dynamics of Master Robot Arm
The haptic device Geomagic TouchX not only sends com-
mands of movement to the master device, but also returns the
interaction force between the telerobot and the environment,
and this is very useful for the operator to regulate the contact
force [48]. The mathematical model of the master includes the
kinematics model and the dynamics model.
The kinematics model of TouchX is built based on its
structure, as shown in Fig. 2. With 6 revolute joints, three
of them are equipped with motors, and the other three are
gimbal joint stylus considered as an end-effector, making it
flexible to move within the workspace. For more concrete and
Fig. 2. The structure of TouchX
Fig. 3. Comparison between two kinematics models of TouchX based on
standard DH parameters and modified DH parameters
intuitive representation of its structure, the Denavit Hartenburg
(DH) parameters are used to build the kinematics model [3].
There are two representations for DH parameters, the stan-
dard DH convention [49] and the modified DH convention
[50], and the latter representation is used for the kinematics
modelling of the TouchX joystick in this work. According to
the standard DH convention, the origins of the coordinates
relevant to joint 4 and joint 5 as shown in the left panel of
Fig. 3 are the same. Consequently, the simulated robot mod-
eled by the MATLAB Robotics Toolbox should be modified.
Specifically, a
i−1
and d
i
are used to represent the link length
and the link offset, respectively, where i represents the ith
joint of the master device. α
i−1
and θ
i
are used to represent
the joint twist angle and joint angle, respectively. All the 6
joints of the master device are revolute, and the modified DH
parameters of the TouchX are obtained in Tab. I.
The DH parameters in Tab. I represent the structure char-
acteristics of the master device, from which the kinematics
model could be obtained. According to [3], the homogeneous
transformation between two adjacent coordinates in Fig. 1
TABLE I
DH PARAMET ERS (MODIFIED CONVENTION) OF THE MASTER DEVICE
Link i θ
i
(angle limit(deg)) d
i
a
i−1
α
i−1
(rad)
1 q
1
(-60∼60) 0 0 0
2
q
2
(0∼105) 0 0 −π/2
3
q
3
(-180∼180) 0 L
m1
0
4
q
4
(-145∼145) L
m2
0 −π/2
5 q
5
(-70∼70) 0 0 −π/2
6
q
6
(-145∼145) 0 0 −π/2
could be formulated using DH parameters as follows:
i−1
A
i
(θ
i
, d
i
, a
i
, α
i
) =
cθ
i
−sθ
i
cα
i
sθ
i
cα
i
a
i
cθ
i
cθ
i
cθ
i
cα
i
−cθ
i
cα
i
a
i
sθ
i
0 sα
i
cα
i
d
i
0 0 0 1
(1)
where “c” is short for trigonometric function “ cos ” and “s”
is short for “ sin ”.
Moreover, the relationship between the position of the end
effector and the base could be calculated as follows:
n
X
0
=
0
A
1
1
A
2
...
n−1
A
n
· X
n
(2)
where n is 6 for the master device, X = [x, y, z, 1] represents
the position of the related joint, and
i
A
i+1
(i = 0, 1, ..., n −1)
represents the adjacent coordinate in (1).
The dynamics model of the master device reveals the
relationship between the driving torque or related force and
joint motion, and could be represented as follows:
M
m
(q
m
)¨q
m
+ h
m
(q
m
, ˙q
m
) = J
T
m
F
h
− τ
m
+ f
m
(3)
where
h
m
(q
m
, ˙q
m
) = C
m
(q
m
, ˙q
m
) ˙q
m
+ G
m
(q
m
) (4)
with the subscript “m” used to indicate master. For a robot
manipulator with n-DOF serial links and all the joints revolute,
q
m
, ˙q
m
and ¨q
m
∈ R
n
are the joint position, velocity and
acceleration, respectively. M
m
(q
m
)¨q
m
∈ R
n×n
is the inertia
matrix. h
m
(q
m
, ˙q
m
) represents the nonlinear coupling term
of the centripetal force, Coriolis force and the gravity force.
f
m
represents coulomb friction, load changes, time-delayed
jamming and other disturbances. J
m
is the Jacobian matrix
and J
T
m
is its transpose. F
h
is the force exerted by the human
operator and τ
m
is the torque control signal, both of which
will be applied to the master device. The terms on the left
hand side of Eqs. (3) and (4) satisfy the following properties
[51]:
Property 1: The matrix M
m
(q
m
) ∈ R
n×n
is a symmetric
positive-definite matrix.
Property 2: The matrix
˙
M
m
(q
m
)−2C
m
(q
m
, ˙q
m
) is a skew-
symmetric matrix, i.e., z
T
˙
M
m
− 2C
m
z = 0, ∀z ∈ R
n
.
Property 3: M
m
(q
m
) and G
m
(q
m
) are bounded, and
C
m
(q
m
, ˙q
m
) satisfies that ∀q
m
, ˙q
m
∈ R
n
, ∃K
cm
∈ R
>0
such
that kC
m
(q
m
, ˙q
m
)k ≤ K
cm
|˙q
m
|
2
.
C. Model of the Slave Robot
Fig. 4 shows the structure of the Baxter robot, which is
a dual-arm robot with 7-DOFs per arm. In this paper, the
simulated left arm of the Baxter robot is taken as the slave
telerobot.
The standard DH parameters are used to describe the
structure of the left arm of the Baxter robot, as shown in
Tab. II. The lengths mentioned in Fig. 4 and Tab. II are L
s0
= 0.27m, L
s1
= 0.069m, L
s2
= 0.364m, L
s3
= 0.069m, L
s4
Fig. 4. The structure of the Baxter robot
TABLE II
DH PARAMETERS (STANDARD CONV ENTION) OF THE SLAVE ROBOT
Link i θ
i
(anglelimit(deg)) d
i
a
i
α
i
(rad)
1 q
1
(-97.5∼97.5) L
s0
L
s1
−π/2
2
q
2
+
π
2
(-123∼60)
0 0 π/2
3
q
3
(-175 ∼ 175) L
s2
L
s3
−π/2
4
q
4
(2.865 ∼ 150) 0 0 π/2
5
q
5
(-175.27 ∼ 175.27) L
s4
L
s5
−π/2
6
q
6
(-90 ∼ 120) 0 0 π/2
7
q
7
(-175.27 ∼ 175.27) L
s6
0 0
= 0.375m, L
s5
= 0.01m and L
s6
= 0.28m. According to Eqs.
(1) and (2), the forward kinematics model of the Baxter robot
could be obtained, and n used in (2) is 7.
In order to achieve a precise tracking of the position com-
manded by the master joystick, a workspace matching between
the master joystick and the slave telerobot is essential. The
Monte Carlo method used in [3] was applied to approximate
the workspace for the master and the slave. In order to make
sure that the transformed workspace of the master device is
constrained within that of the slave robot, the workspace of
the master is scaled in a fixed proportion [52]. Fig. 5 shows
the workspace of the master and that of the slave after the
matching process. The top left panel of Fig. 5 shows the
enveloped surface, generated as convex hull of the 3D clouds
of the workspaces of both master and slave after the matching
process. Similar to the workspace transformation developed in
[3] between the Baxter robot arm and the Omni joystick, in
this work, the workspace transformation between the Baxter
robot arm and the TouchX joystick is given as below:
x
s
y
s
z
s
=
cosδ −sinδ 0
sinδ cosδ 0
0 0 1
×
S
x
0 0
0 S
y
0
0 0 S
z
x
m
y
m
z
m
+
T
x
T
y
T
z
(5)
where [x
s
, y
s
, z
s
]
T
, [x
m
, y
m
, z
m
]
T
represent the Cartesian co-
ordinates of the end-effectors of Baxter and TouchX joystick,
respectively. δ is the rotation angle of Z axis for the base
of the master device, [S
x
, S
y
, S
z
]
T
and [T
x
, T
y
, T
z
]
T
are the
proportionality factors and offset correction terms about the
X, Y and Z axes, respectively. According to [3], the matching
parameters of (6) are given by
δ =
π
4
,
S
x
S
y
S
z
=
0.0041
0.0040
0.0041
,
T
x
T
y
T
z
=
0.701
0.210
0.129
1.5
1
0.5
Y
0
-0.5
-11
0.5
X
0
-0.5
-1
0.5
1
1.5
-0.5
0
Z
(a)
X
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
Y
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
(b)
X
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
Z
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
(c)
Y
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2
Z
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
(d)
Fig. 5. Workspace matching. (a) The 3-D envelope surface of the master and
the slave. (b) Workspace matching in the X-Y plane. (c) Workspace matching
in the X-Z plane. (d) Workspace matching in the Y-Z plane
The dynamics model of the telerobot could be represented
as follows
M
s
(q
s
)¨q
s
+ h
s
(q
s
, ˙q
s
) = τ
s
− J
T
s
F
e
+ f
s
(6)
h
s
(q
s
, ˙q
s
) = C
s
(q
s
, ˙q
s
) ˙q
s
+ G
s
(q
s
) (7)
where the subscript “s” is used to indicate slave. The denota-
tions of the components on the left hand side of Eqs. (6) and
(7) are similar to those of Eqs. (3) and (4). F
e
is the interaction
force between the environment and the telerobot, and τ
s
is the
control input to the slave.
D. Model of the Human Operator
Early studies have shown that the muscle property of
the human hand could be modelled as a spring. A more
complete mass spring damper model is proposed in [53]. In
the teleoperation system, the human hand holds the stylus of
the master joystick, and gives the corresponding position and
velocity commands. Even if the master device is subjected
to the environmental force transmitted from the slave, the
operator could also adjust the output of the hand to make the
master track the desired movement of the operator. Therefore,
the human hand could actually be regarded as controlled by
an intelligent proportional-integral (PI) controller, which could
adjust the output force of the hand according to the error
between the actual position x
m
and the desired position x
md
of the master described by
F
h
= K
hp
(x
md
− x
m
) + K
hi
Z
t
t
0
(x
md
− x
m
)dt (8)
where K
hp
and K
hi
are the proportional gain and integral
gain of the human hand, respectively. The notations of t
0
and t are the initial time instant and the current time instant,
respectively.