Reactionless Visual Servoing of a Dual-Arm Space Robot
A. H. Abdul Hafez, V. V. Anurag , S. V. Shah, K. Madhava Krishna, and C. V. Jawahar
Abstract— This paper presents a novel visual servoing con-
troller for a satellite mounted dual-arm space robot. The con-
troller is designed to complete the task of servoing the robot’s
endeffectors to the desired pose, while regulating orientation
of the base-satellite. Task redundancy approach is utilized to
coordinate the servoing process and attitude of the base satellite.
The visual task is defined as a primary task, while regulating
attitude of the base satellite to zero is defined as a secondary
task. The secondary task is formulated as an optimization
problem in such a way that it does not affect the primary
task, and simultaneously minimizes its cost function. A set of
numerical experiments are carried out on a dual-arm space
robot showing efficacy of the proposed control methodology.
I. INTRODUCTION
One of the major areas in space science that demands for
immediate attention and commercial drive is the On Orbit
Services (OSS) [1], e.g., orbital detritus management, refur-
bishment and refuelling of orbiting satellites, construction
in space, etc. The OSS can be divided into rendezvous,
proximity operations, and servicing. Use of robots boosts
reliability, safety and ease of execution of operations during
or after proximity operations [2]. It is highly desired that
the closing over manoeuvre of the robot is carried out
autonomously due to communication time delay between
the service satellite and ground station [3]. This calls for
a control technique which makes use of the on-board ma-
chine vision facility for successfully completing OOS in an
autonomous manner. Visual servoing is one such vision-
based technique commonly used for control of the earth-
based robots, and have made many recent progresses [4], [5],
[6], [7]. In this work, we attempt to solve a visual servoing
problem for a satellite mounted space robot.
Visual servoing is inevitable for the space robots due
to communication time delay [3]. Moreover, variable light
conditions, absence of marker, and limited computing power
make visual servoing further challenging [8]. Visual servoing
was demonstrated for autonomous satellite capture in [3].
Later, they [8] showed visual servoing under limited avail-
ability of computing power and severe lighting conditions.
Earth-based experimental evaluation for robotic capture of a
helium airship using visual servoing was illustrated in [9].
There, the objective was to emulate capture of a free-floating
object without emphasizing on effect of the robot’s motion
∗
This research was supported by INSPIRE research Grant (IFA-13 ENG-
52) by Department of Science and Technology, India.
A. H. Abdul Hafez was with International Institute of Information
Technology, Hyderabad - 500032, India, during this work. He is now with
Hasan Kalyoncu University, Sahinbey - 27410, Gaziantep, Turkey
V. V. Anurag, S. V. Shah, K. Madhava Krishna, and C. V. Jawahar are
with International Institute of Information Technology, Hyderabad - 500032,
India
on the satellite. In the above works [3], [8], [9], satellite
is either assumed to be controlled or operated in the free-
floating mode. Use of attitude controller will consume fuel
which is reserved mainly for orbital manoeuvres.
Robot’s operation in the free-floating mode using the
Generalized Jacobian Matrix (GJM) [10] is an another alter-
native. This although helps in reducing fuel consumption but
causes change in the orientation of the base satellite. This
may destabilise the satellite, cause damage to its internal
hardware, and result in loss of communication with data
relay satellite or ground station. Regulating attitude of the
base satellite to zero in the free-floating mode is another
objective of this work along with visual servoing.
In this regard, several researchers have focused on robotic
manipulation with minimum attitude disturbance of the base
satellite. This is also known as reactionless manipulation
of robotic arm. This not only helps in keeping attitude
disturbances minimum but also results into fuel saving. In
this regard, the disturbance maps were proposed [11] to
minimise change in the base attitude, but were not able to
completely eliminate it. A Reaction Null Space (RNS) based
formulation was also proposed in [12] which led to zero
attitude disturbances of the base satellite. Recently, strategies
for reactionless capture of tumbling objects using a dual-
arm robot were proposed in [13]. These works, however,
stressed on reactionless path planning without emphasizing
on vision-based control. In this work, we introduce a vision-
based control for reactionless manipulation of a dual-arm
robot. In our proposal, both visual servoing and reaction-
less manipulation are treated as different tasks. Multi-task
approach was presented in [14] for visual servoing of an
earth-based Cartesian robot with joint-limit and singularity
avoidance as the additional tasks using gradient projection
method. Success of such approach depends on proper tuning
of a parameter that decides amplitude of secondary task.
In order to overcome this disadvantage, an iterative scheme
was presented for joint limit avoidance as a secondary task
in [15]. Both the above methods required free degrees-of-
freedom (DOF) for secondary task. An improved task func-
tion approach was illustrated in [16], where the secondary
task exploits DOF constrained by the main task in addition
to the redundant DOF for faster completion of the task. The
success of task functional approach on the earth-based robots
inspired to use it for space robot.
However, implementation of visual servoing as the primary
task and reactionless manipulation as the secondary task
is a non-trivial problem for space robot mainly due to,
1) dynamic coupling between two arms and satellite, and
2) nonholonomic nature of the constraints for reactionless
manipulation. In this work, a solution is proposed for the first
two problems, which form the fundamental contributions of
this work. The efficacy of the proposed approach is illustrated
using a 6-DOF planar dual-arm robotic system mounted on
a service satellite.
The rest of the paper is organized as follows: The Gen-
eralized Jacobian for a dual-arm space robot is presented
in Section II. Visual servoing using the GJM is presented in
Section III, whereas reactionless visual servoing is discussed
in Section IV, Section V presents results and discussion.
Finally, conclusions are given in Section VI.
II. THE GENERALIZED JACOBIAN MATRIX (GJM)
In the proposed visual servoing, the base satellite is
assumed to be free floating, and hence, it is very important
to obtain the Jacobian that maps joint velocities into the
end-effectors’ velocities by incorporating motion of the base
satellite. For an n-DOF dual-arm robotic system mounted
on a floating-base with 2-end-effectors, as shown in Fig. 1,
the end-effectors’ velocities (t
e
= [t
e1
T
t
e2
T
]
T
∈ R
12
) are
expressed in terms of the base velocity and joint velocity as
t
e
= J
be
t
b
+ J
me
˙
θ, (1)
where t
b
= [v
T
b
, ω
T
b
]
T
∈ R
6
is the twist vector constituting
linear velocity (v
b
) and angular velocity (ω
b
) of the base,
˙
θ = [
˙
θ
1
T
˙
θ
2
T
]
T
∈ R
n
is the vector of manipulator joint
velocities, J
be
= [J
be1
T
J
be2
T
]
T
∈ R
12×6
is the Jacobian
matrix for base, and J
me
=
J
me1
O
O J
me2
∈ R
12×n
is
the Jacobian matrix for manipulator. Note that 1 and 2 in the
subscripts represent arm-1 and arm-2, respectively, and O is
the null matrix of compatible dimensions.
In (1), end-effectors’ velocities (t
e
) is represented in terms
of both base velocities (t
b
) and joint velocities (
˙
θ). In order to
obtain Jacobian which maps
˙
θ directly into t
e
, it is required
to calculate t
b
in terms of
˙
θ. This is obtained from the
expressions of linear momentum (p) and angular momentum
(l) as an extension of single-arm robot in [12], as
p
l
= I
b
t
b
+ I
bm1
˙
θ
1
+ I
bm2
˙
θ
2
+
0
c
0
× p
. (2)
In (2), I
b
∈ R
6×6
is the inertia matrix of the floating-
base, and I
bm1
, I
bm2
∈ R
6×(n/2)
are the coupling inertia
matrices for arm-1 and arm-2, respectively. Substituting t
b
from (2) into (1), one obtains
t
e
=
J
me
− J
be
I
−1
b
I
bm
˙
θ
1
˙
θ
2
+J
be
I
−1
b
p
l − c
0
× p
,
(3)
where I
bm
= [I
bm1
I
bm2
]. If no external force is acting on
the base, and the system starts from rest, p = l = 0, and
t
e
= J
g
˙
θ, where J
g
=
J
me
− J
be
I
−1
b
I
bm
. (4)
In (4), J
g
is referred to as Generalized Jacobian Matrix
(GJM) [10]. The GJM in (4) is different than the Jacobian
of the earth-based manipulator as the former contains inertia
terms. It will be used for reactionless visual servoing of the
dual-arm space robot in the subsequent sections.
Fig. 1. A dual-arm robot mounted on a service satellite
III. VISUAL SERVOING OF A DUAL-ARM SPACE ROBOT
In contrast to the earth-based dual-arm robot, the dual-
arm space robot has coupled motion of arms and base
satellite. This is evident from (4) where mapping between
end-effectors’ velocity and joint rates is no more function of
mere kinematic parameters. Therefore, this section empha-
sizes on visual servoing of a dual-arm space robot without
reactionless manipulation.
Visual servoing uses signals extracted from visual infor-
mation as a feedback to close the control loop [17]. The
dual-arm robot carries one camera each on both arms, hence,
the visual servoing control law can be defined as
t
c
= −λL
+
s
e, (5)
where L
S
∈ R
2N×12
is the image Jacobian or interaction
matrix, t
c
is the camera velocity, λ is a scalar gain which
determines the speed of convergence of the visual servoing,
and e is the error between the current features (s) and
the desired features (s
∗
). L
S
and t
c
have the following
representation:
t
c
=
t
c1
t
c2
, and L
S
=
L
s1
O
O L
s2
. (6)
In (6), L
s1
and L
s2
are the image Jacobians for arm-1
and arm-2, respectively, and t
ci
= [v
T
ci
, ω
T
ci
]
T
is the twist
vector of the camera-i, for i = 1, 2. It is convenient to obtain
feature velocities in terms of joint velocities as the actuators
are placed at the joints.
For the dual-arm space robot, it is not possible to decouple
motion of the arms, and obtain two independent Jacobians.
As discussed in Section 2, the joint velocities of the space
robot can be mapped into end-effector velocities using the
GJM, J
g
, as given in (4). Therefore, (5) can be rewritten in
terms of joint velocities of arm-1 and arm-2 using (4) as
˙
θ
1
˙
θ
2
= −λJ
+
1
˙e
1
˙e
2
, (7)
where the modified image Jacobian J
1
= L
S
J
g
. It is worth
noting that even though
˙
θ
1
and
˙
θ
2
, obtained from the (7),
will servo the manipulator to achieve the desired features, it
will produce uncontrolled motion of the floating-base, i.e.,
base satellite. Having solution of
˙
θ = [
˙
θ
T
1
˙
θ
T
1
]
T
, and t
e
=
J
g
˙
θ from (4), the motion of the base-satellite is obtained
using (1) as
t
b
= J
−1
be
(t
e
− J
me
˙
θ). (8)
As there is no control on the motion of the base satellite,
this can result into attitude disturbance as will be shown in
Section V.
IV. REACTIONLESS VISUAL SERVOING CONTROL
As discussed in the previous section, the GJM-based visual
servoing in (7) leads to attitude disturbances. In this section,
a solution based on task function approach is proposed to
overcome the above disadvantage.
A. Task Function Approach for Reactionless Visual Servoing
Task function approach uses some DOFs for completion
of visual servoing task whereas redundant DOFs are used
for completion of additional tasks. In (6), e is designed as a
visual servoing task for the dual-arm robot. This is referred
to as the primary task. On the other hand regulating the
base attitude disturbance to zero is treated as the secondary
task and denoted as e
s
. Using the task function approach
[18], regulation of the primary and secondary robotic tasks
is formulated as
e
t
= J
+
1
e + β(I
n
− J
+
1
J
1
)e
s
, (9)
where e
t
is the total robotic task. The projection operator
(I
n
− J
+
1
J
1
) ensures that the realization of the secondary
task, e
s
, does not affect the main task, e. In fact, the columns
of the operator (I
n
− J
+
1
J
1
) belongs to the null space
of J
1
. In (9), the value of β is critical and needs to be
accurately tuned [15]. Large value of β will result into some
oscillations, while too small values may cause a steady state
error in the end effector velocity. The proposed solution to
the reactionless visual servoing takes care of tuning of β.
It is done automatically within the optimization framework
that defines the secondary task. Finally, the visual servoing
control law in terms of joint velocities is obtained as
˙
θ = −λe
t
. (10)
B. Defining the Secondary Task
The secondary task is traditionally used in the robotic
literature to satisfy a set of constraints in addition to the main
task to be completed. These constraints are represented using
a cost function whose value is null when these constraints
are satisfied. Hence, the gradient of the cost function is used
as a secondary task to be regulated to zero during the main
task, i.e., the servoing task [15]. For the satellite mounted
dual-arm robot under study, the objective is to perform the
servoing without any base reaction moment. Hence, it will
be referred to as reactionless visual servoing. It is assumed
that the satellite is free to move along Cartesian axes. In
order to obtain constraints for reactionless manipulation, (2)
is first rewritten only in terms of ω
0
. Note that matrices I
b
and I
bm
in (2) have the following block representation:
I
b
=
I
b,v
I
T
b,c
I
b,c
I
b,ω
; I
bm
=
I
bm1,v
I
bm2,v
I
bm1,ω
I
bm2,ω
. (11)
Using (2) and (11), the expression of angular momentum
l in (2) can also be reformulated in terms of ω
0
as
l =
˜
I
b
ω
0
+
˜
I
bm1
˙
θ
1
+
˜
I
bm2
˙
θ
2
+ c
com
× p, (12)
where
˜
I
b
= I
b,ω
− I
−1
b,v
I
b,c
I
T
b,c
;
˜
I
bmi
= I
bmi,ω
− I
−1
b,v
I
b,c
I
bmi,v
.
(13)
As the angular momentum is conserved and system starts
from the rest, i.e., l = p = 0, (12) simplifies to
˜
I
b
ω
0
+
˜
I
bm1
˙
θ
1
+
˜
I
bm2
˙
θ
2
= 0. (14)
If stationary state of the attitude of the base is maintained,
i.e., ω
0
= 0 , then
˜
I
bm1
˙
θ
1
+
˜
I
bm2
˙
θ
2
= 0. (15)
where
˜
I
bmi
is referred to as coupling angular momentum.
It is clear from (15) that for a dual-arm robot, it is sum
of the coupling angular momenta of both arms, not of the
individual arms, has to be zero for reactionless manipulation.
Hence, the secondary cost function is taken as
h
s
= k
˜
I
bm
˙
θk, (16)
where
˜
I
bm
= [
˜
I
bm1
˜
I
bm2
] and
˙
θ = [
˙
θ
T
1
˙
θ
T
2
]
T
.
Classical way to define the secondary task is the gradient
of the cost function, i.e. e
s
= ∂hs/∂θ. However, here, h
s
is
nonholonomic and function of
˙
θ, which is also the output of
visual servoing controller (10). Hence, the gradient cannot
be derived in terms of θ.
There are many alternatives to avoid the direct analytical
computation of the gradient [19]. For example, the gradient
can be locally estimated using numerical methods. Another
method is to consider the secondary task as a design variable
˜e
s
. This variable is estimated in such a way that it minimizes
the cost function h
s
. i.e.,
˜e
s
= arg min
e
s
(h
s
). (17)
Using (9), (16) and (10), (17) can be rewritten as follows:
˜e
s
= arg min
e
s
(k−λ
˜
I
bm
(J
+
1
e + β(I
n
− J
+
1
J
1
)e
s
)k) (18)
where βe
s
= ˜e
s
, and β is set to one. In other words, the
variable β is augmented with the secondary task e
s
as design
parameter. The local optimization estimates both of them
as the parameter ˜e
s
that minimize the cost function h
s
. As
mentioned earlier, the parameter β is critical since it affects
the behaviour of the visual servoing control law. In this work,
no special procedure is required to estimated value of β.
Efficacy of the proposed approach for reactionless visual
serving of a 6-DOF planar dual-arm robot is illustrated in
the next section.
Fig. 2. Schematic of a 6-DOF planar dual-arm space robot
TABLE I
MODEL PARAMETERS OF THE SATELLITE AND DUAL-ARM ROBOT
Satellite Arm-1 and 2
Link-1 Link-2 Link-3
mass(Kg) 500 10 10 10
length(m) 1×1 1 1 1
I
zz
(Kg.m
2
) 83.61 1.05 1.05 1.05
V. RESULTS AND DISCUSSIONS
In order to illustrate the proposed reactionless visual
servoing, a 6-DOF planar dual-arm robotic system mounted
on a service satellite is considered, as shown in Fig. 2.
The model parameters of the dual-arm and satellite are
shown in Table I. Each manipulator has 3-DOF and three
links connected by the revolute joints. The dual-arm is
placed in XY-plane, and the relative joint angles for both
arms are shown in Fig. 2. The end-effector of each arm
is mounted with a camera. Two points to be tracked are
placed at (−0.5m, 2m, −0.1m) and (−0.5m, 2m, 0.1m).
The cameras observe the point features and the image co-
ordinates are extracted for the purpose of visual servoing.
The desired features are calculated based on the desired pose
of camera-1 and camera-2 as (−0.14m, 1.78m, 2.56rad) and
(−0.7m, , 1.57m, 1.04rad), respectively. In order to validate
the formulation developed in Sections III-IV, the following
numerical experiments were carried out:
• Case A: Visual servoing with the GJM-based control.
• Case B: Reactionless visual servoing with the aug-
mented GJM-based control and task function approach.
Results are obtained using space robot module of Recursive
Dynamics Simulator (ReDySim) [20], and elaborated in the
−2 −1 0 1 2
−2
−1
0
1
2
3
X (m)
Y (m)
0 5
−0.14
−0.12
−0.1
−0.08
−0.06
−0.04
−0.02
0
Time (sec)
Base Angular Position (rad)
Fig. 3. Initial (dashed line) and final (solid lines) configurations of the
robot (left), and angular position of the base satellite (right) in Case A.
0 2 4
0
100
200
300
400
500
Time (sec)
norm(e) pixel
cam−1
cam−2
0 2 4
−10
−5
0
5
10
Time(sec)
Joint Rates (rad/s)
˙
θ
1
˙
θ
2
˙
θ
3
Fig. 4. Norm of pixel error and joint rates of arm-1 in Case A
Fig. 5. Variation of features for arm-1 (left) and arm-2 (right) in Case A
next subsections.
A. Case A: Visual servoing with the GJM-based control
In order to validate theory developed in Section III, and
to highlight the problems that needs to be mitigated in the
reactionless visual servoing, the GJM-based visual servoing
of the dual-arm robot is presented in this section. Here, the
attitude controller is assumed to be shut-off, and the base
satellite works in the free-floating mode. Next, the GJM-
based control law derived in (7) is applied. Since there is
no external force acting on the satellite, the total linear and
angular momenta are conserved. Therefore, any reaction due
to motion of the dual-arm causes change in the orientation
of the base satellite which is evident from Fig. 3, where the
maximum change in the orientation is 0.13rad, which is not
desired. Figure 4, however, shows successful visual servoing
where both norm of pixel error and joint rates for arm-1
converge to zero. Moreover, observed features (in blue) move
towards the desired features (in red) as depicted in Fig. 5.
0 5 10
−5
0
5
10
15
x 10
−5
Time (s)
Linear momentum (N.s)
p
x
p
y
0 5 10
−4
−2
0
2
4
x 10
−4
Time (s)
Angular Momentum (Nm.s)
Fig. 6. Conservation of linear and angular momenta in Case A.
In order to validate the results of numerical experiment, the
linear and angular momenta are plotted in Fig. 6. It can be
seen that they are of order 10
−4
. This confirms correctness
of the simulation results as momentum is conserved.
B. Case B: Reactionless Visual Servoing
The problem of base attitude disturbance with the GJM-
based visual servoing has been brought forward through
the experiment illustrated in the previous section. In order
to overcome this disadvantage, the task function approach
proposed in Section IV is implemented for reactionless visual
servoing. For this, optimal control law derived in (18), which
ensures that the motion of the dual-arm does not affect the
base attitude, is used. It is worth noting that the modified
image Jacobian J
1
∈ R
8×6
has maximum rank equals to 6.
Here, as the two feature points are along vertical line in the
image plane, and the robot moves in a plane perpendicular
to the vertical line, the rank of J
1
is reduced by two, i.e.,
rank(J
1
) = 4. Hence, there exits a null space, dimension
of which is 2. This is utilized in designing the secondary
task. Figure 7 shows that the change in the orientation
of the base satellite is 1.22 × 10
−6
rad using the proposed
method. This shows significant improvement in comparison
to the previous subsection where the change in attitude of
the base satellite was 0.13rad. It can also be seen from
Fig. 8 that the norm of pixel error showing the performance
of primary task, and joint rates reduce to zero proving
successful visual servoing. This is also depicted in Fig. 9
where the observed feature (in blue) approached the desired
feature (in red). Cost of secondary task, i.e., coupling angular
momentum (
˜
I
bm
˙
θ), is also plotted in Fig. 10 with dashed line
and compared with the same for Case A, discussed in the
previous subsection. It can be seen that the maximum value
of
˜
I
bm
˙
θ was 160.55Nm.s in the case of mere GJM-based
control (Case A) on the other hand it was reduced to only
1.7 × 10
−4
Nm.s in the case of reactionless visual servoing
(Case B). This shows efficacy of the proposed control.
C. Performance Evaluation of the Proposed Algorithm
In order to evaluate performance of the proposed reac-
tionless visual servoing (Case B) in minimizing the base
attitude disturbance, six more numerical experiments have
been carried out with different initial configurations and
−2 −1 0 1 2
−2
−1
0
1
2
3
X (m)
Y (m)
0 5
−1
−0.5
0
0.5
1
x 10
−5
Time (sec)
Base Angular Position (rad)
Fig. 7. Initial (dashed line) and final (solid lines) configurations of the
robot (left), and angular position of the base satellite (right) in Case B.
0 2 4
0
100
200
300
400
500
Time (sec)
norm(e) pixel
cam−1
cam−2
0 2 4
−20
0
20
40
60
Time(sec)
Joint Rates (rad/s)
˙
θ
1
˙
θ
2
˙
θ
3
Fig. 8. Norm of pixel error and joint rates of arm-1 in Case B
Fig. 9. Variation of features for arm-1 (left) and arm-2 (right) in Case B.
location of the feature points. The results are compared
with the GJM-based visual servoing (Case A) for the norm
of pixel error, maximum coupling angular momentum and
maximum change in the base orientation as shown in Table
II. Note that the first experiment is the one discussed in the
previous two subsections.
Table II clearly shows that for Case B, the end result
of primary task, i.e., pixel error, is not influenced by both
presence of the secondary task and variation of initial condi-
tions. This shows robustness of the method in completing the
primary task. It can also be seen that in Case A the maximum
value of coupling angular momentum was 1414.6N m.s
(Experiment 5), where as the same for case B was 251.52
1.7 × 10
−4
Nm.s. This shows effectiveness of the proposed
approach in successful completion of the secondary task in
all the seven experiments. This fact is also evident from
the values of maximum change in the attitude of the base
satellite, which was limited to 0.5rad in Case A, whereas the
0 1 2 3 4 5
−50
0
50
100
150
200
Time(sec)
Coupling Angular Momentum (Nm.s)
Case A Case B
Fig. 10. Variation of secondary cost function h
s
=
˜
I
bm
˙
θ
