Uncalibrated Visual Servo for Unmanned Aerial Manipulation
Summary (4 min read)
Introduction
- Unmanned aerial vehicles (UAVs), and in particular multirotor systems, have substantially gained popularity in recent years, motivated by their significant increase in maneuverability, together with a decrease in weight and cost [1].
- In addition, IBVS is more robust than PBVS with respect to uncertainties and disturbances affecting the model of the robot, as well as the calibration of the camera [4], [5].
- In all image-based and hybrid approaches the resulting image Jacobian or interaction matrix, which relates the camera velocity to the image feature velocities, depends on a priori knowledge of the intrinsic camera parameters.
- The second contribution is the proposal of a hierarchical control law that exploits the extra degrees of freedom of the UAV-arm system which, in contrast to their previous solution [23], uses a less restrictive control law that only actuates on the components of the secondary tasks that do not conflict directly with tasks higher up in the hierarchy.
- The next section presents their uncalibrated approach to visual servo.
II. UNCALIBRATED IMAGE-BASED VISUAL SERVOING
- Drawing inspiration on the UPnP algorithm [25], the authors describe in the following subsection a method to solve for the camera pose and focal length using a reference system attached to the target object.
- The method is extended in Sec. II-B to compute a calibration-free image Jacobian for their servo task, and in Sec. II-C to compute the desired control law.
A. Uncalibrated PnP
- 3D target features are parameterized with their barycentric coordinates, and the basis of these coordinates is used to define a set of control points.
- A least squares solution for the control point coordinates albeit scale, is given by the null eigenvector of a linear system made up of all 2D to 3D perspective projection relations between the target points.
- The terms aij are the barycentric coordinates of the i-th target feature which are constant regardless of the location of the camera reference frame, and α is their unknown focal length.
- In the noise-free case, M>M is only rank deficient by one, but when image noise is severe it might loose rank, and a more accurate solution can be found as a linear combination of the basis of its null space.
- It is sufficient for their purposes to consider only the least squares approximation; that is, to compute the solution only using the eigenvector associated to the smallest eigenvalue.
A. Coordinate Systems
- Consider the quadrotor-arm system equipped with a camera mounted at the end-effector’s arm as shown in Fig.
- The position of the camera (c) with respect to the target frame, expressed as a homogeneous transform Twc , can be computed integrating the camera velocities obtained from the uncalibrated visual servo approach presented in the previous section.
- A quadrotor is at the high level of control an underactuated vehicle with only 4 DoF, namely the linear velocities plus the yaw angular velocity (νqx, νqy, νqz, ωqz) acting on the body frame.
- And at the low level, the attitude controller stabilizes horizontally the quadrotor body.
- With the arm base frame coincident with the quadrotor body frame, the relation between the quadrotor body and camera frames is Tbc = T b t(qa) T t c, with T b t(qa) the arm kinematics and Ttc the tool-camera transform.
B. Robot Kinematics
- Combining Eqs. 9 and 10 the authors can relate the desired highlevel control velocities with their visual servo task, which they term now σS Jqavqa = −ΛS J+vse︸︷︷︸ σS . (13) Unfortunately as said before, the quadrotor is an underactuated vehicle.
- So, to remove the non-controllable variables from the control command, their contribution to the image error can be isolated from that of the controllable ones by extracting the columns of Jqa and the rows of vqa corresponding to ωqx and ωqy , reading out these values from the platform gyroscopes, and subtracting them from the camera velocity [26].
C. Motion Distribution
- In order to penalize the motion of the quadrotor vs the arm to account for their different motion capabilities, the authors can define a weighted norm of the whole velocity vector ‖q̇‖W = √ q̇>.
- Large movements should be achieved by the quadrotor whereas the precise movements should be devoted to the robotic arm due to its dexterity when the platform is close to the target.
- The blocks of W weight differently the velocity components of the arm and the quadrotor by increasing the velocity of the quadrotor when the distance to the target d > ∆W , while for distances d < δW the quadrotor is slowed down and the arm is commanded to accommodate for the precise movements.
A. Hierarchical Task Composition
- Exploiting this redundancy, the authors can achieve additional tasks acting on the null space of the quadrotor-arm Jacobian [29], while preserving the primary task.
- These tasks can be used to reconfigure the robot structure without changing the position and orientation of the arm end-effector.
- Multiple secondary tasks can be arranged in hierarchy and, to avoid conservative stability conditions [31], the augmented inversebased projections method is here considered [21].
- Lower priority tasks are not only projected onto the null space of the task up in the hierarchy, but onto the null space of an augmented Jacobian with all higher priority tasks.
- In Section III-B the authors showed how to compute a visual servo control law that takes into account the uncontrollable state variables.
B. Stability analysis
- To assess the stability of each i-th individual task, the authors use Lyapunov analysis by considering the positive definite candidate Lyapunov function L = 12 ‖σi(t)‖.
- Notice how the secondary task does not affect the dynamics of the main task thanks to the null space projector, hence the stability of the main task is again achieved.
- The previous stability analysis can be straightforwardly extended to the general case of η subtasks.
C. Task Order
- And lower in the hierarchy, the alignment of the center of gravity of the UAM (G), and a technique to stay away from the arm’s joint limits (L).the authors.
- Notice that the safety task Jacobian pseudo-inverse J#I is also weighted and how in the null space projectors NI|S and NI|S|G from Eq. 31, the involved pseudo-inverses do not need to be weighted because the center of gravity alignment and the joint limits avoidance involve only arm movements and also should be accomplished during the flight.
- The authors now give more detailed descriptions of task Jacobians and task errors involved.
D. Collision Avoidance
- The most important task during a mission is to preserve flight safety.
- When a rotor operates near an obstacle, different aerodynamic effects are revealed, such as the so called "ground" or "ceiling" effects, that can lead to an accident.
- Hence, to avoid them, the authors propose a task with the highest priority to maintain a safety distance to obstacles by defining a safety sphere around the flying platform, and comparing the Euclidean distance to the obstacle (do) with the sphere radius (rI ).
- Note that this corresponds to a proportional control law although integral or derivative errors could also be considered.
- This is usually called joint clamping (JC).
E. Center of Gravity
- If the arm and quadrotor center of gravity (CoG) are not vertically aligned, the motion of the arm produces an undesired torque on the quadrotor base that perturbs the system attitude and position.
- This effect can be mitigated by minimizing the distance between the arm CoG and the vertical line of the quadrotor gravity vector.
- The position of the arm CoG pbG is a function of the arm joint configuration defined as pbG = ∑ν i=1 mi p b Gi∑ν i=1 , (37) where mi and pbGi are the mass and the position of the CoG of link i.
- The authors can compute the arm CoG with respect to the body frame for the sequence of links j to the end-effector with p∗bGj = R b j ∑ν i=j mi p b Gi∑ν i=j , (38) where Rbj is the rotation between link j and the body frame.
- Notice that all these quantities are a function of the current joint configuration qa.
B. UAM system
- To demonstrate the proposed hierarchical task composition the authors designed and built a lightweight robotic arm with a joint setting to compensate the possible noise existing in the quadrotor positioning while hovering, and to avoid self collisions during take off and landing maneuvers.
- To address the dynamical effects of the overall system their cascaded architecture considers two different control loops at very high frequency (1KHz), one for the arm and one for the attitude of the UAV; and a hierarchical task controller running at much lower frequency (camera frame rate), hence avoiding dynamic coupling between them.
- Then the authors use the inter-distance constraints to solve for scale and focal length.
- When only the visual servoing (S) is executed, the time to reach the target is significantly higher than those cases in which the arm CoG is vertically aligned (S+G and S+G+L).
VI. REAL ROBOT EXPERIMENTS
- The authors conducted a series of experiments with missions similar to those shown in simulations, i.e. autonomously taking off and flying to a location in which the target appears in the field of view of the camera, turning on then the hierarchical task controller to servo the system towards a desired camera pose, and finally autonomously landing the system.
- The last task is designed to favor a desired arm configuration and it can be used to push the joints away from singularities and potentially increase maneuverability.
- It does not imply that the subtask can always be fulfilled.
VII. CONCLUSIONS
- The authors have presented an uncalibrated image-based visual servo scheme for manipulation UAVs.
- The authors have presented a control law to achieve not only the visual servoing but also other tasks taking into account their specific priorities.
- Moreover, the presented control law only requires independent tasks for the uncontrollable variables to guarantee exponentially stability of the system.
- The technique is demonstrated using Matlab and ROS in both simulation and a real UAM.
- The authors can think of two avenues for further research.
Did you find this useful? Give us your feedback
Citations
167 citations
Cites background from "Uncalibrated Visual Servo for Unman..."
...In [19] the feedback output from a camera attached to the end effector is adopted in a hierarchical control law....
[...]
144 citations
94 citations
Cites methods from "Uncalibrated Visual Servo for Unman..."
...In [8], an uncalibrated image-based visual servo scheme was given for the manipulation of unmanned aerial vehicles....
[...]
81 citations
65 citations
References
380 citations
"Uncalibrated Visual Servo for Unman..." refers methods in this paper
...To assess the stability of each ith individual task, we use Lyapunov analysis by considering the positive definite candidate Lyapunov function L = 12 ‖σi(t)‖2 and its derivative L̇ = σTi σ̇i ....
[...]
...This is done for instance, using Gauss–Newton to minimize the squared image error and nonlinear least squares optimization for the image Jacobian [7]; using weighted recursive least squares, not to obtain the true parameters, but instead an approximation that still guarantees asymptotic stability of the control law in the sense of Lyapunov [8], [9]; using k-nearest neighbor regression to store previously estimated local models or previous movements, and estimating the Jacobian using local least squares [10], or building a secant model using the population of the previous iterates [11]....
[...]
...eters, but instead an approximation that still guarantees asymptotic stability of the control law in the sense of Lyapunov [8], [9]; using k-nearest neighbor regression to store previously estimated local models or previous movements, and estimating the Jacobian using local least squares [10], or building a secant model using the population of the previous iterates [11]....
[...]
256 citations
"Uncalibrated Visual Servo for Unman..." refers background in this paper
...Flying with a suspended load is a challenging task and it is essential to have the ability to minimize the undesired effects of the arm in the flying system [15]....
[...]
236 citations
Additional excerpts
...Multiple secondary tasks can be arranged in hierarchy and, to avoid conservative stability conditions [31], the augmented inverse-based projections method is here considered [21]....
[...]
214 citations
Additional excerpts
...Multiple secondary tasks can be arranged in hierarchy and, to avoid conservative stability conditions [31], the augmented inverse-based projections method is here considered [21]....
[...]
145 citations
"Uncalibrated Visual Servo for Unman..." refers background in this paper
...Vision-based robot control systems are usually classified in three groups: position-based visual servo (PBVS), image-based visual servo (IBVS), and hybrid control systems [2], [3]....
[...]
Related Papers (5)
Frequently Asked Questions (15)
Q2. What have the authors stated for future works in "Uncalibrated visual servo for unmanned aerial manipulation" ?
The authors can think of two avenues for further research. On the one hand, the activation and deactivation of the safety task as well as a dynamic exchange of task priority roles can induce some chattering phenomena, which can be avoided by introducing a hysteresis scheme. Secondly, the dimensionality of the subspace associated to each null space projector is a necessary condition to be considered when designing subtasks, however it might not be sufficient to guarantee the fulfilment of the subtask and a thorough analytical study of these spaces can be required.
Q3. How many tasks can be added to the hierarchy?
The addition of more tasks in cascade is possible as long as there exist remaining DoF from the concatenation of tasks higher up in the hierarchy.
Q4. How many DoF can be used for the visual servo mission task?
The visual servo mission task requires 6 DoF, and the secondary and comfort tasks with lower priority can take advantage of the remaining 4 DoF.
Q5. What is the primary task of the servoing algorithm?
1) Primary task: Among all other tasks, the one with the highest priority must be the safety task, not to compromise the platform integrity.
Q6. How many DoF are required for the joint limits avoidance task?
The gravitational vector alignment task and the joint limits avoidance task require 1 DoF each being scalar cost functions to minimize (see Eq. 35 and 43).
Q7. what is the qa of the i-th joint?
The desired task variable is σ∗L = 0 (i.e. σ̃L = −σL), while the corresponding task Jacobian isJL = [ 01×4 −2 (ΛL (qa − q∗a))> ] . (45)One common choice of q∗a for the joint limit avoidance is the middle of the joint limit ranges (if this configuration is far from kinematic singularities), q∗a = qa + 1 2 (qa − qa).
Q8. What is the simplest way to evaluate the stability of the system?
Finally the dynamics of the system can be written as[ σ̇0 σ̇1 ] = [ −Λ0 O −J1J+0 Λ0 −Λ1 ] [ σ0 σ1 ] , (29)which is characterized by a Hurwitz matrix as in [23] that guarantees the exponential stability of the system.
Q9. What is the way to guarantee stability of the control law?
This guarantees asymptotic stability of the control law regardless of the target point selection, as long as planar configurations are avoided.
Q10. What is the main task of the servoing algorithm?
When the obstacle does not violate the inflation radius, the safety task becomes deactivated and the other subtasks can regain access to the previously blocked DoF. Fig. 3(a) shows how the servoing task is elusive during the first 10 seconds of the simulation when the obstacle is present, but is accomplished afterwards when the obstacle is no longer an impediment to the secondary task.
Q11. what is the velocity vector of the quadrotor in the camera frame?
Ja q̇a = R c b Ja q̇a, (11)with Rcb the rotation matrix of the body frame with respect to the camera frame, and where vcq corresponds to the velocity of the quadrotor expressed in the camera framevcq = R c b[ νbq + ω b q × rbcωbq] = [ Rcb −Rcb [ rbc ] ×0 Rcb] vbq, (12)with rbc(qa) the distance vector between the body and camera frames, and vbq = [νqx, νqy, νqz, ωqx, ωqy, ωqz]> the velocity vector of the quadrotor in the body frame.
Q12. What is the simplest way to evaluate the stability of a task?
for the primary task the authors can substitute Eq. 21 into Eq. 20, giving σ̇0 = Λ0σ̃0, which for a defined main task error σ̃0 = σ∗0 − σ0 and σ∗0 = 0, the asymptotic stability is proven with L̇ = −σT0 Λ0σ0.
Q13. What is the generalization of Eq. 23 to the case of i-th sub?
By denoting with JI , JS , JG and JL theJacobian matrices of the above-mentioned tasks, the desired system velocity can be written as followsq̇ = J#I σ̃I + NI J # S ΛSσ̃S + NI|S J + G σ̃G+NI|S|G J + L σ̃L − JI|S|G|L$, (30)where NI , NI|S , NI|S|G are the projectors of the safety, the visual servoing and of the center of gravity tasks, which are defined asNI = (I− J#I JI) (31a) NI|S = (I− J+I|S JI|S) (31b) NI|S|G = (I− J+I|S|G JI|S|G) , (31c)with JI|S and JI|S|G the augmented Jacobians computed as in Eq. 27.
Q14. what is the function of the i-th joint?
The sum of normalized distances of the position of the i-th joint to its desired configuration is given bym∑ i=1 ( qai − q∗ai qai − qai )2 . (42)So their task function is selected as the squared distance of the whole arm joint configuration with respect to the desired oneσL = (qa − q∗a)>ΛL (qa − q∗a), (43)where qa = [ qa1, . . . , qam ]> and q a = [ q a1 , . . . , q am ]> are the high and low joint-limit vectors respectively, and ΛL is a diagonal matrix whose diagonal elements are equal to the inverse of the squared joint limit rangesΛL = diag((qa1 − qa1) −2, . . . , (qam − qam) −2). (44)
Q15. what is the generalization of eq. 23 to the case of prioritized?
The generalization of Eq. 23 to the case of η prioritized subtasks isq̇ = J+0 Λ0σ̃0 + η∑ i=1 N0|...|i−1J + i Λiσ̃i − J0|...|η$ (24)with the recursively-defined compensating matrixJ0|...|η = N0|...|i−1J + i Ji +