In this paper we provide a broad survey of developments in active vision in robotic applications over the last 15 years. With increasing demand for robotic automation, research in this area has received much attention. Among the many factors that can be attributed to a high-performance robotic system, the planned sensing or acquisition of perceptions on the operating environment is a crucial component. The aim of sensor planning is to determine the pose and settings of vision sensors for undertaking a vision-based task that usually requires obtaining multiple views of the object to be manipulated. Planning for robot vision is a complex problem for an active system due to its sensing uncertainty and environmental uncertainty. This paper describes such problems arising from many applications, e.g. object recognition and modeling, site reconstruction and inspection, surveillance, tracking and search, as well as robotic manipulation and assembly, localization and mapping, navigation and exploration. A bundle of solutions and methods have been proposed to solve these problems in the past. They are summarized in this review while enabling readers to easily refer solution methods for practical applications. Representative contributions, their evaluations, analyses, and future research trends are also addressed in an abstract level.

Active vision in robotic systems: A survey of recent developments

In this paper, we present the vision-aided inertial navigation (VISINAV) algorithm that enables precision planetary landing. The vision front-end of the VISINAV system extracts 2-D-to-3-D correspondences between descent images and a surface map (mapped landmarks), as well as 2-D-to-2-D feature tracks through a sequence of descent images (opportunistic features). An extended Kalman filter (EKF) tightly integrates both types of visual feature observations with measurements from an inertial measurement unit. The filter computes accurate estimates of the lander's terrain-relative position, attitude, and velocity, in a resource-adaptive and hence real-time capable fashion. In addition to the technical analysis of the algorithm, the paper presents validation results from a sounding-rocket test flight, showing estimation errors of only 0.16 m/s for velocity and 6.4 m for position at touchdown. These results vastly improve current state of the art for terminal descent navigation without visual updates, and meet the requirements of future planetary exploration missions.

/pdf/vision-aided-inertial-navigation-for-spacecraft-entry-16y1solbjk.pdf

Vision-Aided Inertial Navigation for Spacecraft Entry, Descent, and Landing

Vision guided robotics has been one of the major research issue for more than three decades. The more recent technological development facilitated the advancement in the area which has resulted in a number of successful and even commercial systems using off–the–shelf hardware. The applications of visually guided systems are many: from intelligent homes to automotive industry. However, one of the open and commonly stated problems in the area is the need for exchange of experiences and research ideas. In our opinion, a good starting point for this is to advertise the successes and propose a common terminology in form of a survey paper. The paper concentrates on different types of visual servoing: image based, position based and 2 1/2D visual servoing. Different issues concerning both the hardware and software requirements are considered and the most prominent contributions are reviewed. The proposed terminology is used to introduce a young researcher and lead the experts in the field through a three decades long historical field of vision guided robotics. We also include a number of real–world examples from our own research providing not only a conceptual framework but also illustrating most of the issues covered in the paper.

Survey on Visual Servoing for Manipulation

This paper deals with the problem of position-based visual servoing in a multiarm robotic cell equipped with a hybrid eye-in-hand/eye-to-hand multicamera system. The proposed approach is based on the real-time estimation of the pose of a target object by using the extended Kalman filter. The data provided by all the cameras are selected by a suitable algorithm on the basis of the prediction of the object self-occlusions, as well as of the mutual occlusions caused by the robot links and tools. Only an optimal subset of image features is considered for feature extraction, thus ensuring high estimation accuracy with a computational cost independent of the number of cameras. A salient feature of the paper is the implementation of the proposed approach to the case of a robotic cell composed of two industrial robot manipulators. Two different case studies are presented to test the effectiveness of the hybrid camera configuration and the robustness of the visual servoing algorithm with respect to the occurrence of occlusions

Position-Based Visual Servoing in Industrial Multirobot Cells Using a Hybrid Camera Configuration

Recent vision-based control algorithms for robot manipulators are reviewed. Vision is essential for robots working in an unstructured environment. There are numerous research results in vision-based control. Thus, the scope of this review is limited to providing a brief description on visual servo schemes for manipulators. First, a control architecture for vision-based control is reviewed and some basic ideas for visual servo control are presented. On the basis of these ideas recent research results, especially robust and globally stable controllers, are discussed in detail.

A review on vision-based control of robot manipulators

The use of a camera in a robot control loop can be performed with two types of architecture: for eye-in-hand camera when it rigidly mounted on the robot end-effector; and for eye-to-hand camera when it observes the robot within its work space. These two schemes have technical differences and they can play very complementary parts. Obviously, the eye-in-hand one has a partial but precise sight of the scene whereas the eye-to-hand camera has a less precise but global sight of it. The motivation of our work is to take advantage of both, free-standing and robot-mounted sensors, in a cooperation scheme. The system presented performs two separate tasks: a positioning task to perform in the global image, and a tracking task to perform in the local image. For robustness considerations, the control law stability is proved and several cooperative schemes are studied and compared in experimental results.

/pdf/eye-in-hand-eye-to-hand-cooperation-for-visual-servoing-1u40ltgsak.pdf

Eye-in-hand/eye-to-hand cooperation for visual servoing

Mission planning for a constellation of Earth observation satellites is a complex problem raising significant technological challenges for tomorrow's space systems. The large numbers of customers requests and their dynamic introduction in the planning system result in a huge combinatorial search space with a potentially highly dynamical evolution requirements. The techniques used nowadays have several limitations, in particular, it is impossible to dynamically adapt the plan during its construction even for small modifications. Satellites of a constellation are planned in a chronological way instead of a more collective planning which can provide additional load balancing. In this paper, we propose to solve this difficult and highly dynamic problem using adaptive multi-agent systems, taking advantage from their self-adaptation and self-organization mechanisms. In the proposed system, the agents, through their local interactions, allow to dynamically reach a good solution, while ensuring a controlled distribution of tasks within the constellation of satellites. Finally, a comparison with a classical chronological greedy algorithm, commonly used in the spatial domain, highlights the advantages of the presented system.

https://hal.archives-ouvertes.fr/hal-01665196/document

Multi-satellite Mission Planning Using a Self-Adaptive Multi-agent System

Autonomous navigation is a key enabling technology for future planetary exploration mission. For planetary Landers, vision based navigation concepts, working on complex, unstructured scenes, appear as the most promising while in the same time the most challenging technology. Working on concepts previously proposed for lunar landers, the “Navigation for Planetary Approach and Landing” (or NPAL in the following) ESA study goes one step further in securing two critical components of a vision navigation concept. First the on-line processing of the video stream for extracting the necessary observable for the navigation filtering is designed and an implementation on a dedicated integrated circuit –the FEICis realised. Second the camera itself is bread boarded to support the end-toend demonstration. The study accompanies the system design effort to propose a Lander in the European initiative for Mercury exploration, the BepiColombo project. This paper introduces the context and the objective of the study, the industrial organisation, and present work progress at mid-course. A key achievement is in the selection and definition of the algorithms to be implemented in the FEIC.

Navigation for Planetary Approach and Landing

Robust H?/H? thruster failure detection and isolation with application to the LISA Pathfinder spacecraft

The paper presents a concept of a smart pushbroom imaging system with compensation of attitude instability effects. The compensation is performed by active opto-mechatronic stabilization of the focal plane image motion in a closed loop system with visual feedback on base of an auxiliary matrix image sensor and an onboard optical correlator. In this way the effects of the attitude instability, vibrations and micro shocks can be neutralized, the image quality improved and the requirements to satellite attitude stability reduced. To prove the feasibility and to estimate the effectiveness of the image motion stabilization, a performance model of the smart imaging system has been developed and a simulation experiment has been carried out. The description of the performance model and the results of the simulation experiment are also given.

Grégory Flandin

Papers

Eye-in-hand/eye-to-hand cooperation for visual servoing

Multi-satellite Mission Planning Using a Self-Adaptive Multi-agent System

Navigation for Planetary Approach and Landing

Robust H?/H? thruster failure detection and isolation with application to the LISA Pathfinder spacecraft

Compensation of focal plane image motion perturbations with optical correlator in feedback loop