In one embodiment of the invention, a method for a robotic system is disclosed to track one or more robotic instruments. The method includes generating kinematics information for the robotic instrument within a field of view of a camera; capturing image information in the field of view of the camera; and adaptively fusing the kinematics information and the image information together to determine pose information of the robotic instrument. Additionally disclosed is a robotic medical system with a tool tracking sub-system. The tool tracking sub-system receives raw kinematics information and video image information of the robotic instrument to generate corrected kinematics information for the robotic instrument by adaptively fusing the raw kinematics information and the video image information together.

Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information

An attempt to achieve a high level of interaction between a real-time vision system capable of tracking moving objects in 3-D and a robot arm with gripper that can be used to pick up a moving object is described. The interplay of hand-eye coordination in dynamic grasping tasks such as grasping of parts on a moving conveyor system, assembly of articulated parts, or for grasping from a mobile robotic system is explored. The goal is to build an integrated sensing and actuation system that can operate in dynamic as opposed to static environments. The system built addresses three distinct problems in using robotic hand-eye coordination for grasping moving objects: fast computation of 3-D motion parameters from vision, predictive control of a moving robotic arm to track a moving object, and interception and grasping. The system operates at approximately human arm movement rates. Experimental results in which a moving model train is tracked, stably grasped, and picked up by the system are presented. The algorithms developed to relate sensing to actuation are quite general and applicable to a variety of complex robotic tasks. >

/pdf/automated-tracking-and-grasping-of-a-moving-object-with-a-4gi8l4udod.pdf

Automated tracking and grasping of a moving object with a robotic hand-eye system

This paper attempts to present a comprehensive summary of research results in the use of visual information to control robot manipulators and related mechanisms. An extensive bibliography is provided which also includes important papers from the elemental disciplines upon which visual servoing is based. The research results are discussed in terms of historical context, common-ality of function, algorithmic approach and method of implementation.

Visual control of robot manipulators – a review

The use of a vision sensor in the feedback loop is addressed within the controlled active vision framework. Algorithms are proposed for the solution of the robotic (eye-in-hand configuration) visual tracking and servoing problem. Visual tracking is stated as a problem of combining control with computer vision. The sum-of-squared differences optical flow is used to compute the vector of discrete displacements. The displacements are fed to an adaptive controller (self-tuning regulator) that creates commands for a robot control system. The procedure is based on the online estimation of the relative distance of the target from the camera, but only partial knowledge of the relative distance is required, obviating the need for offline calibration. Three different adaptive control schemes have been implemented, both in simulation and in experiments. The computational complexity and the experimental results demonstrate that the proposed algorithms can be implemented in real time. >

Adaptive robotic visual tracking: theory and experiments

/pdf/design-of-dynamically-reconfigurable-red-time-software-using-st5mym2mv3.pdf

Design of Dynamically Reconfigurable Red-Time Software using Port-Based Objects

Algorithms for robotic real-time visual tracking of arbitrary 3-D objects traveling at unknown velocities in a 2-D space are presented. The problem of visual tracking is formulated as a problem of combining control with computer vision. A mathematical formulation that is general enough to be extended to the problem of tracking 3-D objects in 3-D space is presented. The authors propose the use of sum-of-squared differences optical flow for the computation of the vector of discrete displacements each instant of time. These displacements can be fed either directly to a PI controller, a pole assignment controller, or a discrete steady-state Kalman filter. In the latter case, the Kalman filter calculates the estimated values of the system's states and exogenous disturbances, and a discrete LQG controller computes the desired motion of the robotic system. The outputs of the controllers are sent to a Cartesian robotic controller that drives the robot. >

T. Kanada

Papers

Vision and control techniques for robotic visual tracking