Keeping multiple objects in the field of view of a single PTZ camera
10 Jun 2009-pp 5259-5264
TL;DR: A set of task functions are developed to regulate the mean and variance of a set of image features to deter feature points from leaving the camera field-of-view.
Abstract: This paper introduces a novel visual servo controller designed to keep multiple moving objects in the camera field-of-view using a pan/tilt/zoom camera. In contrast to most visual servo controllers, there is no goal pose or goal image. In this paper, a set of task functions are developed to regulate the mean and variance of a set of image features. Regulating these task functions will deter feature points from leaving the camera field-of-view. An additional task function is used to maintain a high level of motion perceptibility, which ensures that desired feature point velocities can be achieved. To provide proper control of a pan/tilt/zoom camera, an image Jacobian is developed utilizing actuation of the focal length. Simulations of several object tracking tasks have verified the performance of the proposed method.
Citations
More filters
21 Oct 2013
TL;DR: This work presents a method to generate aesthetic video from a robotic camera by incorporating a virtual camera operating on a delay, and a hybrid controller which uses feedback from both the robotic and virtual cameras.
Abstract: We present a method to generate aesthetic video from a robotic camera by incorporating a virtual camera operating on a delay, and a hybrid controller which uses feedback from both the robotic and virtual cameras. Our strategy employs a robotic camera to follow a coarse region-of-interest identified by a realtime computer vision system, and then resamples the captured images to synthesize the video that would have been recorded along a smooth, aesthetic camera trajectory. The smooth motion trajectory is obtained by operating the virtual camera on a short delay so that perfect knowledge of immediate future events is known. Previous autonomous camera installations have employed either robotic cameras or stationary wide-angle cameras with subregion cropping. Robotic cameras track the subject using realtime sensor data, and regulate a smoothness-latency trade-off through control gains. Fixed cameras post-process the data and suffer significant reductions in image resolution when the subject moves freely over a large area.Our approach provides a solution for broadcasting events from locations where camera operators cannot easily access. We can also offer broadcasters additional actuated camera angles without the overhead of additional human operators. Experiments on our prototype system for college basketball illustrate how our approach better mimics human operators compared to traditional robotic control approaches, while avoiding the loss in resolution that occurs from fixed camera system.
74 citations
Cites methods from "Keeping multiple objects in the fie..."
...[10] use taskpriority kinematic control to keep a set of interest points within the camera field of view....
[...]
...As a result, we employ assumptions common in visual servoing [9, 10] and approxi-...
[...]
TL;DR: A novel visual servo controller that is designed to control the pose of the camera to keep multiple objects in the field of view (FOV) of a mobile camera and a proof of stability is presented for tracking three or fewer targets.
Abstract: This study introduces a novel visual servo controller that is designed to control the pose of the camera to keep multiple objects in the field of view (FOV) of a mobile camera. In contrast with other visual servo methods, the control objective is not formulated in terms of a goal pose or a goal image. Rather, a set of underdetermined task functions are developed to regulate the mean and variance of a set of image features. Regulating these task functions inhibits feature points from leaving the camera FOV. An additional task function is used to maintain a high level of motion perceptibility, which ensures that desired feature point velocities can be achieved. These task functions are mapped to camera velocity, which serves as the system input. A proof of stability is presented for tracking three or fewer targets. Experiments of tracking eight or more targets have verified the performance of the proposed method.
51 citations
Cites background or methods from "Keeping multiple objects in the fie..."
...Specifically, Chebyshev’s inequality proves that at least 75% of all values are within two standard deviations of the mean, and at least 89% of values are within three standard deviations....
[...]
...The camera is a Matrix Vision BlueFox, with a resolution of 1024 × 768 pixels....
[...]
TL;DR: A robust visual servo system for object tracking applications of a nonholonomic mobile robot using a Lyapunov-based stability analysis and a sliding mode control technique is presented.
Abstract: In this paper, we present the development of a robust visual servo system for object tracking applications of a nonholonomic mobile robot. The system mainly consists of an adaptive shape tracking algorithm and a robust visual servo controller. The adaptive shape tracking algorithm is designed to automatically detect the shape contours of moving objects, extract the shape parameters, and continuously track the object in shape parameter space. Based on direct measurements of the shape parameters, the visual servo controller is designed using the sliding mode control technique. Through a Lyapunov-based stability analysis, a sufficient condition on the selection of control gains to achieve the tracking goal in finite time is provided, and simulation and experimental tests of the proposed approach are illustrated.
39 citations
Cites methods from "Keeping multiple objects in the fie..."
...Recently, for tracking multiple objects, an image Jacobian was developed in [18] for controlling the desired feature points within the field of view of a single pan-tilt-zoom camera and a configuration-homography-based analysis is proposed in [19] for deriving related...
[...]
Proceedings Article•
01 Jan 2014TL;DR: This work reviews autonomous camera systems developed over the past twenty years and discusses a trend towards more datadriven approaches fueled by continuous improvements in underlying sensing and signal processing technology.
Abstract: Autonomous cameras allow live events, such as lectures and sports matches, to be broadcast to larger audiences. In this work, we review autonomous camera systems developed over the past twenty years. Quite often, these systems were demonstrated on scripted stage productions (typically cooking shows), lectures, or team sports. We organize the discussion in terms of three core tasks: (1) planning where the cameras should look, (2) controlling the cameras as they transition from one parameter configuration to another, and (3) selecting which camera to put “on-air” in multi-camera systems. We conclude by discussing a trend towards more datadriven approaches fueled by continuous improvements in underlying sensing and signal processing technology.
22 citations
Cites methods from "Keeping multiple objects in the fie..."
...Gans et al. (2009) used a task-priority kinematic controller to keep a set of interest points within the camera field of view....
[...]
TL;DR: New information value functions that are additive and can be optimized efficiently over time by deriving a lower bound of the KL divergence are presented and combined with a convex approximation of the sensor field of view can be used to obtain real-time sensor control by a lexicographic approach.
Abstract: Information value functions based on the Kullback–Leibler (KL) divergence have been shown the most effective for planning sensor measurements by means of greedy strategies. The problem of optimizing information value over a finite time horizon to date has been considered computationally intractable and, as proven here, is $\text{NP}$ -hard. This paper presents new information value functions that are additive and can be optimized efficiently over time by deriving a lower bound of the KL divergence. Combined with a convex approximation of the sensor field of view, these information value functions can be used to obtain real-time sensor control by a lexicographic approach, and to derive performance guarantees. Numerical and experimental results on pedestrian data show that the lexicographic control system significantly improves target modeling and prediction performance when compared to existing algorithms.
17 citations
Cites background or methods from "Keeping multiple objects in the fie..."
...Based on the pinhole camera model [24], the camera lens is symmetric about a so-called optical axis, and images of S are projected onto a two-dimensional (2-D) virtual image plane perpendicular to the optical axis and located at a distance λ from the pinhole [25], [26]....
[...]
...is the image Jacobian matrix, derived in [24]....
[...]
References
More filters
20 Sep 1999
TL;DR: Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds.
Abstract: An object recognition system has been developed that uses a new class of local image features. The features are invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection. These features share similar properties with neurons in inferior temporal cortex that are used for object recognition in primate vision. Features are efficiently detected through a staged filtering approach that identifies stable points in scale space. Image keys are created that allow for local geometric deformations by representing blurred image gradients in multiple orientation planes and at multiple scales. The keys are used as input to a nearest neighbor indexing method that identifies candidate object matches. Final verification of each match is achieved by finding a low residual least squares solution for the unknown model parameters. Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds.
16,989 citations
"Keeping multiple objects in the fie..." refers background in this paper
...The extraction of such points could come from a variety of algorithms, such as tracking distinguishable points on moving targets [10], [11], the centroid of segmented blobs [12] or the corner points of bounding rectangle of tracked moving targets [13]....
[...]
21 Jun 1994
TL;DR: A feature selection criterion that is optimal by construction because it is based on how the tracker works, and a feature monitoring method that can detect occlusions, disocclusions, and features that do not correspond to points in the world are proposed.
Abstract: No feature-based vision system can work unless good features can be identified and tracked from frame to frame. Although tracking itself is by and large a solved problem, selecting features that can be tracked well and correspond to physical points in the world is still hard. We propose a feature selection criterion that is optimal by construction because it is based on how the tracker works, and a feature monitoring method that can detect occlusions, disocclusions, and features that do not correspond to points in the world. These methods are based on a new tracking algorithm that extends previous Newton-Raphson style search methods to work under affine image transformations. We test performance with several simulations and experiments. >
8,432 citations
"Keeping multiple objects in the fie..." refers background in this paper
...The extraction of such points could come from a variety of algorithms, such as tracking distinguishable points on moving targets [10], [11], the centroid of segmented blobs [12] or the corner points of bounding rectangle of tracked moving targets [13]....
[...]
01 Oct 1996
TL;DR: This article provides a tutorial introduction to visual servo control of robotic manipulators by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process.
Abstract: This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.
3,619 citations
"Keeping multiple objects in the fie..." refers background or methods in this paper
...The feature point depths must be known or accurately estimated for six-DOF IBVS, and IBVS is brittle to depth estimation errors [14]....
[...]
...The method is rooted in classic IBVS [1]–[4], however there is no goal image or goal feature trajectory....
[...]
...There are many approaches, including image based visual servoing (IBVS), position based visual servoing, partitioned methods and switching methods....
[...]
...The first two columns of Lλ(t) are the same as the fourth and fifth columns of the classic six -DOF image Jacobian [1], [3]....
[...]
...The time-varying image feature velocity is mapped to camera motions through IBVS methods....
[...]
30 Nov 2006
TL;DR: This paper is the first of a two-part series on the topic of visual servo control using computer vision data in the servo loop to control the motion of a robot using basic techniques that are by now well established in the field.
Abstract: This paper is the first of a two-part series on the topic of visual servo control using computer vision data in the servo loop to control the motion of a robot. In this paper, we describe the basic techniques that are by now well established in the field. We first give a general overview of the formulation of the visual servo control problem. We then describe the two archetypal visual servo control schemes: image-based and position-based visual servo control. Finally, we discuss performance and stability issues that pertain to these two schemes, motivating the second article in the series, in which we consider advanced techniques
2,026 citations
01 Jun 1992
TL;DR: Vision-based control in robotics based on considering a vision system as a specific sensor dedicated to a task and included in a control servo loop is described, and stability and robustness questions arise.
Abstract: Vision-based control in robotics based on considering a vision system as a specific sensor dedicated to a task and included in a control servo loop is described. Once the necessary modeling stage is performed, the framework becomes one of automatic control, and stability and robustness questions arise. State-of-the-art visual servoing is reviewed, and the basic concepts for modeling the concerned interactions are given. The interaction screw is thus defined in a general way, and the application to images follows. Starting from the concept of task function, the general framework of the control is described, and stability results are recalled. The concept of the hybrid task is presented and then applied to visual sensors. Simulation and experimental results are presented, and guidelines for future work are drawn in the conclusion. >
1,463 citations