Computer vision is moving into a new era in which the aim is to develop visual skills for robots that allow them to interact with a dynamic, unconstrained environment. To achieve this aim, new kinds of vision algorithms need to be developed which run in real time and subserve the robot's goals. Two fundamental goals are determining the identity of an object with a known location, and determining the location of a known object. Color can be successfully used for both tasks.
This dissertation demonstrates that color histograms of multicolored objects provide a robust, efficient cue for indexing into a large database of models. It shows that color histograms are stable object representations in the presence of occlusion and over change in view, and that they can differentiate among a large number of objects. For solving the identification problem, it introduces a technique called Histogram Intersection, which matches model and image histograms and a fast incremental version of Histogram Intersection which allows real-time indexing into a large database of stored models. It demonstrates techniques for dealing with crowded scenes and with models with similar color signatures. For solving the location problem it introduces an algorithm called Histogram Backprojection which performs this task efficiently in crowded scenes.

Color indexing

Humans perceive the three-dimensional structure of the world with apparent ease. However, despite all of the recent advances in computer vision research, the dream of having a computer interpret an image at the same level as a two-year old remains elusive. Why is computer vision such a challenging problem and what is the current state of the art? Computer Vision: Algorithms and Applications explores the variety of techniques commonly used to analyze and interpret images. It also describes challenging real-world applications where vision is being successfully used, both for specialized applications such as medical imaging, and for fun, consumer-level tasks such as image editing and stitching, which students can apply to their own personal photos and videos. More than just a source of recipes, this exceptionally authoritative and comprehensive textbook/reference also takes a scientific approach to basic vision problems, formulating physical models of the imaging process before inverting them to produce descriptions of a scene. These problems are also analyzed using statistical models and solved using rigorous engineering techniques Topics and features: structured to support active curricula and project-oriented courses, with tips in the Introduction for using the book in a variety of customized courses; presents exercises at the end of each chapter with a heavy emphasis on testing algorithms and containing numerous suggestions for small mid-term projects; provides additional material and more detailed mathematical topics in the Appendices, which cover linear algebra, numerical techniques, and Bayesian estimation theory; suggests additional reading at the end of each chapter, including the latest research in each sub-field, in addition to a full Bibliography at the end of the book; supplies supplementary course material for students at the associated website, http://szeliski.org/Book/. Suitable for an upper-level undergraduate or graduate-level course in computer science or engineering, this textbook focuses on basic techniques that work under real-world conditions and encourages students to push their creative boundaries. Its design and exposition also make it eminently suitable as a unique reference to the fundamental techniques and current research literature in computer vision.

/pdf/computer-vision-algorithms-and-applications-25dn6wu83j.pdf

Computer Vision: Algorithms and Applications

This paper discusses a new method for capturing the complete appearance of both synthetic and real world objects and scenes, representing this information, and then using this representation to render images of the object from new camera positions. Unlike the shape capture process traditionally used in computer vision and the rendering process traditionally used in computer graphics, our approach does not rely on geometric representations. Instead we sample and reconstruct a 4D function, which we call a Lumigraph. The Lumigraph is a subset of the complete plenoptic function that describes the flow of light at all positions in all directions. With the Lumigraph, new images of the object can be generated very quickly, independent of the geometric or illumination complexity of the scene or object. The paper discusses a complete working system including the capture of samples, the construction of the Lumigraph, and the subsequent rendering of images from this new representation.

The lumigraph

We present a novel statistical and variational approach to image segmentation based on a new algorithm, named region competition. This algorithm is derived by minimizing a generalized Bayes/minimum description length (MDL) criterion using the variational principle. The algorithm is guaranteed to converge to a local minimum and combines aspects of snakes/balloons and region growing. The classic snakes/balloons and region growing algorithms can be directly derived from our approach. We provide theoretical analysis of region competition including accuracy of boundary location, criteria for initial conditions, and the relationship to edge detection using filters. It is straightforward to generalize the algorithm to multiband segmentation and we demonstrate it on gray level images, color images and texture images. The novel color model allows us to eliminate intensity gradients and shadows, thereby obtaining segmentation based on the albedos of objects. It also helps detect highlight regions.

Region competition: unifying snakes, region growing, and Bayes/MDL for multiband image segmentation

Image category recognition is important to access visual information on the level of objects and scene types. So far, intensity-based descriptors have been widely used for feature extraction at salient points. To increase illumination invariance and discriminative power, color descriptors have been proposed. Because many different descriptors exist, a structured overview is required of color invariant descriptors in the context of image category recognition. Therefore, this paper studies the invariance properties and the distinctiveness of color descriptors (software to compute the color descriptors from this paper is available from http://www.colordescriptors.com) in a structured way. The analytical invariance properties of color descriptors are explored, using a taxonomy based on invariance properties with respect to photometric transformations, and tested experimentally using a data set with known illumination conditions. In addition, the distinctiveness of color descriptors is assessed experimentally using two benchmarks, one from the image domain and one from the video domain. From the theoretical and experimental results, it can be derived that invariance to light intensity changes and light color changes affects category recognition. The results further reveal that, for light intensity shifts, the usefulness of invariance is category-specific. Overall, when choosing a single descriptor and no prior knowledge about the data set and object and scene categories is available, the OpponentSIFT is recommended. Furthermore, a combined set of color descriptors outperforms intensity-based SIFT and improves category recognition by 8 percent on the PASCAL VOC 2007 and by 7 percent on the Mediamill Challenge.

Evaluating Color Descriptors for Object and Scene Recognition

In computer vision, the goal of which is to identify objects and their positions by examining images, one of the key steps is computing the surface normal of the visible surface at each point (“pixel”) in the image. Many sources of information are studied, such as outlines ofsuifaces, intensity gradients, object motion, and color. This article presents a method for analyzing a standard color image to determine the amount of interface (“specular”) and body (“diffuse”) reflection at each pixel. The interface reflection represents the highlights from the original image, and the body reflection represents the original image with highlights removed. Such intrinsic images are of interest because the geometric properties of each type of reflection are simpler than the geometric properties of intensity in a black-and-white image. The method is based upon a physical model of reflection which states that two distinct types of reflection–interface and body reflection–occur, and that each type can be decomposed into a relative spectral distribution and a geometric scale factor. This model is far more general than typical models used in computer vision and computer graphics, and includes most such models as special cases. In addition, the model does not assume a point light source or uniform illumination distribution over the scene. The properties of tristimulus integration are used to derive a new model of pixel-value color distribution, and this model is exploited in an algorithm to derive the desired quantities. Suggestions are provided for extending the model to deal with diffuse illumination and for analyzing the two components of reflection.

/pdf/using-color-to-separate-reflection-components-2nmip8zh33.pdf

Using color to separate reflection components

A distributed architecture articulated around the CODGER (communication database with geometric reasoning) knowledge database is described for a mobile robot system that includes both perception and navigation tools. Results are described for vision and navigation tests using a mobile testbed that integrates perception and navigation capabilities that are based on two types of vision algorithms: color vision for road following, and 3-D vision for obstacle detection and avoidance. The perception modules are integrated into a system that allows the vehicle to drive continuously in an actual outdoor environment. The resulting system is able to navigate continuously on roads while avoiding obstacles. >

Vision and navigation for the Carnegie-Mellon Navlab

Vision and Navigation for the Carnegie-Mellon Navlab

In this paper, we present an approach to color image understanding that accounts for color variations due to highlights and shading. We demonstrate that the reflected light from every point on a dielectric object, such as plastic, can be described as a linear combination of the object color and the highlight color. The colors of all light rays reflected from one object then form a planar cluster in the color space. The shape of this cluster is determined by the object and highlight colors and by the object shape and illumination geometry. We present a method that exploits the difference between object color and highlight color to separate the color of every pixel into a matte component and a highlight component. This generates two intrinsic images, one showing the scene without highlights, and the other one showing only the highlights. The intrinsic images may be a useful tool for a variety of algorithms in computer vision, such as stereo vision, motion analysis, shape from shading, and shape from highlights. Our method combines the analysis of matte and highlight reflection with a sensor model that accounts for camera limitations. This enables us to successfully run our algorithm on real images taken in a laboratory setting. We show and discuss the results.

/pdf/the-measurement-of-highlights-in-color-images-4c038ectu9.pdf

The measurement of highlights in color images

This thesis uses a physical, intrinsic reflection model to analyze the effects of shading and highlights on dielectric, non-uniform materials, such as plastics and paints. Since black-and-white images do not provide sufficient information to interpret these effects appropriately, this work uses color images. The colors seen on a dielectric object are an additive mixture of the object color and the illumination color, i.e.: the colors are a linear combination of those two color vectors. They therefore lie in a plane in the color space. Within the plane, they form a cluster that looks like a skewed T, due to the geometric properties of shading and highlight reflection.
When combined with a camera model, this description of physically possible color changes on one object can be used to guide an algorithm in interpreting color images. It is the basis of an algorithm that analyzes color variation in real color images. The algorithm looks for characteristic color clusters in the image and relates their shape to hypotheses on the object color and the amount of shading, as well as the position, strength and color of highlights on the object. It verifies the hypotheses by reapplying them back to the image, testing whether and how they apply to the pixels in the local image area from which the respective hypothesis was derived. In this way, it adapts the image interpretation process to local scene characteristics and reacts differently to color and intensity changes at different places in the image.
The result is an image segmentation that ignores color changes due to shading and highlights, as well as a set of intrinsic images, one describing the amount of shading at every pixel in the image, and the other one showing the amount highlight reflection at every pixel. Such intrinsic images can be a useful tool for other areas of computer vision that are currently disturbed by highlights in images, such as stereo vision and motion analysis. The approach can also be used to determine the illumination color from the highlights and as a preprocessor for methods to determine object shape from shading or highlights. This line of research may lead to physics-based image understanding methods that are both more reliable and more useful than traditional methods. (Abstract shortened with permission of author.)

/pdf/a-physical-approach-to-color-image-understanding-34sisikqri.pdf

Steven A. Shafer

Papers

Using color to separate reflection components

Vision and navigation for the Carnegie-Mellon Navlab

Vision and Navigation for the Carnegie-Mellon Navlab

The measurement of highlights in color images

A physical approach to color image understanding