In 1997, Azuma published a survey on augmented reality (AR). Our goal is to complement, rather than replace, the original survey by presenting representative examples of the new advances. We refer one to the original survey for descriptions of potential applications (such as medical visualization, maintenance and repair of complex equipment, annotation, and path planning); summaries of AR system characteristics (such as the advantages and disadvantages of optical and video approaches to blending virtual and real, problems in display focus and contrast, and system portability); and an introduction to the crucial problem of registration, including sources of registration error and error-reduction strategies.

https://www.cc.gatech.edu/~blair/papers/ARsurveyCGA.pdf

Recent advances in augmented reality

This paper provides a classification of perceptual issues in augmented reality, created with a visual processing and interpretation pipeline in mind. We organize issues into ones related to the environment, capturing, augmentation, display, and individual user differences. We also illuminate issues associated with more recent platforms such as handhelds or projector-camera systems. Throughout, we describe current approaches to addressing these problems, and suggest directions for future research.

https://web.cse.msstate.edu/~swan/publications/papers/2010_Kruijff-etal_Perceptual-Issues_ISMAR.pdf

Perceptual issues in augmented reality revisited

To prevent moving shadows being misclassified as moving objects or parts of moving objects, this paper presents an explicit method for detection of moving cast shadows on a dominating scene background. Those shadows are generated by objects moving between a light source and the background. Moving cast shadows cause a frame difference between two succeeding images of a monocular video image sequence. For shadow detection, these frame differences are detected and classified into regions covered and regions uncovered by a moving shadow. The detection and classification assume plane background and a nonnegligible size and intensity of the light sources. A cast shadow is detected by temporal integration of the covered background regions while subtracting the uncovered background regions. The shadow detection method is integrated into an algorithm for two-dimensional (2-D) shape estimation of moving objects from the informative part of the description of the international standard ISO/MPEG-4. The extended segmentation algorithm compensates first apparent camera motion. Then, a spatially adaptive relaxation scheme estimates a change detection mask for two consecutive images. An object mask is derived from the change detection mask by elimination of changes due to background uncovered by moving objects and by elimination of changes due to background covered or uncovered by moving cast shadows. Results obtained with MPEG-4 test sequences and additional sequences show that the accuracy of object segmentation is substantially improved in presence of moving cast shadows. Objects and shadows are detected and tracked separately.

/pdf/detection-of-moving-cast-shadows-for-object-segmentation-3l46s4gd1a.pdf

Detection of moving cast shadows for object segmentation

Overview of the MPEG-4 Standard and Error Resilience Investigations

We present an algorithm of color image understanding which segments the image after analyzing sur-faces with color variations due to lighting condition and object colors. The work is based on dichromaticreflection model according to the strategy of hypothesis plus test, following the continuity of image and thefeature of color clusters, the algorithm completes the image segmentation and gives forth a physical des-cription of imaging process, including intrinsic images, segmented image, the color of light and objects. Re-flecting respectively the propertics of light condition and every objects, both matte image and highlight im-age compose the intrinsic images.

A physical approach to color image understanding

In applications of augmented reality like virtual studio TV production, multisite video conference applications using a virtual meeting room and synthetic/natural hybrid coding according to the new ISO/MPEG-4 standard, a synthetic scene is mixed into a natural scene to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. In this paper, the illumination of the natural scene is estimated automatically and applied to the synthetic scene. The natural scenes are restricted to scenes with nonoccluding, simple, moving, mainly rigid objects. For illumination estimation, these natural objects are automatically segmented in the natural image sequence and three-dimensionally (3-D) modeled using ellipsoid-like models. The 3-D shape, 3-D motion, and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual object mixed into real video telephone sequences show that the virtual object appears naturally having the same shading and shadows as the real objects. Further, shading and shadow allows the viewer to understand the motion trajectory of the objects much better.

/pdf/augmented-reality-with-automatic-illumination-control-3pco6lppb5.pdf

Augmented reality with automatic illumination control incorporating ellipsoidal models

In this paper, the source model of moving rigid 3D objects of an object-based analysis-synthesis coder (OBASC) is extended from diffuse to non-diffuse illumination introducing the explicit illumination model of a distant point light source and ambient diffuse light. For each image of a real image sequence containing moving objects, first, shape and 3D motion parameters describing the objects are estimated assuming an ellipsoid-like smooth shape. Then, the illumination parameters are estimated by a fast iterative maximum-likelihood Gauβ-Newton estimation method. Typically, the illumination parameters converge after very few images close to the true ones. The accurateness depends on the amount of object rotation and the correctness of the shape assumptions. For a real image sequence showing a textured ball covering 20% of image area, rotating about 10 ° per frame, and illuminated by spot and ambient light, the extension of the source model reduces the model failures from 9.9% of the image area to 6.7%. In the area of model failures, the image synthesized from the source model parameters differ significantly from the real image. In this early experiment, source model parameters are coded losslessly. Since model failures are expensive by means of bit-rate, a significant reduction of bit-rate can be expected.

Estimation of point light source parameters for object-based coding

For virtual studio TV production, multi-site videoconference applications using a virtual meeting room and for synthetic/natural hybrid coding according to the forthcoming ISO/MPEG-4 standard, a synthetic and a natural scene are mixed to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. The illumination of the natural scene is estimated automatically and applied to the synthetic scene. For illumination estimation, moving real objects are segmented in the natural image sequence and 3D modeled. The 3D shape, 3D motion and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual ball mixed automatically into a real video telephone sequence show that virtual objects appear naturally having the same shading and shadows as the real person.

Illumination analysis for synthetic/natural hybrid image sequence generation

In this paper, a non-complex estimator is developed for the tilt angle of the rotation axis of an object that is illuminated by a point light source. The tilt angle defines the orientation of the 2D projection of the 3D object rotation axis in the image plane and is a strong clue for image understanding. The estimator evaluates two images of a video image sequence showing a moving object. Additionally, only a displacement vector field and the 2D object silhouette are required. No 3D information is required. Therefore, the object is assumed to be rigid, to be matte, to have equally distributed surface normals and to be illuminated by a distant point light source and ambient light. For estimation, the displaced frame ratio (DFR), i.e. the frame ratio after motion compensation, is evaluated statistically. The DFR depends only on the photometric effect of temporally changing object shading. Experimental results with real images show the proper performance of the derived estimator for real objects.

/pdf/object-rotation-axis-from-shading-4fv6z1309i.pdf

Object rotation axis from shading

In video telephone scenes to be transmitted at low data rates non–diffuse scene illumination has to be expected In this contribution, non–diffuse scene illumination and 3D object motion is estimated concurrently from a monocular image sequence Image analysis uses the source model ”3D rigid moving objects with non–diffuse illumination” The illumination is modeled by a distant point light source and ambient light Experimental results of estimating and compensating non–diffuse scene illumination and 3D object motion using real and artificially animated image sequences are given For a real image sequence, a prediction gain of about 06 dB can be stated compared to the source model ”3D rigid moving objects with diffuse illumination”

J. Stauder

Papers

Augmented reality with automatic illumination control incorporating ellipsoidal models

Estimation of point light source parameters for object-based coding

Illumination analysis for synthetic/natural hybrid image sequence generation

Object rotation axis from shading

Joint Estimation of Object Motion and Point Light Illumination for 3D–Object–Based Analysis–Synthesis Coding