On robust estimation of the location parameter

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

/pdf/kinectfusion-real-time-3d-reconstruction-and-interaction-lu8d05hunj.pdf

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

Automatic generation and detection of highly reliable fiducial markers under occlusion

We are on the verge of ubiquitously adopting Augmented Reality (AR) technologies to enhance our percep- tion and help us see, hear, and feel our environments in new and enriched ways. AR will support us in fields such as education, maintenance, design and reconnaissance, to name but a few. This paper describes the field of AR, including a brief definition and development history, the enabling technologies and their characteristics. It surveys the state of the art by reviewing some recent applications of AR technology as well as some known limitations regarding human factors in the use of AR systems that developers will need to overcome.

/pdf/a-survey-of-augmented-reality-technologies-applications-and-3d9hf1tnc4.pdf

A Survey of Augmented Reality Technologies, Applications and Limitations

While the use of naturally-occurring features is a central focus of machine perception, artificial features (fiducials) play an important role in creating controllable experiments, ground truthing, and in simplifying the development of systems where perception is not the central objective. We describe a new visual fiducial system that uses a 2D bar code style “tag”, allowing full 6 DOF localization of features from a single image. Our system improves upon previous systems, incorporating a fast and robust line detection system, a stronger digital coding system, and greater robustness to occlusion, warping, and lens distortion. While similar in concept to the ARTag system, our method is fully open and the algorithms are documented in detail.

/pdf/apriltag-a-robust-and-flexible-visual-fiducial-system-3i6i7330tw.pdf

AprilTag: A robust and flexible visual fiducial system

In this paper we present two techniques for natural feature tracking in real-time on mobile phones. We achieve interactive frame rates of up to 20 Hz for natural feature tracking from textured planar targets on current-generation phones. We use an approach based on heavily modified state-of-the-art feature descriptors, namely SIFT and Ferns. While SIFT is known to be a strong, but computationally expensive feature descriptor, Ferns classification is fast, but requires large amounts of memory. This renders both original designs unsuitable for mobile phones. We give detailed descriptions on how we modified both approaches to make them suitable for mobile phones. We present evaluations on robustness and performance on various devices and finally discuss their appropriateness for augmented reality applications.

Pose tracking from natural features on mobile phones

In this paper we describe the first stand-alone AugmentedReality (AR) system with self-tracking running on anunmodified personal digital assistant (PDA) with acommercial camera. The project exploits the readyavailability of consumer devices with a minimal need forinfrastructure. The application provides the user with athree-dimensional augmented view of the environment.Our system achieves good overlay registration accuracyby using a popular marker-based tracking toolkit(ARToolKit), which runs directly on the PDA. Weintroduce an optional client/server architecture that isbased on wireless networking and is able to dynamicallyand transparently offload the tracking task in order toprovide better performance in select areas. Thehardware/software framework is modular and can beeasily combined with many elements of an existing ARframework. As a demonstration of the effectiveness, wepresent a 3D navigation application that guides a userthrough an unknown building to a chosen location.

/pdf/first-steps-towards-handheld-augmented-reality-2ssb3pqd40.pdf

First steps towards handheld augmented reality

In this paper, we present three techniques for 6DOF natural feature tracking in real time on mobile phones. We achieve interactive frame rates of up to 30 Hz for natural feature tracking from textured planar targets on current generation phones. We use an approach based on heavily modified state-of-the-art feature descriptors, namely SIFT and Ferns plus a template-matching-based tracker. While SIFT is known to be a strong, but computationally expensive feature descriptor, Ferns classification is fast, but requires large amounts of memory. This renders both original designs unsuitable for mobile phones. We give detailed descriptions on how we modified both approaches to make them suitable for mobile phones. The template-based tracker further increases the performance and robustness of the SIFT- and Ferns-based approaches. We present evaluations on robustness and performance and discuss their appropriateness for Augmented Reality applications.

Real-Time Detection and Tracking for Augmented Reality on Mobile Phones

In this paper we present ARToolKitPlus, a successor to the popular ARToolKit pose tracking library. ARToolKitPlus has been optimized and extended for the usage on mobile devices such as smartphones, PDAs and Ultra Mobile PCs (UMPCs). We explain the need and specific requirements of pose tracking on mobile devices and how we met those requirements. To prove the applicability we performed an extensive benchmark series on a broad range of off-the-shelf handhelds.

ARToolKitPlus for Pose Trackin on Mobile Devices

Augmented Reality (AR) can naturally complement mobile computing on wearable devices by providing an intuitive interface to a three-dimensional information space embedded within physical reality. Unfortunately, current wearable AR systems are relatively complex, expensive, fragile and heavy, rendering them unfit for large-scale deployment involving untrained users outside constrained laboratory environments. Consequently, the scale of collaborative multi-user experiments have not yet exceeded a handful of participants. In this paper, we present a system architecture for interactive, infrastructure-independent multi-user AR applications running on off-the-shelf handheld devices. We implemented a four-user interactive game installation as an evaluation setup to encourage playful engagement of participants in a cooperative task. Over the course of five weeks, more than five thousand visitors from a wide range of professional and socio-demographic backgrounds interacted with our system at four different locations.

Daniel Wagner

Papers

Pose tracking from natural features on mobile phones

First steps towards handheld augmented reality

Real-Time Detection and Tracking for Augmented Reality on Mobile Phones

ARToolKitPlus for Pose Trackin on Mobile Devices

Towards massively multi-user augmented reality on handheld devices