Showing papers by "Larry Matthies published in 2011"

Unmanned ground vehicle perception using thermal infrared cameras

Abstract: The ability to perform off-road autonomous navigation at any time of day or night is a requirement for some unmanned ground vehicle (UGV) programs. Because there are times when it is desirable for military UGVs to operate without emitting strong, detectable electromagnetic signals, a passive only terrain perception mode of operation is also often a requirement. Thermal infrared (TIR) cameras can be used to provide day and night passive terrain perception. TIR cameras have a detector sensitive to either mid-wave infrared (MWIR) radiation (3-5?m) or long-wave infrared (LWIR) radiation (8-12?m). With the recent emergence of high-quality uncooled LWIR cameras, TIR cameras have become viable passive perception options for some UGV programs. The Jet Propulsion Laboratory (JPL) has used a stereo pair of TIR cameras under several UGV programs to perform stereo ranging, terrain mapping, tree-trunk detection, pedestrian detection, negative obstacle detection, and water detection based on object reflections. In addition, we have evaluated stereo range data at a variety of UGV speeds, evaluated dual-band TIR classification of soil, vegetation, and rock terrain types, analyzed 24 hour water and 12 hour mud TIR imagery, and analyzed TIR imagery for hazard detection through smoke. Since TIR cameras do not currently provide the resolution available from megapixel color cameras, a UGV's daytime safe speed is often reduced when using TIR instead of color cameras. In this paper, we summarize the UGV terrain perception work JPL has performed with TIR cameras over the last decade and describe a calibration target developed by General Dynamics Robotic Systems (GDRS) for TIR cameras and other sensors.

Systems and methods for automated water detection using visible sensors

Abstract: Systems and methods are disclosed that include automated machine vision that can utilize images of scenes captured by a 3D imaging system configured to image light within the visible light spectrum to detect water. One embodiment includes autonomously detecting water bodies within a scene including capturing at least one 3D image of a scene using a sensor system configured to detect visible light and to measure distance from points within the scene to the sensor system, and detecting water within the scene using a processor configured to detect regions within each of the at least one 3D images that possess at least one characteristic indicative of the presence of water.

Daytime water detection based on sky reflections

Abstract: Robust water detection is a critical perception requirement for unmanned ground vehicle (UGV) autonomous navigation. This is particularly true in wide-open areas where water can collect in naturally occurring terrain depressions during periods of heavy precipitation and form large water bodies. One of the properties of water useful for detecting it is that its surface acts as a horizontal mirror at large incidence angles. Water bodies can be indirectly detected by detecting reflections of the sky below the horizon in color imagery. The Jet Propulsion Laboratory (JPL) has implemented a water detector based on sky reflections that geometrically locates the pixel in the sky that is reflecting on a candidate water pixel on the ground and predicts if the ground pixel is water based on color similarity and local terrain features. This software detects water bodies in wide-open areas on cross-country terrain at mid- to far-range using imagery acquired from a forward-looking stereo pair of color cameras mounted on a terrestrial UGV. In three test sequences approaching a pond under a clear, overcast, and cloudy sky, the true positive detection rate was 100% when the UGV was beyond 7 meters of the water's leading edge and the largest false positive detection rate was 0.58%. The sky reflection based water detector has been integrated on an experimental unmanned vehicle and field tested at Ft. Indiantown Gap, PA, USA.

Feature and pose constrained visual Aided Inertial Navigation for computationally constrained aerial vehicles

Abstract: A Feature and Pose Constrained Extended Kalman Filter (FPC-EKF) is developed for highly dynamic computationally constrained micro aerial vehicles. Vehicle localization is achieved using only a low performance inertial measurement unit and a single camera. The FPC-EKF framework augments the vehicle's state with both previous vehicle poses and critical environmental features, including vertical edges. This filter framework efficiently incorporates measurements from hundreds of opportunistic visual features to constrain the motion estimate, while allowing navigating and sustained tracking with respect to a few persistent features. In addition, vertical features in the environment are opportunistically used to provide global attitude references. Accurate pose estimation is demonstrated on a sequence including fast traversing, where visual features enter and exit the field-of-view quickly, as well as hover and ingress maneuvers where drift free navigation is achieved with respect to the environment.

Autonomous landing and ingress of micro-air-vehicles in urban environments based on monocular vision

Abstract: Unmanned micro air vehicles (MAVs) will play an important role in future reconnaissance and search and rescue applications. In order to conduct persistent surveillance and to conserve energy, MAVs need the ability to land, and they need the ability to enter (ingress) buildings and other structures to conduct reconnaissance. To be safe and practical under a wide range of environmental conditions, landing and ingress maneuvers must be autonomous, using real-time, onboard sensor feedback. To address these key behaviors, we present a novel method for vision-based autonomous MAV landing and ingress using a single camera for two urban scenarios: landing on an elevated surface, representative of a rooftop, and ingress through a rectangular opening, representative of a door or window. Real-world scenarios will not include special navigation markers, so we rely on tracking arbitrary scene features; however, we do currently exploit planarity of the scene. Our vision system uses a planar homography decomposition to detect navigation targets and to produce approach waypoints as inputs to the vehicle control algorithm. Scene perception, planning, and control run onboard in real-time; at present we obtain aircraft position knowledge from an external motion capture system, but we expect to replace this in the near future with a fully self-contained, onboard, vision-aided state estimation algorithm. We demonstrate autonomous vision-based landing and ingress target detection with two different quadrotor MAV platforms. To our knowledge, this is the first demonstration of onboard, vision-based autonomous landing and ingress algorithms that do not use special purpose scene markers to identify the destination.

Stereo and IMU assisted visual odometry on an OMAP3530 for small robots

Abstract: Small robots require very compact, low-power, yet high performance processors for vision-based navigation algorithms like stereo vision and visual odometry. Research on real-time implementations of these algorithms has focused on FPGAs, GPUs, ASICs, and general purpose processors, which are either too big, too hot, or too hard to program. System-on-a-chip (SoC) processors for smart phones have not been exploited yet for these functions. Here we present a real-time stereo vision system with IMU assisted visual odometry implemented on a single Texas Instruments 720Mhz/520Mhz OMAP3530 SoC. We achieve frame rates of 46 fps at QVGA or 8 fps at VGA resolutions while simultaneously tracking up to 200 features, taking full advantage of the OMAP3530's integer DSP and floating point ARM processors. This is a substantial advancement over previous work as the stereo implementation produces 146Mde/s in 2.5W, yielding a stereo energy efficiency of 58.8Mde/J, which is 3.75× better than prior DSP stereo while providing more functionality.

FPGA implementation of stereo disparity with high throughput for mobility applications

Abstract: High speed stereo vision can allow unmanned robotic systems to navigate safely in unstructured terrain, but the computational cost can exceed the capacity of typical embedded CPUs. 1 2 In this paper, we describe an end-to-end stereo computation co-processing system optimized for fast throughput that has been implemented on a single Virtex 4 LX160 FPGA. This system is capable of operating on images from a 1024x768 3CCD (true RGB) camera pair at 15Hz. Data enters the FPGA directly from the cameras via Camera Link and is rectified, pre-filtered and converted into a disparity image all within the FPGA, incurring no CPU load. Once complete, a rectified image and the final disparity image are read out over the PCI bus, for a bandwidth cost of 68MB/sec. Within the FPGA there are 4 distinct algorithms: Camera Link capture, Bilinear rectification, Bilateral subtraction pre-filtering and the Sum of Absolute Difference (SAD) disparity. Each module will be described in brief along with the data flow and control logic for the system. The system has been successfully fielded upon the Carnegie Mellon University's National Robotics Engineering Center (NREC) Crusher system during extensive field trials in 2007 and 2008 and is being implemented for other surface mobility systems at JPL.

MER Spirit rover localization: Comparison of ground image– and orbital image–based methods and science applications

Abstract: [1] During 6 years of continuous operations on the Martian surface, the Mars Exploration Rover (MER) Spirit has covered a traverse of approximately 7 km from the landing point to its current position at “Troy” near Home Plate Localization of Spirit (and Opportunity) has been performed using two different methods: one that employs an incremental bundle adjustment (IBA) using rover imagery, and one that compares image features common to both a rover orthoimage and an orbital orthoimage The IBA method continuously yields the desired 3-D rover positions at a very high level of accuracy and provides a simultaneous solution for high-quality topographic mapping of neighborhoods surrounding the rover On the other hand, high-resolution orbital imagery can verify rover positions wherever the rover track is visible Rapid rover localization on the orbital orthoimage is often achieved by comparing a rover orthoimage to the orbital orthoimage In this paper, we present research results from a systematic comparison of these two localization methods over the entire length of the Spirit traverse Two orbital orthoimages were generated from High Resolution Imaging Science Experiment (HiRISE) imagery Integration of Mars Orbiter Laser Altimeter (MOLA) data into the HiRISE digital elevation model (DEM) and orthoimage generation was performed and proved to be effective in reducing large inconsistencies between MOLA and HiRISE data This study found an overall difference of 15 percent of the traversed distance between the two sets of traverse positions derived using the two different localization methods After a geometric transformation from one traverse to the other, the remaining inconsistency then represents the local differences between them and can be reduced to a level of less than 015 percent Discussions of error sources and the strength and weakness of the methods are given Scientific applications of the localization data are also briefly introduced

Distributed autonomous mapping of indoor environments

Abstract: This paper describes the results of a Joint Experiment performed on behalf of the MAST CTA The system developed for the Joint Experiment makes use of three robots which work together to explore and map an unknown environment Each of the robots used in this experiment is equipped with a laser scanner for measuring walls and a camera for locating doorways Information from both of these types of structures is concurrently incorporated into each robot's local map using a graph based SLAM technique A Distributed-Data-Fusion algorithm is used to efficiently combine local maps from each robot into a shared global map Each robot computes a compressed local feature map and transmits it to neighboring robots, which allows each robot to merge its map with the maps of its neighbors Each robot caches the compressed maps from its neighbors, allowing it to maintain a coherent map with a common frame of reference The robots utilize an exploration strategy to efficiently cover the unknown environment which allows collaboration on an unreliable communications channel As each new branching point is discovered by a robot, it broadcasts the information about where this point is along with the robot's path from a known landmark to the other robots When the next robot reaches a dead-end, new branching points are allocated by auction In the event of communication interruption, the robot which observed the branching point will eventually explore it; therefore, the exploration is complete in the face of communication failure

Simultaneous mapping and stereo extrinsic parameter calibration using GPS measurements

Abstract: Stereo vision is useful for a variety of robotics tasks, such as navigation and obstacle avoidance. However, recovery of valid range data from stereo depends on accurate calibration of the extrinsic parameters of the stereo rig, i.e., the 6-DOF transform between the left and right cameras. Stereo self-calibration is possible, but, without additional information, the absolute scale of the stereo baseline cannot be determined. In this paper, we formulate stereo extrinsic parameter calibration as a batch maximum likelihood estimation problem, and use GPS measurements to establish the scale of both the scene and the stereo baseline. Our approach is similar to photogrammetric bundle adjustment, and closely related to many structure from motion algorithms. We present results from simulation experiments using a range of GPS accuracy levels; these accuracies are achievable by varying grades of commercially-available receivers. We then validate the algorithm using stereo and GPS data acquired from a moving vehicle. Our results indicate that the approach is promising.

Implementation of pin point landing vision components in an FPGA system

Abstract: Pin-point landing is required to enable missions to land close, typically within 10 meters, to scientifically important targets in generally hazardous terrain. In Pin Point Landing both high accuracy and high speed estimation of position and orientation is needed to provide input to the control system to safely choose and navigate to a safe landing site. A proposed algorithm called VISion aided Inertial NAVigation (VISINAV) has shown that the accuracy requirements can be met. [2][3] VISINAV was shown in software only, and was expected to use FPGA enhancements in the future to improve the computational speed needed for pin point landing during Entry Descent and Landing (EDL). Homography, feature detection and spatial correlation are computationally intensive parts of VISINAV. Homography aligns the map image with the descent image so that small correlation windows can be used, and feature detection provides regions that spatial correlation can track from frame to frame in order to estimate vehicle motion. On MER the image Homography, Feature Detection and Correlation would take approximately 650ms tracking 75 features between frames. We implemented Homography, Feature detection and Correlation on a Virtex 4 LX160 FPGA to run in under 25ms while tracking 500 features to improve algorithm reliability and throughput. 1 2