Image and template matching is probably the most important function in digital photogrammetry and also in automated modelling and mapping. Many approaches for matching have evolved over the years, but the problem is still unsolved in general terms. This paper describes the development of image matching techniques in photogrammetry over the past 50 years, addresses the results of some empirical accuracy studies and also provides a critical account of some of the problems that remain.Although automated approaches have quite a number of advantages, the quality of the results is still not satisfactory and, in some cases, far from acceptable. Even with the most advanced techniques, it is not yet possible to achieve the quality of results that a human operator can produce. There is an urgent need for further improvements and innovations, be it through more powerful multi-sensor approaches, thereby enlarging the information spectrum, and/or through advancements in image understanding algorithms, thus coming closer to human capabilities of reading and understanding image content.

Resume

L’appariement d’images et de formes est sans doute l’operation la plus importante en photogrammetrie numerique et en modelisation et cartographie automatiques. De nombreuses approches se sont succede depuis des annees mais le probleme n’est toujours pas totalement resolu. Cet article decrit le developpement des techniques d’appariement d’images en photogrammetrie pendant les 50 dernieres annees, presente les resultats obtenus dans quelques etudes de precision empiriques, et dresse un bilan critique des problemes qui subsistent. Bien que les approches automatiques aient un grand nombre d’avantages, la qualite des resultats n’est toujours pas satisfaisante, et meme loin d’etre acceptable dans certains cas. Meme avec les techniques les plus avancees, nous sommes toujours dans l’incapacite d’atteindre la qualite des resultats obtenus par un operateur humain. Il y a un besoin urgent d’ameliorations et d’innovations, soit a travers des approches multi-capteurs plus puissantes et consistant aelargir le spectre d’information, soit a travers une amelioration des algorithmes de comprehension d’images visant a les rendre plus proches des possibilites humaines de lecture et de comprehension du contenu des images.

Zusammenfassung

Bild- und Musterkorrelation gehoren zu den wichtigsten Grundfunktionen der Digitalen Photogrammetrie und somit auch der automatischen 3D Modellierung und Kartierung. Viele Ansatze zur Korrelation wurden uber die Jahre entwickelt, aber das Problem gilt grundsatzlich noch immer als ungelost. Dieser Beitrag beschreibt die Entwicklung der Verfahren der Bildkorrelation in der Photogrammetrie uber die letzten 50 Jahre, verweist auf die Ergebnisse einiger empirischer Genauigkeitsstudien und diskutiert einige der immer noch bestehenden Probleme.Obwohl automatische Verfahren eine ganze Reihe von Vorteilen aufweisen, ist doch die Qualitat der Ergebnisse meist nicht ausreichend, teilweise ja sogar weit entfernt von jeglicher Akzeptanz. Selbst mit den hochstentwickelten Verfahren sind wir immer noch nicht in der Lage, die Qualitat der Ergebnisse eines menschlichen Operateurs zu erreichen. Wir benotigen dringend weitere Verbesserungen und Innovationen. Dazu gibt es gegenwartig zwei grundsatzlich gangbare Wege: (a) Nutzung von Multi-Sensor Informationen und somit Erweiterung der Informationsgrundlagen und/oder (b) durch Fortschritte bei den Algorithmen des Bildverstehens und somit besserer Modellierung des menschlichen Prozesses des Bildverstehens.

Resumen

La correspondencia de imagenes y muestras es, probablemente, la funcion mas importante en la fotogrametria digital, en el modelado 3D y en la cartografia automatica. Muchos metodos de correspondencia han evolucionado a lo largo de los anos pero, en terminos generales, el problema se considera aun no resuelto completamente. Este articulo describe la evolucion de las tecnicas de correspondencia de imagenes en la fotogrametria a lo largo de los ultimos 50 anos, analiza los resultados de algunos estudios empiricos de la exactitud, y ofrece una valoracion critica de los problemas aun sin resolver. Aunque los metodos automaticos poseen un gran numero de ventajas, la calidad de los resultados no es todavia satisfactoria y, en algunos casos, incluso esta lejos de ser aceptable. Incluso con las mas avanzadas tecnicas no somos capaces de lograr la calidad de los resultados que un operador humano puede conseguir. Hay una necesidad urgente de continuar las mejoras e innovaciones, ya sea mediante la utilizacion de multiples sensores que incrementen el espectro de la informacion, o por avances en los algoritmos de comprension de la imagen que permitan acercarnos mas a la capacidad humana de lectura e interpretacion de su contenido.

https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1477-9730.2011.00671.x

Development and Status of Image Matching in Photogrammetry

This dissertation addresses the topic of automatic Digital Surface Model (DSM) generation from linear array images. Research on this issue is mainly motivated by the following facts: In recent years, CCD linear array sensors are widely used to acquire high-quality, highresolution panchromatic and multispectral imagery for photogrammetric and remote sensing applications. Most of these sensors have the ability to acquire more than 2 views of the terrain surface during a single flight line or orbit. The processing of these kinds of images provides a challenge for algorithmic redesign and this opens the possibility to reconsider and improve many photogrammetric processing components. In particular, automatic DSM generation through image matching techniques is one of the main topics. Although this topic has gained much attention in the past years and a wide variety of approaches have been developed, the accuracy performance and the problems encountered are very similar in the major approaches and the performance of these approaches does by far not live up to the standards set by manual measurements. Therefore, efforts have to be made to develop a general framework for automatic DSM generation from linear array images, into which specific algorithms can be inserted easily, investigated and combined in order to achieve reasonable results in terms of precision and reliability. In this dissertation, an image matching approach for automatic DSM generation from linear array images, which has the ability to provide dense, precise and reliable results is presented. The approach integrates different matching primitives, uses available and explicit knowledge concerning the image geometry and radiometry information, combines several image matching algorithms and automatic quality control, and works with a coarse-to-fine hierarchical matching strategy. The most outstanding characteristics are the efficient utilisation of multiple images and the integration of multiple matching primitives. With this approach, the linear array images and the given or previously triangulated orientation elements are taken as inputs. After pre-processing of the original images and production of the image pyramids, the matches of 3 kinds of features, i.e. feature points, grid points and edges, on the original resolution images are finally found progressively starting from the lowdensity features on the images with the lowest resolution. An intermediate DSM is reconstructed from the matched features on each level of the pyramid by using the constrained Delauney triangulation method, which in turn is used in the subsequent pyramid level for the approximations and self-tuning of the matching parameters. Finally least squares matching methods are used to achieve more precise matches for all the matched features and identify some false matches. The presented approach essentially consists of several mutually connected components: the image pre-processing, the multiple primitive multi-image (MPM) matching, the refined matching and the system performance evaluation. Each of them is important and possesses particular features, which are fully elaborated in different parts of the dissertation. First of all, a pre-processing method, which combines an adaptive smoothing filter and the Wallis filter, is used in order to reduce the effects of the inherent radiometric problems and optimize the images for subsequent feature extraction and image matching procedure. The method mainly consists of 3 processing stages. In the first stage, the noise characteristics of the images are analyzed quantitatively in both homogeneous and non-homogeneous image regions. The image blur problem (image unsharpness) is also addressed through the analysis of the image’s Modulation Transfer Function (MTF). Then, an adaptive smoothing filter is applied to reduce the noise level and at the same time, to sharpen edges and preserve even fine detail such as corners and line end-points. Finally, the Wallis filter is applied to strongly enhance and sharpen the already existing texture patterns. The MPM matching procedure is the core of our approach. In this approach, the matching is performed with the aid of multiple images, incorporating multiple matching primitives – feature points, grid points and edges, integrating local and global image information and, utilizing a coarse-to-fine hierarchical matching strategy. The MPM approach mainly consists of 3 integrated subsystems: point extraction and matching procedure, edge extraction and matching procedure and relational matching procedure. These 3 subsystems are followed through the image pyramid and the results at higher levels are used for guidance at the lower levels. At each pyramid level, the correspondence is established in two matching stages – locally and globally. In the local matching stage dense patterns of points and edges are matched. A unique and robust matching algorithm The Geometrically Constrained Cross-Correlation (GC) algorithm is employed to provide matching candidates for points and edge pixels. The algorithm is based on the concept of multi-image matching guided from the object space and allows reconstruction of 3D objects by matching all the images at the same time, without having to go through the processing of all individual stereo-pairs and the merging of all stereo-pair results. The GC method, with the self-tuning of the parameters, leads to a reduction of problems caused by occlusions, multiple solutions and surface discontinuities. The global matching stage is responsible for imposing global consistency among the candidate matches in order to disambiguate the multiple candidates and avoid mismatches. The global matching is resolved by a probability relaxation based relational matching method. It uses the local support provided by points within a 2D neighborhood. This corresponds to imposing a piecewise smoothness constraint, in which the matched edges serve as breaklines in order to prohibit the smoothness constraint crossing these edges and preserves the surface discontinuities. The modified Multiphoto Geometrically Constrained Matching (MPGC) and the Least Squares B-Spline Snakes (LSB-Snakes) methods are used to achieve potentially sub-pixel accuracy matches and identify some inaccurate and possibly false matches. The DSM derived from the MPM module provides good enough approximations for these methods and increases the convergence rate. The initial values of the shaping parameters in MPGC matching can also be predetermined by using the image geometry and the derived DSM data. Finally, for each matched point, a reliability indicator is assigned based on the analysis of the matching results. For edges, a simplified version of the LSB-Snakes is implemented to match the edges, which are represented by parametric linear B-spline functions in object space. With this method, the parameters of linear B-spline functions of the edges in object space are directly estimated, together with the matching parameters in the image spaces of multiple images. The system has been tested extensively of linear array images with different image resolution and over different landcover types. The accuracy evaluation is based on the comparison between high quality DEMs/DSMs derived from airborne Laser Scanner or manual measurements and the automatically extracted DSMs. As evidenced by the visual inspection of the results, we can reproduce not only the general geomorphological features of the terrain relief, but also detailed features of relief. The results from the quantitative accuracy test indicate that the presented concept has the capability to give good and encouraging results. If the bias introduced by trees and buildings is taken out, we can expect a height accuracy of one pixel or even better from satellite imagery as “best case” scenario. In case of very highresolution TLS/SI images (footprint 8 cm and better) it is obvious that the “one pixel rule” cannot be maintained any more. Alone surface roughness and modeling errors will lead to larger deviations, such that an accuracy of 2 to 5 pixels should be considered an acceptable result.

Automatic Digital Surface Model (DSM) generation from linear array images

There are billions of photographs on the Internet, representing an extremely large, rich, and nearly comprehensive visual record of virtually every famous place on Earth. Unfortu- nately, these massive community photo collections are almost completely unstructured, making it very difficult to use them for applicationssuchasthevirtualexplorationofourworld.Overthe past several years, advances in computer vision have made it possible to automatically reconstruct 3-D geometryVincluding camera positions and scene modelsVfrom these large, diverse photo collections. Once the geometry is known, we can recover higher level information from the spatial distribution of photos, such as the most common viewpoints and paths through the scene. This paper reviews recent progress on these challenging computer vision problems, and describes how we can use the recovered structure to turn community photo collections into immersive, interactive 3-D experiences.

Scene Reconstruction and Visualization From Community Photo Collections Recent progress is described in digitizing and visualizing the world from data captured by people taking photos and uploading them to the web.

/pdf/algorithm-theoretical-basis-44pwose6nk.pdf

Algorithm Theoretical Basis

CCD linear array sensors are widely used to acquire panchromatic and multispectral imagery for photogrammetric and remote sensing applications. The processing of this kind of images provides a challenge for algorithmic redesign and this opens the possibility to reconsider and improve many photogrammetric processing components. In addition, the basic capabilities of image matching techniques have so far not been fully utilized yet. This paper presents a matching procedure for automatic DSM generation from linear array imagery data. It can provide dense, precise and reliable results. The method uses a coarse-to-fine hierarchical solution with an effective combination of several image matching algorithms and automatic quality control. The DSMs are generated by combination of matching results of feature points, grid points and edges. Finally, a modified multi-photo geometrically constrained (MPGC) matching algorithm is employed to achieve sub-pixel accuracy for all the matched features with multi-image or multi-strip image data. The proposed approach in this paper has been applied to different areas with varying textures and terrain types. The accuracy tests are based on the comparison between the high quality DEMs / DSMs derived from airborne Laser Scanner or manual measurements and the automatic extracted DSMs. Results with STARIMAGER, IKONOS and SPOT5 HRS images are reported. We demonstrate with these experiments that our approach leads to good results.

/pdf/automatic-dsm-generation-from-linear-array-imagery-data-5030pek26u.pdf

Automatic dsm generation from linear array imagery data

/pdf/a-region-growing-algorithm-for-matching-of-terrain-images-1pyhubw7ri.pdf

T. K. W. Chau

Papers

A "Region-Growing" Algorithm for Matching of Terrain Images.