Turning images into 3-D models
TL;DR: A multistage image-based modeling approach that requires only a limited amount of human interactivity and is capable of capturing the fine geometric details with similar accuracy as close-range active range sensors is proposed.
Abstract: In this article developments and performance analysis of image matching for detailed surface reconstruction of heritage objects is discussed. Three dimensional image-based modeling of heritages is a very interesting topic with many possible applications. In this article we propose a multistage image-based modeling approach that requires only a limited amount of human interactivity and is capable of capturing the fine geometric details with similar accuracy as close-range active range sensors. It can also cope with wide baselines using several advancements over standard stereo matching techniques. Our approach is sequential, starting from a sparse basic segmented model created with a small number of interactively measured points. This model, specifically the equation of each surface, is then used as a guide to automatically add the fine details. The following three techniques are used, each where best suited, to retrieve the details: 1) for regularly shaped patches such as planes, cylinders, or quadrics, we apply a fast relative stereo matching technique. 2) For more complex or irregular segments with unknown shape, we use a global multi-image geometrically constrained technique. 3) For segments unsuited for stereo matching, we employ depth from shading (DFS). The goal is not the development of a fully automated procedure for 3D object reconstruction from image data or a sparse stereo approach, but we aim at the digital reconstruction of detailed and accurate surfaces from calibrated and oriented images for practical daily documentation and digital conservation of wide variety of heritage objects.
Citations
More filters
TL;DR: This article reviews the actual optical 3D measurement sensors and 3D modeling techniques, with their limitations and potentialities, requirements and specifications.
Abstract: The importance of landscape and heritage recording and documentation with optical remote sensing sensors is well recognized at international level. The continuous development of new sensors, data capture methodologies and multi-resolution 3D representations, contributes significantly to the digital 3D documentation, mapping, conservation and representation of landscapes and heritages and to the growth of research in this field. This article reviews the actual optical 3D measurement sensors and 3D modeling techniques, with their limitations and potentialities, requirements and specifications. Examples of 3D surveying and modeling of heritage sites and objects are also shown throughout the paper.
655 citations
TL;DR: A critical review and analysis of four dense image-matching algorithms, available as open-source and commercial software, for the generation of dense point clouds are presented.
Abstract: Image matching has a history of more than 50 years, with the first experiments performed with analogue procedures for cartographic and mapping purposes. The recent integration of computer vision algorithms and photogrammetric methods is leading to interesting procedures which have increasingly automated the entire image-based 3D modelling process. Image matching is one of the key steps in 3D modelling and mapping. This paper presents a critical review and analysis of four dense image-matching algorithms, available as open-source and commercial software, for the generation of dense point clouds. The eight datasets employed include scenes recorded from terrestrial and aerial blocks, acquired with convergent and normal (parallel axes) images, and with different scales. Geometric analyses are reported in which the point clouds produced with each of the different algorithms are compared with one another and also to ground-truth data.
515 citations
Cites background or methods from "Turning images into 3-D models"
...…2012; Hermann and Klette, 2013) or via identification of correspondences in multiple images (multi-view stereo – MVS) (Collins, 1996; Zhang, 2005; Pierrot-Deseilligny and Paparoditis, 2006; Goesele et al., 2007; Remondino et al., 2008; Furukawa and Ponce, 2010; Vu et al., 2012; Toldo et al., 2013)....
[...]
...…algorithms is based on the utilised primitives, namely, image intensity patterns (windows composed of grey values around a point of interest) or features (for example, edges and regions), leading to area-based matching (ABM) or feature-based matching (FBM) algorithms (Remondino et al., 2008)....
[...]
TL;DR: Following an overview of the state-of-art of 3D imaging sensors, a number of significant examples of their use are presented, with particular reference to industry, heritage, medicine, and criminal investigation applications.
Abstract: 3D imaging sensors for the acquisition of three dimensional (3D) shapes have created, in recent years, a considerable degree of interest for a number of applications. The miniaturization and integration of the optical and electronic components used to build them have played a crucial role in the achievement of compactness, robustness and flexibility of the sensors. Today, several 3D sensors are available on the market, even in combination with other sensors in a “sensor fusion” approach. An importance equal to that of physical miniaturization has the portability of the measurements, via suitable interfaces, into software environments designed for their elaboration, e.g., CAD-CAM systems, virtual renders, and rapid prototyping tools. In this paper, following an overview of the state-of-art of 3D imaging sensors, a number of significant examples of their use are presented, with particular reference to industry, heritage, medicine, and criminal investigation applications.
511 citations
TL;DR: Comprehensive evaluation of efficiency, distribution quality, and positional accuracy of the extracted point pairs proves the capabilities of the proposed matching algorithm on a variety of optical remote sensing images.
Abstract: Extracting well-distributed, reliable, and precisely aligned point pairs for accurate image registration is a difficult task, particularly for multisource remote sensing images that have significant illumination, rotation, and scene differences. The scale-invariant feature transform (SIFT) approach, as a well-known feature-based image matching algorithm, has been successfully applied in a number of automatic registration of remote sensing images. Regardless of its distinctiveness and robustness, the SIFT algorithm suffers from some problems in the quality, quantity, and distribution of extracted features particularly in multisource remote sensing imageries. In this paper, an improved SIFT algorithm is introduced that is fully automated and applicable to various kinds of optical remote sensing images, even with those that are five times the difference in scale. The main key of the proposed approach is a selection strategy of SIFT features in the full distribution of location and scale where the feature qualities are quarantined based on the stability and distinctiveness constraints. Then, the extracted features are introduced to an initial cross-matching process followed by a consistency check in the projective transformation model. Comprehensive evaluation of efficiency, distribution quality, and positional accuracy of the extracted point pairs proves the capabilities of the proposed matching algorithm on a variety of optical remote sensing images.
255 citations
Cites background from "Turning images into 3-D models"
..., 1/50 pixel [11]), they require initial matching positions, and they suffer from monotonous textures, occlusions, image distortions, and illumination differences....
[...]
TL;DR: In this paper, the authors describe the development of image matching techniques in photogrammetry over the past 50 years, address the results of some empirical accuracy studies and also provide a critical account of some of the problems that remain.
Abstract: Image and template matching is probably the most important function in digital photogrammetry and also in automated modelling and mapping. Many approaches for matching have evolved over the years, but the problem is still unsolved in general terms. This paper describes the development of image matching techniques in photogrammetry over the past 50 years, addresses the results of some empirical accuracy studies and also provides a critical account of some of the problems that remain.Although automated approaches have quite a number of advantages, the quality of the results is still not satisfactory and, in some cases, far from acceptable. Even with the most advanced techniques, it is not yet possible to achieve the quality of results that a human operator can produce. There is an urgent need for further improvements and innovations, be it through more powerful multi-sensor approaches, thereby enlarging the information spectrum, and/or through advancements in image understanding algorithms, thus coming closer to human capabilities of reading and understanding image content.
Resume
L’appariement d’images et de formes est sans doute l’operation la plus importante en photogrammetrie numerique et en modelisation et cartographie automatiques. De nombreuses approches se sont succede depuis des annees mais le probleme n’est toujours pas totalement resolu. Cet article decrit le developpement des techniques d’appariement d’images en photogrammetrie pendant les 50 dernieres annees, presente les resultats obtenus dans quelques etudes de precision empiriques, et dresse un bilan critique des problemes qui subsistent. Bien que les approches automatiques aient un grand nombre d’avantages, la qualite des resultats n’est toujours pas satisfaisante, et meme loin d’etre acceptable dans certains cas. Meme avec les techniques les plus avancees, nous sommes toujours dans l’incapacite d’atteindre la qualite des resultats obtenus par un operateur humain. Il y a un besoin urgent d’ameliorations et d’innovations, soit a travers des approches multi-capteurs plus puissantes et consistant aelargir le spectre d’information, soit a travers une amelioration des algorithmes de comprehension d’images visant a les rendre plus proches des possibilites humaines de lecture et de comprehension du contenu des images.
Zusammenfassung
Bild- und Musterkorrelation gehoren zu den wichtigsten Grundfunktionen der Digitalen Photogrammetrie und somit auch der automatischen 3D Modellierung und Kartierung. Viele Ansatze zur Korrelation wurden uber die Jahre entwickelt, aber das Problem gilt grundsatzlich noch immer als ungelost. Dieser Beitrag beschreibt die Entwicklung der Verfahren der Bildkorrelation in der Photogrammetrie uber die letzten 50 Jahre, verweist auf die Ergebnisse einiger empirischer Genauigkeitsstudien und diskutiert einige der immer noch bestehenden Probleme.Obwohl automatische Verfahren eine ganze Reihe von Vorteilen aufweisen, ist doch die Qualitat der Ergebnisse meist nicht ausreichend, teilweise ja sogar weit entfernt von jeglicher Akzeptanz. Selbst mit den hochstentwickelten Verfahren sind wir immer noch nicht in der Lage, die Qualitat der Ergebnisse eines menschlichen Operateurs zu erreichen. Wir benotigen dringend weitere Verbesserungen und Innovationen. Dazu gibt es gegenwartig zwei grundsatzlich gangbare Wege: (a) Nutzung von Multi-Sensor Informationen und somit Erweiterung der Informationsgrundlagen und/oder (b) durch Fortschritte bei den Algorithmen des Bildverstehens und somit besserer Modellierung des menschlichen Prozesses des Bildverstehens.
Resumen
La correspondencia de imagenes y muestras es, probablemente, la funcion mas importante en la fotogrametria digital, en el modelado 3D y en la cartografia automatica. Muchos metodos de correspondencia han evolucionado a lo largo de los anos pero, en terminos generales, el problema se considera aun no resuelto completamente. Este articulo describe la evolucion de las tecnicas de correspondencia de imagenes en la fotogrametria a lo largo de los ultimos 50 anos, analiza los resultados de algunos estudios empiricos de la exactitud, y ofrece una valoracion critica de los problemas aun sin resolver. Aunque los metodos automaticos poseen un gran numero de ventajas, la calidad de los resultados no es todavia satisfactoria y, en algunos casos, incluso esta lejos de ser aceptable. Incluso con las mas avanzadas tecnicas no somos capaces de lograr la calidad de los resultados que un operador humano puede conseguir. Hay una necesidad urgente de continuar las mejoras e innovaciones, ya sea mediante la utilizacion de multiples sensores que incrementen el espectro de la informacion, o por avances en los algoritmos de comprension de la imagen que permitan acercarnos mas a la capacidad humana de lectura e interpretacion de su contenido.
232 citations
Cites background from "Turning images into 3-D models"
...Tests in close range applications have been reported, for example, by Remondino et al. (2008, 2009) and Remondino and Menna (2008), although not always supported by accurate reference data....
[...]
...Some of the results have been reported in Wolff (2009). The key problem with such tests, which were done with aerial images with footprints of 8 and 20 cm, is the generation of sufficiently good reference data....
[...]
References
More filters
TL;DR: There is a natural uncertainty principle between detection and localization performance, which are the two main goals, and with this principle a single operator shape is derived which is optimal at any scale.
Abstract: This paper describes a computational approach to edge detection. The success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points. These goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution. We define detection and localization criteria for a class of edges, and present mathematical forms for these criteria as functionals on the operator impulse response. A third criterion is then added to ensure that the detector has only one response to a single edge. We use the criteria in numerical optimization to derive detectors for several common image features, including step edges. On specializing the analysis to step edges, we find that there is a natural uncertainty principle between detection and localization performance, which are the two main goals. With this principle we derive a single operator shape which is optimal at any scale. The optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a Gaussian-smoothed image. We extend this simple detector using operators of several widths to cope with different signal-to-noise ratios in the image. We present a general method, called feature synthesis, for the fine-to-coarse integration of information from operators at different scales. Finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge.
28,073 citations
"Turning images into 3-D models" refers background or methods in this paper
...The Canny detector [17] is probably the most widely used edge detector and very suitable due to its performance and low sensitivity to parameter variation....
[...]
...Feature points are interest points extracted with the Lue operator [27] and the dominant points of the edges (extracted with Canny operator [17]), computed through a polygon approximation algorithm....
[...]
01 Jan 2001
TL;DR: This book is referred to read because it is an inspiring book to give you more chance to get experiences and also thoughts and it will show the best book collections and completed collections.
Abstract: Downloading the book in this website lists can give you more advantages. It will show you the best book collections and completed collections. So many books can be found in this website. So, this is not only this multiple view geometry in computer vision. However, this book is referred to read because it is an inspiring book to give you more chance to get experiences and also thoughts. This is simple, read the soft file of the book and you get it.
14,282 citations
TL;DR: This paper has designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can easily be extended to include new algorithms.
Abstract: Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can be easily extended to include new algorithms. We have also produced several new multiframe stereo data sets with ground truth, and are making both the code and data sets available on the Web.
7,458 citations
TL;DR: It is observed that the ranking of the descriptors is mostly independent of the interest region detector and that the SIFT-based descriptors perform best and Moments and steerable filters show the best performance among the low dimensional descriptors.
Abstract: In this paper, we compare the performance of descriptors computed for local interest regions, as, for example, extracted by the Harris-Affine detector [Mikolajczyk, K and Schmid, C, 2004]. Many different descriptors have been proposed in the literature. It is unclear which descriptors are more appropriate and how their performance depends on the interest region detector. The descriptors should be distinctive and at the same time robust to changes in viewing conditions as well as to errors of the detector. Our evaluation uses as criterion recall with respect to precision and is carried out for different image transformations. We compare shape context [Belongie, S, et al., April 2002], steerable filters [Freeman, W and Adelson, E, Setp. 1991], PCA-SIFT [Ke, Y and Sukthankar, R, 2004], differential invariants [Koenderink, J and van Doorn, A, 1987], spin images [Lazebnik, S, et al., 2003], SIFT [Lowe, D. G., 1999], complex filters [Schaffalitzky, F and Zisserman, A, 2002], moment invariants [Van Gool, L, et al., 1996], and cross-correlation for different types of interest regions. We also propose an extension of the SIFT descriptor and show that it outperforms the original method. Furthermore, we observe that the ranking of the descriptors is mostly independent of the interest region detector and that the SIFT-based descriptors perform best. Moments and steerable filters show the best performance among the low dimensional descriptors.
7,057 citations
18 Jun 2003
TL;DR: It is observed that the ranking of the descriptors is mostly independent of the interest region detector and that the SIFT-based descriptors perform best and Moments and steerable filters show the best performance among the low dimensional descriptors.
Abstract: In this paper we compare the performance of interest point descriptors. Many different descriptors have been proposed in the literature. However, it is unclear which descriptors are more appropriate and how their performance depends on the interest point detector. The descriptors should be distinctive and at the same time robust to changes in viewing conditions as well as to errors of the point detector. Our evaluation uses as criterion detection rate with respect to false positive rate and is carried out for different image transformations. We compare SIFT descriptors (Lowe, 1999), steerable filters (Freeman and Adelson, 1991), differential invariants (Koenderink ad van Doorn, 1987), complex filters (Schaffalitzky and Zisserman, 2002), moment invariants (Van Gool et al., 1996) and cross-correlation for different types of interest points. In this evaluation, we observe that the ranking of the descriptors does not depend on the point detector and that SIFT descriptors perform best. Steerable filters come second ; they can be considered a good choice given the low dimensionality.
3,362 citations
"Turning images into 3-D models" refers background or methods in this paper
...Features are first extracted and afterwards associated with attributes (“descriptors”) to characterize and match them [15]....
[...]
...There are three types of features: 1) Interest Points [14], [15]: Interest point detectors are generally divided into contour-based methods, which search for maximal curvature or inflexion points along the contour chains; signal-based methods, which analyze the image signal and derive a measure which indicates the presence of an interest point; and methods based on template fitting which try to fit the image signal to a parametric model of a specific type of interest point (e....
[...]