We develop a framework for learning generic, expressive image priors that capture the statistics of natural scenes and can be used for a variety of machine vision tasks. The approach extends traditional Markov random field (MRF) models by learning potential functions over extended pixel neighborhoods. Field potentials are modeled using a Products-of-Experts framework that exploits nonlinear functions of many linear filter responses. In contrast to previous MRF approaches all parameters, including the linear filters themselves, are learned from training data. We demonstrate the capabilities of this Field of Experts model with two example applications, image denoising and image inpainting, which are implemented using a simple, approximate inference scheme. While the model is trained on a generic image database and is not tuned toward a specific application, we obtain results that compete with and even outperform specialized techniques.

/pdf/fields-of-experts-a-framework-for-learning-image-priors-514ncleszh.pdf

Fields of Experts: a framework for learning image priors

We develop a framework for learning generic, expressive image priors that capture the statistics of natural scenes and can be used for a variety of machine vision tasks. The approach provides a practical method for learning high-order Markov random field (MRF) models with potential functions that extend over large pixel neighborhoods. These clique potentials are modeled using the Product-of-Experts framework that uses non-linear functions of many linear filter responses. In contrast to previous MRF approaches all parameters, including the linear filters themselves, are learned from training data. We demonstrate the capabilities of this Field-of-Experts model with two example applications, image denoising and image inpainting, which are implemented using a simple, approximate inference scheme. While the model is trained on a generic image database and is not tuned toward a specific application, we obtain results that compete with specialized techniques.

Fields of Experts

In this paper, we investigate material classification from single images obtained under unknown viewpoint and illumination. It is demonstrated that materials can be classified using the joint distribution of intensity values over extremely compact neighborhoods (starting from as small as 3times3 pixels square) and that this can outperform classification using filter banks with large support. It is also shown that the performance of filter banks is inferior to that of image patches with equivalent neighborhoods. We develop novel texton-based representations which are suited to modeling this joint neighborhood distribution for Markov random fields. The representations are learned from training images and then used to classify novel images (with unknown viewpoint and lighting) into texture classes. Three such representations are proposed and their performance is assessed and compared to that of filter banks. The power of the method is demonstrated by classifying 2,806 images of all 61 materials present in the Columbia-Utrecht database. The classification performance surpasses that of recent state-of-the-art filter bank-based classifiers such as Leung and Malik (IJCV 01), Cula and Dana (IJCV 04), and Varma and Zisserman (IJCV 05). We also benchmark performance by classifying all of the textures present in the UIUC, Microsoft Textile, and San Francisco outdoor data sets. We conclude with discussions on why features based on compact neighborhoods can correctly discriminate between textures with large global structure and why the performance of filter banks is not superior to that of the source image patches from which they were derived.

/pdf/a-statistical-approach-to-material-classification-using-270vhhep9l.pdf

A Statistical Approach to Material Classification Using Image Patch Exemplars

We question the role that large scale filter banks have traditionally played in texture classification. It is demonstrated that textures can be classified using the joint distribution of intensity values over extremely compact neighborhoods (starting from as small as 3 /spl times/ 3 pixels square), and that this outperforms classification using filter banks with large support. We develop a novel texton based representation, which is suited to modeling this joint neighborhood distribution for MRFs. The representation is learnt from training images, and then used to classify novel images (with unknown viewpoint and lighting) into texture classes. The power of the method is demonstrated by classifying over 2800 images of all 61 textures present in the Columbia-Utrecht database. The classification performance surpasses that of recent state-of-the-art filter bank based classifiers such as Leung & Malik, Cula & Dana, and Varma & Zisserman.

/pdf/texture-classification-are-filter-banks-necessary-48zwftlvom.pdf

Texture classification: are filter banks necessary?

In this paper we present a new approach to material classification under unknown viewpoint and illumination. Our texture model is based on the statistical distribution of clustered filter responses. However, unlike previous 3D texton representations, we use rotationally invariant filters and cluster in an extremely low dimensional space. Having built a texton dictionary, we present a novel method of classifying a single image without requiring any a priori knowledge about the viewing or illumination conditions under which it was photographed. We argue that using rotationally invariant filters while clustering in such a low dimensional space improves classification performance and demonstrate this claim with results on all 61 textures in the Columbia-Utrecht database. We then proceed to show how texture models can be further extended by compensating for viewpoint changes using weak isotropy.The new clustering and classification methods are compared to those of Leung and Malik (ICCV 1999), Schmid (CVPR 2001) and Cula and Dana (CVPR 2001), which are the current state-of-the-art approaches.

https://link.springer.com/content/pdf/10.1007/3-540-47977-5_17.pdf

Classifying Images of Materials: Achieving Viewpoint and Illumination Independence

A texture synthesis method is presented that generates similar texture from an example image. It is based on the emulation of simple but rather carefully chosen image intensity statistics. The resulting texture models are compact and no longer require the example image from which they were derived. They make explicit some structural aspects of the textures and the modeling allows knitting together different textures with convincingly looking transition zones. As textures are seldom flat, it is important to also model 3D effects when textures change under changing viewpoint. The simulation of such changes is supported by the model, assuming examples for the different viewpoints are given.

A compact model for viewpoint dependent texture synthesis. Discussion

A Compact Model for Viewpoint Dependent Texture Synthesis

Textures often result from complicated surface geometries. We propose a method that extracts such geometry from raw BTF data. Exploiting the huge amount of input data, only a few and rather weak assumptions about reflectance and geometry suffice. A key element of our approach are viewpoint robustness of reflectance features. We propose a few and compare them for 3D reconstruction.

http://www.macs.hw.ac.uk/texture2005/programme/papers/035.pdf

3D Texture Reconstruction from Extensive BTF Data

Murale is a European IST project that will develop 3D capture and visualisation technology for archaeology. The project will put special emphasis on the usability on the site, by the archaeologists themselves. The paper describes techniques that are being developed by three of the Murale partners in particular. These comprise two methods to generate 3D models of objects, and approaches to deal with the textures of materials and terrain. Put together with the database and visualisation expertise brought in by the other partners, Murale will not only contribute to the enhanced visualisation of archaeological sites and finds, but also to a faster and more complete documentation of the progress of excavations. The ancient city of Sagalassos, one of the major excavation sites in the eastern part of the Mediterranean, will be used as the primary test site.

/pdf/the-murale-project-image-based-3d-modeling-for-archaeology-v2robobo02.pdf

The MURALE project: image-based 3D modeling for Archaeology

Increasingly, realistic object, scene, and event modeling is based on image data rather than manual synthesis The paper describes a system for visits to a virtual, 3D archeological site One can navigate through this environment, with a virtual guide as companion One can ask questions using natural, fluent speech The guide will respond and will bring the visitor to the desired place Simple answers are given as changes in the orientations of his head, by him raising his eyebrows or by head nodding In the near future the head will speak

/pdf/image-based-3d-modeling-modeling-from-reality-263p7l4tlk.pdf

Alexey Zalesny

Papers

A compact model for viewpoint dependent texture synthesis. Discussion

A Compact Model for Viewpoint Dependent Texture Synthesis

3D Texture Reconstruction from Extensive BTF Data

The MURALE project: image-based 3D modeling for Archaeology

Image-based 3D modeling: modeling from reality