Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

/pdf/image-quality-assessment-from-error-visibility-to-structural-1rlwcqe34t.pdf

Image quality assessment: from error visibility to structural similarity

A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail.

/pdf/a-model-of-saliency-based-visual-attention-for-rapid-scene-wv0n3yqxix.pdf

A model of saliency-based visual attention for rapid scene analysis

https://surfer.nmr.mgh.harvard.edu/ftp/articles/dale99-recon1.pdf

Cortical surface-based analysis. I. Segmentation and surface reconstruction

Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268

Camera arrays are used to acquire the 360◦ surround video data presented on 3D immersive displays. The design of these arrays involves a large number of decisions ranging from the placement and orientation of the cameras to the choice of lenses and sensors. We implemented an open-source software environment (iset360) to support engineers designing and evaluating camera arrays for virtual and augmented reality applications. The software uses physically based ray tracing to simulate a 3D virtual spectral scene and traces these rays through multi-element spherical lenses to calculate the irradiance at the imaging sensor. The software then simulates imaging sensors to predict the captured images. The sensor data can be processed to produce the stereo and monoscopic 360◦ panoramas commonly used in virtual reality applications. By simulating the entire capture pipeline, we can visualize how changes in the system components influence the system performance. We demonstrate the use of the software by simulating a variety of different camera rigs, including the Facebook Surround360, the GoPro Odyssey, the GoPro Omni, and the Samsung Gear 360. Introduction Head mounted visual displays can provide a compelling and immersive experience of a three-dimensional scene. Because the experience can be very impactful, there is a great deal of interest in developing applications ranging from clinical medicine, behavioral change, entertainment, education and experience-sharing [1] [2]. In some applications, computer graphics is used to generate content, providing a realistic, but not real, experience (e.g., video games). In other applications, the content is acquired from a real event (e.g., sports, concerts, news, or family gathering) using camera arrays (rigs) and subsequent extensive image processing that capture and render the environment (Figure 1). The design of these rigs involves many different engineering decisions, including the selection of lenses, sensors, and camera positions. In addition to the rig, there are many choices of how to store and process the acquired content. For example, data from multiple cameras are often transformed into a stereo pair of 360◦ panoramas [3] by stitching together images captured by multiple cameras. Based on the user’s head position and orientation, data are extracted from the panorama and rendered on a head mounted display. There is no single quality-limiting element of this system, and moreover, interactions between the hardware and software design choices limit how well metrics of individual components predict overall system quality. To create a good experience, we must be able to assess the combination of hardware and software components that comprise the entire system. Building and testing a complete rig is costly and slow; hence, it can be useful to obtain guidance about design choices by using Figure 1. Overview of the hardware and software components that combine in an camera rig for an immersive head-mounted display application. (A) The simulation includes a 3D spectral scene, the camera rig definition, and the individual camera specifications. This simulation produces a set of image outputs. (B) The images are then processed by a series of software algorithms. In this case, we show a pipeline that produces an intermediate panorama representation and the viewport calculations that render an image dependent on the users head position. a simulation of the system. This paper describes software tools that simulate controlled 3D realistic scenes and image acquisition systems, in order to generate images produced by specific hardware choices. These images are the inputs to the stitching and rendering algorithms. The simulation enables engineers to explore the impact of different design choices on the entire imaging system, including realistic scenes, hardware components, and post-processing algorithms. Software Implementation The iset360 software, which models the image capture pipeline of 360 camera rigs, has portions in MATLAB and portions in C++. The simulation software is freely available in three repositories within the ISET GitHub project: https://github.com/ISET 1. Figure 2 and Figure 3 summarize the initial stages of the workflow. The first portion of the code creates realistic 3D scenes and calculates the expected sensor irradiance given a lens description. To do so, we start with a 3D, virtual scene that is constructed using 3D modeling software (e.g. Blender or Maya). The scene is converted into a format compatible with PBRT [4], which is implemented in C++. PBRT is a quantitative computer graphics tool that we use to calculate the irradiance at the sensor as light travels from the 3D scene, through the lens, and onto the sensor surface. We augmented the PBRT code to return multispectral images, model lens diffraction and simulate light fields [5]. To promote platform in1The three repositories are iset360, iset3d, and isetcam

Image Systems Simulation for 360° Camera Rigs

— ClearType is a subpixel-rendering method designed to improve the perceived quality of text. The method renders text at subpixel resolution and then applies a one-dimensional symmetric mean-preserving filter to reduce color artifacts. This paper describes a computational method and experimental tests to assess user preferences for different filter parameters. The computational method uses a physical display simulation and a perceptual metric that includes a model of human spatial and chromatic sensitivity. The method predicts experimentally measured preferences for filters for a range of characters, fonts, and displays.

Optimizing subpixel rendering using a perceptual metric

Reproducible Tract Profiles (RTP) comprises a set of methods to manage and analyze diffusion weighted imaging (DWI) data for reproducible tractography. The tools take MRI data from the scanner and process them through a series of analysis implemented as Docker containers that are integrated into a modern neuroinformatics platform (Flywheel). The platform guarantees that the entire pipeline can be re-executed, using the same data and computational parameters. In this paper, we describe (1) a cloud based neuroinformatics platform, (2) a tool to programmatically access and control the platform from a client, and (3) the DWI analysis tools that are used to identify the positions of 22 tracts and their diffusion profiles. The combination of these three components defines a system that transforms raw data into reproducible tract profiles for publication. Graphical abstract Reproducible Tract Profiles (RTP) comprises a set of methods to manage and analyze diffusion weighted imaging (DWI) data for reproducible tractography. The RTP methods comprise two main parts.
 Server side software tools for storing data and metadata and managing containerized computations. Client side software tools that enable the researcher to read data and metadata and manage server-side computations. The server-side computational tools are embedded in containers that are linked to a JSON file with a complete specification of the computational parameters. The data and computational infrastructure on the server is fully reproducible.

/pdf/reproducible-tract-profiles-rtp-from-diffusion-mri-198qxn2a9h.pdf

Reproducible Tract Profiles (RTP): from diffusion MRI acquisition to publication

The nature of artificial vision with a retinal prosthesis, and the degree to which the brain can adapt to the unnatural input from such a device, are poorly understood. Therefore, the development of current and future devices may be aided by theory and simulations that help to infer and understand what prosthesis patients see. A biologically-informed, extensible computational framework is presented here to predict visual perception and the potential effect of learning with a subretinal prosthesis. The framework relies on linear reconstruction of the stimulus from retinal responses to infer the visual information available to the patient. A simulation of the physiological optics of the eye and light responses of the major retinal neurons was used to calculate the optimal linear transformation for reconstructing natural images from retinal activity. The result was then used to reconstruct the visual stimulus during the artificial activation expected from a subretinal prosthesis in a degenerated retina, as a proxy for inferred visual perception. Several simple observations reveal the potential utility of such a simulation framework. The inferred perception obtained with prosthesis activation was substantially degraded compared to the inferred perception obtained with normal retinal responses, as expected given the limited resolution and lack of cell type specificity of the prosthesis. Consistent with clinical findings and the importance of cell type specificity, reconstruction using only ON cells, and not OFF cells, was substantially more accurate. Finally, when reconstruction was re-optimized for prosthesis stimulation, simulating idealized learning by the patient, the accuracy of inferred perception was much closer to that of healthy vision. The reconstruction approach thus provides a more complete method for exploring the potential for treating blindness with retinal prostheses than has been available previously. It may also be useful for interpreting patient data in clinical trials, and for improving prosthesis design.

https://www.biorxiv.org/content/biorxiv/early/2017/10/20/206409.full.pdf

Simulation of visual perception and learning with a retinal prosthesis

We investigate the spatial contrast-sensitivity of modern convolutional neural networks (CNNs) and a linear support vector machine (SVM). To measure performance, we compare the CNN contrast sensitivity across a range of patterns with the contrast sensitivity of a Bayesian ideal observer (IO) with the signal-known-exactly and noise-known-statistically. A ResNet-18 reaches optimal performance for harmonic patterns, as well as several classes of real world signals including faces. For these stimuli the CNN substantially outperforms the SVM. We further analyze the case in which the signal might appear in one of multiple locations and found that CNN spatial sensitivity continues to match the IO. However, the CNN sensitivity is far below optimal at detecting certain complex texture patterns. These measurements show that CNNs spatial contrast-sensitivity differs markedly between spatial patterns. The variation in spatial contrast-sensitivity may be a significant factor, influencing the performance level of an imaging system designed to detect low contrast spatial patterns.

/pdf/a-convolutional-neural-network-reaches-optimal-sensitivity-40aqke6e80.pdf

Brian A. Wandell

Papers

Image Systems Simulation for 360° Camera Rigs

Optimizing subpixel rendering using a perceptual metric

Reproducible Tract Profiles (RTP): from diffusion MRI acquisition to publication

Simulation of visual perception and learning with a retinal prosthesis

A Convolutional Neural Network Reaches Optimal Sensitivity for Detecting Some, but Not All, Patterns