scispace - formally typeset
Search or ask a question

Showing papers on "Interpolation published in 2019"


Proceedings ArticleDOI
Tero Karras1, Samuli Laine1, Timo Aila1
15 Jun 2019
TL;DR: This paper proposed an alternative generator architecture for GANs, borrowing from style transfer literature, which leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images.
Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

6,564 citations


Proceedings ArticleDOI
15 Jun 2019
TL;DR: In this paper, an implicit field is used to assign a value to each point in 3D space, so that a shape can be extracted as an iso-surface, and a binary classifier is trained to perform this assignment.
Abstract: We advocate the use of implicit fields for learning generative models of shapes and introduce an implicit field decoder, called IM-NET, for shape generation, aimed at improving the visual quality of the generated shapes. An implicit field assigns a value to each point in 3D space, so that a shape can be extracted as an iso-surface. IM-NET is trained to perform this assignment by means of a binary classifier. Specifically, it takes a point coordinate, along with a feature vector encoding a shape, and outputs a value which indicates whether the point is outside the shape or not. By replacing conventional decoders by our implicit decoder for representation learning (via IM-AE) and shape generation (via IM-GAN), we demonstrate superior results for tasks such as generative shape modeling, interpolation, and single-view 3D reconstruction, particularly in terms of visual quality. Code and supplementary material are available at https://github.com/czq142857/implicit-decoder.

1,261 citations


Journal ArticleDOI
TL;DR: Kluger et al. as mentioned in this paper proposed a heatmap-style visualization for scRNA-seq based on one-dimensional t-distributed stochastic neighbor embedding (t-SNE) for simultaneously visualizing the expression patterns of thousands of genes.
Abstract: t-distributed stochastic neighbor embedding (t-SNE) is widely used for visualizing single-cell RNA-sequencing (scRNA-seq) data, but it scales poorly to large datasets. We dramatically accelerate t-SNE, obviating the need for data downsampling, and hence allowing visualization of rare cell populations. Furthermore, we implement a heatmap-style visualization for scRNA-seq based on one-dimensional t-SNE for simultaneously visualizing the expression patterns of thousands of genes. Software is available at https://github.com/KlugerLab/FIt-SNE and https://github.com/KlugerLab/t-SNE-Heatmaps .

328 citations


Journal ArticleDOI
TL;DR: An improved k‐space reconstruction method using scan‐specific deep learning that is trained on autocalibration signal (ACS) data is developed.
Abstract: Purpose To develop an improved k-space reconstruction method using scan-specific deep learning that is trained on autocalibration signal (ACS) data Theory Robust artificial-neural-networks for k-space interpolation (RAKI) reconstruction trains convolutional neural networks on ACS data This enables nonlinear estimation of missing k-space lines from acquired k-space data with improved noise resilience, as opposed to conventional linear k-space interpolation-based methods, such as GRAPPA, which are based on linear convolutional kernels Methods The training algorithm is implemented using a mean square error loss function over the target points in the ACS region, using a gradient descent algorithm The neural network contains 3 layers of convolutional operators, with 2 of these including nonlinear activation functions The noise performance and reconstruction quality of the RAKI method was compared with GRAPPA in phantom, as well as in neurological and cardiac in vivo data sets Results Phantom imaging shows that the proposed RAKI method outperforms GRAPPA at high (≥4) acceleration rates, both visually and quantitatively Quantitative cardiac imaging shows improved noise resilience at high acceleration rates (rate 4:23% and rate 5:48%) over GRAPPA The same trend of improved noise resilience is also observed in high-resolution brain imaging at high acceleration rates Conclusion The RAKI method offers a training database-free deep learning approach for MRI reconstruction, with the potential to improve many existing reconstruction approaches, and is compatible with conventional data acquisition protocols

309 citations


Proceedings ArticleDOI
01 Jun 2019
TL;DR: DAIN as mentioned in this paper proposes a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones, and then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels.
Abstract: Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. While significant advances have been made from the recent deep convolutional neural networks, the quality of interpolation is often reduced due to large object motion or occlusion. In this work, we propose a video frame interpolation method which explicitly detects the occlusion by exploring the depth information. Specifically, we develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the proposed model performs favorably against state-of-the-art frame interpolation methods on a wide variety of datasets. The source code and pre-trained model are available at https://github.com/baowenbo/DAIN.

266 citations


Proceedings ArticleDOI
Vikas Verma1, Alex Lamb, Juho Kannala1, Yoshua Bengio, David Lopez-Paz2 
01 Aug 2019

258 citations


Proceedings Article
01 Jan 2019
TL;DR: A differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image and to view foreground rasterization as a weighted interpolation of local properties and background rasterized as a distance-based aggregation of global geometry.
Abstract: Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present DIB-Render, a novel rendering framework through which gradients can be analytically computed. Key to our approach is to view rasterization as a weighted interpolation, allowing image gradients to back-propagate through various standard vertex shaders within a single framework. Our approach supports optimizing over vertex positions, colors, normals, light directions and texture coordinates, and allows us to incorporate various well-known lighting models from graphics. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively 2D supervision.

231 citations


Posted Content
TL;DR: A video frame interpolation method which explicitly detects the occlusion by exploring the depth information, and develops a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones.
Abstract: Video frame interpolation aims to synthesize nonexistent frames in-between the original frames. While significant advances have been made from the recent deep convolutional neural networks, the quality of interpolation is often reduced due to large object motion or occlusion. In this work, we propose a video frame interpolation method which explicitly detects the occlusion by exploring the depth information. Specifically, we develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones. In addition, we learn hierarchical features to gather contextual information from neighboring pixels. The proposed model then warps the input frames, depth maps, and contextual features based on the optical flow and local interpolation kernels for synthesizing the output frame. Our model is compact, efficient, and fully differentiable. Quantitative and qualitative results demonstrate that the proposed model performs favorably against state-of-the-art frame interpolation methods on a wide variety of datasets.

228 citations


Journal ArticleDOI
TL;DR: In this paper, a convolutional neural network is trained on a collection of discrete, parameterizable fluid simulation velocity fields to synthesize fluid simulations from a set of reduced parameters.
Abstract: This paper presents a novel generative model to synthesize fluid simulations from a set of reduced parameters. A convolutional neural network is trained on a collection of discrete, parameterizable fluid simulation velocity fields. Due to the capability of deep learning architectures to learn representative features of the data, our generative model is able to accurately approximate the training data set, while providing plausible interpolated in-betweens. The proposed generative model is optimized for fluids by a novel loss function that guarantees divergence-free velocity fields at all times. In addition, we demonstrate that we can handle complex parameterizations in reduced spaces, and advance simulations in time by integrating in the latent space with a second network. Our method models a wide variety of fluid behaviors, thus enabling applications such as fast construction of simulations, interpolation of fluids with different parameters, time re-sampling, latent space simulations, and compression of fluid simulation data. Reconstructed velocity fields are generated up to 700x faster than re-simulating the data with the underlying CPU solver, while achieving compression rates of up to 1300x.

225 citations


Proceedings ArticleDOI
01 Oct 2019
TL;DR: A novel Interpolated Convolution operation, InterpConv, is proposed to tackle the point cloud feature learning and understanding problem, to utilize a set of discrete kernel weights and interpolate point features to neighboring kernel-weight coordinates by an interpolation function for convolution.
Abstract: Point cloud is an important type of 3D representation. However, directly applying convolutions on point clouds is challenging due to the sparse, irregular and unordered data structure. In this paper, we propose a novel Interpolated Convolution operation, InterpConv, to tackle the point cloud feature learning and understanding problem. The key idea is to utilize a set of discrete kernel weights and interpolate point features to neighboring kernel-weight coordinates by an interpolation function for convolution. A normalization term is introduced to handle neighborhoods of different sparsity levels. Our InterpConv is shown to be permutation and sparsity invariant, and can directly handle irregular inputs. We further design Interpolated Convolutional Neural Networks (InterpCNNs) based on InterpConv layers to handle point cloud recognition tasks including shape classification, object part segmentation and indoor scene semantic parsing. Experiments show that the networks can capture both fine-grained local structures and global shape context information effectively. The proposed approach achieves state-of-the-art performance on public benchmarks including ModelNet40, ShapeNet Parts and S3DIS.

223 citations


Journal ArticleDOI
TL;DR: Deep-learning-based approaches for seismic data antialiasing interpolation are used, which could extract deeper features of the training data in a nonlinear way by self-learning and avoid linear events, sparsity, and low-rank assumptions of the traditional interpolation methods.
Abstract: Seismic data interpolation is a longstanding issue. Most current methods are only suitable for randomly missing cases. To deal with regularly missing cases, an antialiasing strategy should ...

Journal ArticleDOI
17 Jul 2019
TL;DR: The results show a remarkable phenomenon that GANs can preserve latent space interpolation across categories, even though the discriminator is never exposed to such vectors.
Abstract: Generative Adversarial networks (GANs) have obtained remarkable success in many unsupervised learning tasks and unarguably, clustering is an important unsupervised learning problem. While one can potentially exploit the latent-space back-projection in GANs to cluster, we demonstrate that the cluster structure is not retained in the GAN latent space. In this paper, we propose ClusterGAN as a new mechanism for clustering using GANs. By sampling latent variables from a mixture of one-hot encoded variables and continuous latent variables, coupled with an inverse network (which projects the data to the latent space) trained jointly with a clustering specific loss, we are able to achieve clustering in the latent space. Our results show a remarkable phenomenon that GANs can preserve latent space interpolation across categories, even though the discriminator is never exposed to such vectors. We compare our results with various clustering baselines and demonstrate superior performance on both synthetic and real datasets.

Journal ArticleDOI
10 Oct 2019
TL;DR: Two new fast and accurate surrogate models are presented that predict the gravitational waveform and the properties of the remnant black hole in numerical relativity simulations of binary black hole mergers, and are shown to be more accurate than existing models by at least an order of magnitude.
Abstract: This paper presents two surrogate models for simulations of binary black holes. These models are trained on previous results and produce an interpolation between them. The final outcome is shown to match the accuracy of numerical relativity simulations without some of the computational expense.

Journal ArticleDOI
TL;DR: Improvements in bias adjustment and trend preservation are demonstrated in a cross-validation framework and the new statistical downscaling method is stochastic and better at adjusting spatial variability than the old interpolation method.
Abstract: . In this paper I present new methods for bias adjustment and statistical downscaling that are tailored to the requirements of the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP). In comparison to their predecessors, the new methods allow for a more robust bias adjustment of extreme values, preserve trends more accurately across quantiles, and facilitate a clearer separation of bias adjustment and statistical downscaling. The new statistical downscaling method is stochastic and better at adjusting spatial variability than the old interpolation method. Improvements in bias adjustment and trend preservation are demonstrated in a cross-validation framework.

Journal ArticleDOI
17 Jul 2019
TL;DR: A new loss term, the cycle consistency loss, which can better utilize the training data to not only enhance the interpolation results, but also maintain the performance better with less training data is introduced.
Abstract: Video frame interpolation algorithms predict intermediate frames to produce videos with higher frame rates and smooth view transitions given two consecutive frames as inputs. We propose that: synthesized frames are more reliable if they can be used to reconstruct the input frames with high quality. Based on this idea, we introduce a new loss term, the cycle consistency loss. The cycle consistency loss can better utilize the training data to not only enhance the interpolation results, but also maintain the performance better with less training data. It can be integrated into any frame interpolation network and trained in an end-to-end manner. In addition to the cycle consistency loss, we propose two extensions: motion linearity loss and edge-guided training. The motion linearity loss approximates the motion between two input frames to be linear and regularizes the training. By applying edge-guided training, we further improve results by integrating edge information into training. Both qualitative and quantitative experiments demonstrate that our model outperforms the state-of-the-art methods. The source codes of the proposed method and more experimental results will be available at https://github.com/alex04072000/CyclicGen.

Proceedings ArticleDOI
Xiong Zhang, Qiang Li, Hong Mo1, Zhang Wenbo, Zheng Wen 
01 Oct 2019
TL;DR: Zhang et al. as mentioned in this paper reconstruct the full 3D mesh of a human hand from a single RGB image by parameterizing a generic 3D hand model with shape and relative 3D joint angles.
Abstract: In this paper, we present a HAnd Mesh Recovery (HAMR) framework to tackle the problem of reconstructing the full 3D mesh of a human hand from a single RGB image. In contrast to existing research on 2D or 3D hand pose estimation from RGB or/and depth image data, HAMR can provide a more expressive and useful mesh representation for monocular hand image understanding. In particular, the mesh representation is achieved by parameterizing a generic 3D hand model with shape and relative 3D joint angles. By utilizing this mesh representation, we can easily compute the 3D joint locations via linear interpolations between the vertexes of the mesh, while obtain the 2D joint locations with a projection of the 3D joints. To this end, a differentiable re-projection loss can be defined in terms of the derived representations and the ground-truth labels, thus making our framework end-to-end trainable. Qualitative experiments show that our framework is capable of recovering appealing 3D hand mesh even in the presence of severe occlusions. Quantitatively, our approach also outperforms the state-of-the-art methods for both 2D and 3D hand pose estimation from a monocular RGB image on several benchmark datasets.

Posted Content
TL;DR: Two strategies for the adaption of Mixup on sentence classification are proposed: one performs interpolation on word embeddings and another on sentence embedDings, and both serve as an effective, domain independent data augmentation approach for sentence classification.
Abstract: Mixup, a recent proposed data augmentation method through linearly interpolating inputs and modeling targets of random samples, has demonstrated its capability of significantly improving the predictive accuracy of the state-of-the-art networks for image classification. However, how this technique can be applied to and what is its effectiveness on natural language processing (NLP) tasks have not been investigated. In this paper, we propose two strategies for the adaption of Mixup on sentence classification: one performs interpolation on word embeddings and another on sentence embeddings. We conduct experiments to evaluate our methods using several benchmark datasets. Our studies show that such interpolation strategies serve as an effective, domain independent data augmentation approach for sentence classification, and can result in significant accuracy improvement for both CNN and LSTM models.

Journal ArticleDOI
TL;DR: A new interpolation scheme for a three-dimensional curvilinear C grid and its properties is developed and illustrated in a brief study of the distribution of floating microplastic in the northwest European continental shelf and its sensitivity to various physical processes.
Abstract: . With the increasing number of data produced by numerical ocean models, so increases the need for efficient tools to analyse these data. One of these tools is Lagrangian ocean analysis, where a set of virtual particles is released and their dynamics are integrated in time based on fields defining the ocean state, including the hydrodynamics and biogeochemistry if available. This popular methodology needs to adapt to the large variety of models producing these fields at different formats. This is precisely the aim of Parcels, a Lagrangian ocean analysis framework designed to combine (1) a wide flexibility to model particles of different natures and (2) an efficient implementation in accordance with modern computing infrastructure. In the new Parcels v2.0, we implement a set of interpolation schemes to read various types of discretized fields, from rectilinear to curvilinear grids in the horizontal direction, from z to s levels in the vertical direction and using grid staggering with the Arakawa A, B and C grids. In particular, we develop a new interpolation scheme for a three-dimensional curvilinear C grid and analyse its properties. Parcels v2.0 capabilities, including a suite of meta-field objects, are then illustrated in a brief study of the distribution of floating microplastic in the northwest European continental shelf and its sensitivity to various physical processes.

Journal ArticleDOI
01 Mar 2019
TL;DR: A deep-neural-network-enabled sinogram synthesis method for sparse-view CT is introduced and its outperformance to the existing interpolation methods and also to the iterative image reconstruction approach is shown.
Abstract: Recently, a number of approaches to low-dose computed tomography (CT) have been developed and deployed in commercialized CT scanners. Tube current reduction is perhaps the most actively explored technology with advanced image reconstruction algorithms. Sparse data sampling is another viable option to the low-dose CT, and sparse-view CT has been particularly of interest among the researchers in CT community. Since analytic image reconstruction algorithms would lead to severe image artifacts, various iterative algorithms have been developed for reconstructing images from sparsely view-sampled projection data. However, iterative algorithms take much longer computation time than the analytic algorithms, and images are usually prone to different types of image artifacts that heavily depend on the reconstruction parameters. Interpolation methods have also been utilized to fill the missing data in the sinogram of sparse-view CT thus providing synthetically full data for analytic image reconstruction. In this paper, we introduce a deep-neural-network-enabled sinogram synthesis method for sparse-view CT, and show its outperformance to the existing interpolation methods and also to the iterative image reconstruction approach.

Book ChapterDOI
03 Jun 2019
TL;DR: In this paper, the main results in classical approximation theory such as Weierstrass and Stone and Weierstass-type results, quantitative estimates in approximation by polynomials, interpolation results, best approximation results, etc. to fuzzy setting are generalized.
Abstract: In this survey main results in classical approximation theory such as Weierstrass and Stone and Weierstrass-type results, quantitative estimates in approximation by polynomials, interpolation results, best approximation results, etc. to fuzzy setting are generalized. Besides known results, several new results are proved.

Journal ArticleDOI
TL;DR: This work presents a new proof scheme that is quite straightforward with respect to the previous ones and has a much wider range of applicability and also sheds new insights on the reasons for the validity of replica formulas in Bayesian inference.
Abstract: In recent years important progress has been achieved towards proving the validity of the replica predictions for the (asymptotic) mutual information (or “free energy”) in Bayesian inference problems. The proof techniques that have emerged appear to be quite general, despite they have been worked out on a case-by-case basis. Unfortunately, a common point between all these schemes is their relatively high level of technicality. We present a new proof scheme that is quite straightforward with respect to the previous ones. We call it the adaptive interpolation method because it can be seen as an extension of the interpolation method developped by Guerra and Toninelli in the context of spin glasses, with an interpolation path that is adaptive. In order to illustrate our method we show how to prove the replica formula for three non-trivial inference problems. The first one is symmetric rank-one matrix estimation (or factorisation), which is the simplest problem considered here and the one for which the method is presented in full details. Then we generalize to symmetric tensor estimation and random linear estimation. We believe that the present method has a much wider range of applicability and also sheds new insights on the reasons for the validity of replica formulas in Bayesian inference.

Journal ArticleDOI
TL;DR: In this article, a two-step interpolation method (i.e., the mixed-effect model and inverse distance weighting technology) was developed to replace the missing values in AOD.

Posted Content
TL;DR: Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm, achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets.
Abstract: We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark datasets. Our theoretical analysis shows that ICT corresponds to a certain type of data-adaptive regularization with unlabeled points which reduces overfitting to labeled points under high confidence values.

Proceedings ArticleDOI
07 Jul 2019
TL;DR: A bound on how well such interpolative solutions can generalize to fresh test data is given, and it is shown that this bound generically decays to zero with the number of extra features, thus characterizing an explicit benefit of overparameterization.
Abstract: A continuing mystery in understanding the empirical success of deep neural networks has been in their ability to achieve zero training error and yet generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this "overparametrization" phenomena in the classical underdetermined linear regression problem, where all solutions that minimize training error interpolate the data, including noise. We give a bound on how well such interpolative solutions can generalize to fresh test data, and show that this bound generically decays to zero with the number of extra features, thus characterizing an explicit benefit of overparameterization. For appropriately sparse linear models, we provide a hybrid interpolating scheme (combining classical sparse recovery schemes with harmless noise-fitting) to achieve generalization error close to the bound on interpolative solutions.

Journal ArticleDOI
TL;DR: In this article, a boundary element analysis of two-dimensional linear elasticity problems by expanding element interpolation method is proposed, where the expanding element is made up based on a traditional discontinuous element by adding virtual nodes along the perimeter of the element.
Abstract: This paper aims to propose a boundary element analysis of two-dimensional linear elasticity problems by a new expanding element interpolation method.,The expanding element is made up based on a traditional discontinuous element by adding virtual nodes along the perimeter of the element. The internal nodes of the original discontinuous element are referred to as source nodes and its shape function as raw shape function. The shape functions of the expanding element constructed on both source nodes and virtual nodes are referred as fine shape functions. Boundary variables are interpolated by the fine shape functions, while the boundary integral equations are collocated on source nodes.,The expanding element inherits the advantages of both the continuous and discontinuous elements while overcomes their disadvantages. The polynomial order of fine shape functions of the expanding elements increases by two compared with their corresponding raw shape functions, while the expanding elements still keep independence to each other as the original discontinuous elements. This feature makes the expanding elements able to naturally and accurately interpolate both continuous and discontinuous fields.,Numerical examples are presented to verify the proposed method. Results have demonstrated that the accuracy, efficiency and convergence rate of the expanding element method.

Journal ArticleDOI
TL;DR: This study proposes zero-padding for resizing images to the same size and compares it with the conventional approach of scaling images up (zooming in) using interpolation, showing that zero- padding had no effect on the classification accuracy but considerably reduced the training time.
Abstract: The input to a machine learning model is a one-dimensional feature vector. However, in recent learning models, such as convolutional and recurrent neural networks, two- and three-dimensional feature tensors can also be inputted to the model. During training, the machine adjusts its internal parameters to project each feature tensor close to its target. After training, the machine can be used to predict the target for previously unseen feature tensors. What this study focuses on is the requirement that feature tensors must be of the same size. In other words, the same number of features must be present for each sample. This creates a barrier in processing images and texts, as they usually have different sizes, and thus different numbers of features. In classifying an image using a convolutional neural network (CNN), the input is a three-dimensional tensor, where the value of each pixel in each channel is one feature. The three-dimensional feature tensor must be the same size for all images. However, images are not usually of the same size and so are not their corresponding feature tensors. Resizing images to the same size without deforming patterns contained therein is a major challenge. This study proposes zero-padding for resizing images to the same size and compares it with the conventional approach of scaling images up (zooming in) using interpolation. Our study showed that zero-padding had no effect on the classification accuracy but considerably reduced the training time. The reason is that neighboring zero input units (pixels) will not activate their corresponding convolutional unit in the next layer. Therefore, the synaptic weights on outgoing links from input units do not need to be updated if they contain a zero value. Theoretical justification along with experimental endorsements are provided in this paper.

Journal ArticleDOI
TL;DR: A novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views and it is indicated that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI).
Abstract: In this paper, a novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We indicate that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI). The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling. Directly upsampling or super-resolving the light field in the angular dimensions causes ghosting effects. To suppress these ghosting effects, we contribute a novel “blur-restoration-deblur” framework. First, the “blur” step is applied to extract the low-frequency components of the light field in the spatial dimensions by convolving each EPI slice with a selected blur kernel. Then, the “restoration” step is implemented by a CNN, which is trained to restore the angular details of the EPI. Finally, we use a non-blind “deblur” operation to recover the spatial high frequencies suppressed by the EPI blur. We evaluate our approach on several datasets, including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms. We further show extended applications, including depth enhancement and interpolation for unstructured input. More importantly, a novel rendering approach is presented by combining the proposed framework and depth information to handle large disparities.

Posted Content
TL;DR: DIB-R as discussed by the authors is a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image, which allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates.
Abstract: Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present {\emph DIB-R}, a differentiable rendering framework which allows gradients to be analytically computed for all pixels in an image. Key to our approach is to view foreground rasterization as a weighted interpolation of local properties and background rasterization as a distance-based aggregation of global geometry. Our approach allows for accurate optimization over vertex positions, colors, normals, light directions and texture coordinates through a variety of lighting models. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively using 2D supervision. Our project website is: this https URL

Journal ArticleDOI
TL;DR: In this article, the authors evaluated transferability of models produced using 11 ENM algorithms from the perspective of interpolation and extrapolation in a virtual species framework and defined fundamental niches and potential distributions of 16 virtual species distributed across Eurasia.
Abstract: Ecological niche modeling (ENM) is used widely to study species’ geographic distributions. ENM applications frequently involve transferring models calibrated with environmental data from one region to other regions or times that may include novel environmental conditions. When novel conditions are present, transferability implies extrapolation, whereas, in absence of such conditions, transferability is an interpolation step only. We evaluated transferability of models produced using 11 ENM algorithms from the perspective of interpolation and extrapolation in a virtual species framework. We defined fundamental niches and potential distributions of 16 virtual species distributed across Eurasia. To simulate real situations of incomplete understanding of species’ distribution or existing fundamental niche (environmental conditions suitable for the species contained in the study area; N*F), we divided Eurasia into six regions and used 1–5 regions for model calibration and the rest for model evaluation. The models produced with the 11 ENM algorithms were evaluated in environmental space, to complement the traditional geographic evaluation of models. None of the algorithms accurately estimated the existing fundamental niche (N*F) given one region in calibration, and model evaluation scores decreased as the novelty of the environments in the evaluation regions increased. Thus, we recommend quantifying environmental similarity between calibration and transfer regions prior to model transfer, providing an avenue for assessing uncertainty of model transferability. Different algorithms had different sensitivity to completeness of knowledge of N*F, with implications for algorithm selection. If the goal is to reconstruct fundamental niches, users should choose algorithms with limited extrapolation when N*F is well known, or choose algorithms with increased extrapolation when N*F is poorly known. Our assessment can inform applications of ecological niche modeling transference to anticipate species invasions into novel areas, disease emergence in new regions, and forecasts of species distributions under future climate conditions.

Journal ArticleDOI
TL;DR: In this article, the impact and sensitivity of void interpolation methods on estimates of volume change was evaluated using two spatially complete, high-resolution digital elevation models over southeast Alaska, USA.
Abstract: . Glacier mass balance has been estimated on individual glacier and regional scales using repeat digital elevation models (DEMs). DEMs often have gaps in coverage (“voids”), the properties of which depend on the nature of the sensor used and the surface being measured. The way that these voids are accounted for has a direct impact on the estimate of geodetic glacier mass balance, though a systematic comparison of different proposed methods has been heretofore lacking. In this study, we determine the impact and sensitivity of void interpolation methods on estimates of volume change. Using two spatially complete, high-resolution DEMs over southeast Alaska, USA, we artificially generate voids in one of the DEMs using correlation values derived from photogrammetric processing of Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) scenes. We then compare 11 different void interpolation methods on a glacier-by-glacier and regional basis. We find that a few methods introduce biases of up to 20 % in the regional results, while other methods give results very close ( % difference) to the true, non-voided volume change estimates. By comparing results from a few of the best-performing methods, an estimate of the uncertainty introduced by interpolating voids can be obtained. Finally, by increasing the number of voids, we show that with these best-performing methods, reliable estimates of glacier-wide volume change can be obtained, even with sparse DEM coverage.