Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

/pdf/feature-pyramid-networks-for-object-detection-4b1j4aq4vv.pdf

Feature Pyramid Networks for Object Detection

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet/ .

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm. AlphaFold predicts protein structures with an accuracy competitive with experimental structures in the majority of cases using a novel deep learning architecture.

/pdf/highly-accurate-protein-structure-prediction-with-alphafold-2xjbzfwu3n.pdf

Highly accurate protein structure prediction with AlphaFold

Large-scale methods and robust algorithms are needed for a quantitative analysis of cells status/geometry in situ. It allows the understanding the cellular mechanisms that direct organ growth in response to internal and environmental cues. Using advanced whole-stack imaging in combination with pattern analysis, we have developed a new approach to investigate root zonation under different dark/light conditions. This method is based on the determination of 3 different parameters: cell length, cell volume and cell proliferation on the cell-layer level. This method allowed to build a precise quantitative 3D cell atlas of the Arabidopsis root tip. Using this approach we showed that the meristematic (proliferation) zone length differs between cell layers. Considering only the rapid increase of cortex cell length to determine the meristematic zone overestimates of the proliferation zone for epidermis/cortex and underestimates it for pericycle. The use of cell volume instead of cell length to define the meristematic zone correlates better with cell proliferation zone.

/pdf/light-dynamically-regulates-growth-rate-and-cellular-4p6xgdzbb8.pdf

Light dynamically regulates growth rate and cellular organisation of the Arabidopsis root meristem

Cryo-electron microscopy (cryo-EM) has revolutionized experimental protein structure determination. Despite advances in high resolution reconstruction, a majority of cryo-EM experiments provide either a single state of the studied macromolecule, or a relatively small number of its conformations. This reduces the effectiveness of the technique for proteins with flexible regions, which are known to play a key role in protein function. Recent methods for capturing conformational heterogeneity in cryo-EM data model it in volume space, making recovery of continuous atomic structures challenging. Here we present a fully deep-learning-based approach using variational auto-encoders (VAEs) to recover a continuous distribution of atomic protein structures and poses directly from picked particle images and demonstrate its efficacy on realistic simulated data. We hope that methods built on this work will allow incorporation of stronger prior information about protein structure and enable better understanding of non-rigid protein structures.

Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs.

We present a variational approach to simultaneously trace the axis and determine the thickness of 3-D (or 2-D) tubular structures defined by sparsely and unevenly sampled noisy surface points Many existing approaches try to solve the axis-tracing and the precise fitting in two subsequent steps In contrast to this our model is initialized with a small cylinder segment and converges to the final tubular structure in a single energy minimization using a gradient descent scheme The energy is based on the error of fit and simultaneously penalizes strong curvature and thickness variations We demonstrate the performance of this closed formulation on volumetric microscopic data sets of the Arabidopsis root tip, where only the nuclei of the cells are visible

/pdf/modeling-of-sparsely-sampled-tubular-surfaces-using-coupled-abz5rbif4h.pdf

Modeling of Sparsely Sampled Tubular Surfaces Using Coupled Curves

Abstract Several tissues contain cells with multiple motile cilia that generate a fluid or particle flow to support development and organ functions; defective motility causes human disease. Developmental cues orient motile cilia, but how cilia are locked into their final position to maintain a directional flow is not understood. Here we find that the actin cytoskeleton is highly dynamic during early development of multiciliated cells (MCCs). While apical actin bundles become increasingly more static, subapical actin filaments are nucleated from the distal tip of ciliary rootlets. Anchorage of these subapical actin filaments requires the presence of microridge-like structures formed during MCC development, and the activity of Nonmuscle Myosin II. Optogenetic manipulation of Ezrin, a core component of the microridge actin-anchoring complex, or inhibition of Myosin Light Chain Kinase interfere with rootlet anchorage and orientation. These observations identify microridge-like structures as an essential component of basal body rootlet anchoring in MCCs. 

/pdf/microridge-like-structures-anchor-motile-cilia-clbkt0ip.pdf

Microridge-like structures anchor motile cilia

We present an active surface model designed for the segmentation of Drosophila Schneider cell nuclei and nucleoli from wide-field microscopic data. The imaging technique as well as the biological application impose some major challenges to the segmentation. On the one hand, we have to deal with strong blurring of the 3D data, especially in z-direction. On the other hand, concerning the biological application, we have to deal with non-closed object boundaries and touching objects. To cope with these problems, we have designed a fully 3D active surface model. Our model prefers roundish object shapes and especially imposes roughly spherical surfaces where there is little gradient information. We have adapted an external force field for this model, which is based on gradient vector flow ($\text{\it{GVF}}$) and has a much larger capture range than standard $\text{\it{GVF}}$ force fields.

/pdf/a-3d-active-surface-model-for-the-accurate-segmentation-of-3ty5ms9hps.pdf

Olaf Ronneberger

Papers

Light dynamically regulates growth rate and cellular organisation of the Arabidopsis root meristem

Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs.

Modeling of Sparsely Sampled Tubular Surfaces Using Coupled Curves

Microridge-like structures anchor motile cilia

A 3D Active Surface Model for the Accurate Segmentation of Drosophila Schneider Cell Nuclei and Nucleoli