Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

/pdf/feature-pyramid-networks-for-object-detection-4b1j4aq4vv.pdf

Feature Pyramid Networks for Object Detection

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet/ .

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm. AlphaFold predicts protein structures with an accuracy competitive with experimental structures in the majority of cases using a novel deep learning architecture.

/pdf/highly-accurate-protein-structure-prediction-with-alphafold-2xjbzfwu3n.pdf

Highly accurate protein structure prediction with AlphaFold

Dominic Mai 1,3 maid@informatik.uni-freiburg.de Jasmin Durr 2 jasmin.duerr@biologie.uni-freiburg.de Klaus Palme 2,3 klaus.palme@biologie.uni-freiburg.de Olaf Ronneberger 1,3 ronneber@informatik.uni-freiburg.de Computer Science Department University of Freiburg Germany 2 Institute of Biology II Botany University of Freiburg Germany BIOSS Centre for Biological Signalling Studies University of Freiburg Germany

/pdf/improving-detection-of-deformable-objects-in-volumetric-data-2u2xvtqknm.pdf

Improving Detection of Deformable Objects in Volumetric Data

This report provides additional results on multiview deblurring and accompanies the work in [11]. In particular, we provide images of all data sets before and after reconstruction. Moreover, we give more insights into the point spread function (PSF) of the Single Plane Illumination Microscope (SPIM) theoretically and empirically. Finally, the influence of the number of views on the resolution in frequency space is discussed.

Multiview Deblurring for 3-D Images from Light Sheet based Fluorescence Microscopy - Supporting Online Material

Identification of Pollen in Microscopic Images by Pattern Recognition Techniques

3-D convolutional neural networks for organ segmentation in medical images for radiotherapy planning

In der gleichnamigen Dissertation [Ron07] wird die Entwicklung von 3D Invarianten zur Erkennung von biologischen Strukturen beschrieben. Diese Invarianten wurden mit Hilfe eines Schemas entwickelt, das ursprünglich von H. Schulz-Mirbach [SM95] ausgearbeitet wurde und auf der Haar-Integration beruht. Die benötigten Invarianzeigenschaften werden dabei durch die Integration über die entsprechende Transformationsgruppe wie z.B. Rotation und Translation erreicht. Verschiedene Erweiterungen des Haar-Integrations-Schemas werden beschrieben, die für die Anwendung auf 3D Volumendaten biologischer Strukturen benötigt werden. Die wichtigste Erweiterung ist dabei die Einführung eines Deformationsmodells, so dass die resultierenden Merkmale robust gegenüber elastischen Deformationen der betrachteten Strukturen werden, ohne aber feine relevante Strukturen zu unterdrücken. Ein weiterer wichtiger Aspekt ist die erreichte Robustheit gegenüber nicht-linearen Grauwert-Transformationen, die gewisse Veränderungen der Aufnahmebedingungen zwischen den Aufnahmen der Trainingsund der Test-Objekte erlaubt. Die beschriebenen Verfahren wurden erfolgreich für die Erkennung von Pollen eingesetzt. Dabei wurden sowohl auf reinen Laborproben als auch auf einem ”Real-World“-Datensatz von dem ersten Prototypen eines vollautomatischen Pollenmonitors Ergebnisse erzielt, die weit über dem internationalen Stand der Technik liegen.

Olaf Ronneberger

Papers

Improving Detection of Deformable Objects in Volumetric Data

Multiview Deblurring for 3-D Images from Light Sheet based Fluorescence Microscopy - Supporting Online Material

Identification of Pollen in Microscopic Images by Pattern Recognition Techniques

3-D convolutional neural networks for organ segmentation in medical images for radiotherapy planning

3D-Invarianten zur automatischen Pollenerkennung