Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

/pdf/feature-pyramid-networks-for-object-detection-4b1j4aq4vv.pdf

Feature Pyramid Networks for Object Detection

We present a novel and practical deep fully convolutional neural network architecture for semantic pixel-wise segmentation termed SegNet. This core trainable segmentation engine consists of an encoder network, a corresponding decoder network followed by a pixel-wise classification layer. The architecture of the encoder network is topologically identical to the 13 convolutional layers in the VGG16 network [1] . The role of the decoder network is to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise classification. The novelty of SegNet lies is in the manner in which the decoder upsamples its lower resolution input feature map(s). Specifically, the decoder uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. This eliminates the need for learning to upsample. The upsampled maps are sparse and are then convolved with trainable filters to produce dense feature maps. We compare our proposed architecture with the widely adopted FCN [2] and also with the well known DeepLab-LargeFOV [3] , DeconvNet [4] architectures. This comparison reveals the memory versus accuracy trade-off involved in achieving good segmentation performance. SegNet was primarily motivated by scene understanding applications. Hence, it is designed to be efficient both in terms of memory and computational time during inference. It is also significantly smaller in the number of trainable parameters than other competing architectures and can be trained end-to-end using stochastic gradient descent. We also performed a controlled benchmark of SegNet and other architectures on both road scenes and SUN RGB-D indoor scene segmentation tasks. These quantitative assessments show that SegNet provides good performance with competitive inference time and most efficient inference memory-wise as compared to other architectures. We also provide a Caffe implementation of SegNet and a web demo at http://mi.eng.cam.ac.uk/projects/segnet/ .

SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

/pdf/image-to-image-translation-with-conditional-adversarial-5058t24et2.pdf

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm. AlphaFold predicts protein structures with an accuracy competitive with experimental structures in the majority of cases using a novel deep learning architecture.

/pdf/highly-accurate-protein-structure-prediction-with-alphafold-2xjbzfwu3n.pdf

Highly accurate protein structure prediction with AlphaFold

Since biology and medicine apply increasingly fast volumetric imaging techniques and aim at extracting quantitative data from these images, the need for efficient image analysis techniques like detection and classification of 3D structures is obvious. A common approach is to extract local features, e.g. group integration has been used to gain invariance against rotation and translation. We extend these group integration features by including vectorial information and spherical harmonics descriptors. From our vectorial invariants we derive a very robust detector for spherical structures in low-quality images and show that it can be computed very fast. We apply these new invariants to 3D confocal laser-scanning microscope images of the Arabidopsis root tip and extract position and type of the cell nuclei. Then it is possible to build a biologically relevant, architectural model of the root tip.

Fast scalar and vectorial grayscale based invariant features for 3d cell nuclei localization and classification

In this paper, we present a novel approach for a trainable rotation invariant detection of complex structures in 3D microscopic multichannel data using a nonlinear filter approach. The basic idea of our approach is to compute local features in a window around each 3D position and map these features by means of a nonlinear mapping onto new local harmonic descriptors of the local window. These local harmonic descriptors are then combined in a linear way to form the output of the filter. The optimal combination of the computed local harmonic descriptors is determined in previous training step, and allows the filter to be adapted to an arbitrary structure depending on the problem at hand. Our approach is not limited to scalar-valued images and can also be used for vector-valued (multichannel) images such as gradient vector flow fields. We present realizations of a scalar-valued and a vector-valued multichannel filter. Our proposed algorithm was quantitatively evaluated on colorectal cancer cell lines (cells grown under controlled conditions), on which we successfully detected complex 3D mitotic structures. For a qualitative evaluation we tested our algorithms on human 3D tissue samples of colorectal cancer. We compare our results with a steerable filter approach as well as a morphology-based approach.

Harmonic Filters for 3D Multichannel Data: Rotation Invariant Detection of Mitoses in Colorectal Cancer

In this article we present an approach for a precise segmentation of spherical particles in transmitted light image stacks. A main goal was its fast operation and a high robustness to occlusions and agglomerations of the particles. The system is based on a voting procedure that finds the centers and radii of the particles and a subsequent precise segmentation with an active contour approach. To meet the demands of an online pollenmonitor for high speed and low memory consumption a multi-scale approach was applied. The proposed techniques successfully segmented the pollen grains in a vast amount of different air samples (about 2.7TB of raw data). The results on one of the most cluttered samples are presented in this paper.

/pdf/fast-and-robust-segmentation-of-spherical-particles-in-3aympkww37.pdf

Fast and robust segmentation of spherical particles in volumetric data sets from brightfield microscopy

Radiotherapy is one of the main ways head and neck cancers are treated;
radiation is used to kill cancerous cells and prevent their recurrence.
Complex treatment planning is required to ensure that enough radiation is given
to the tumour, and little to other sensitive structures (known as organs at risk)
such as the eyes and nerves which might otherwise be damaged. This is
especially difficult in the head and neck, where multiple at-risk structures often
lie in extremely close proximity to the tumour. It can take radiotherapy experts
four hours or more to pick out the important areas on planning scans (known as
segmentation).
This research will focus on applying machine learning algorithms to automatic
segmentation of head and neck planning computed tomography (CT) and
magnetic resonance imaging (MRI) scans at University College London
Hospital NHS Foundation Trust patients. Through analysis of the images used
in radiotherapy DeepMind Health will investigate improvements in efficiency of
cancer treatment pathways.

/pdf/applying-machine-learning-to-automated-segmentation-of-head-nxqkw7ju22.pdf

Olaf Ronneberger

Papers

Fast scalar and vectorial grayscale based invariant features for 3d cell nuclei localization and classification

Harmonic Filters for 3D Multichannel Data: Rotation Invariant Detection of Mitoses in Colorectal Cancer

Fast and robust segmentation of spherical particles in volumetric data sets from brightfield microscopy

Applying machine learning to automated segmentation of head and neck tumour volumes and organs at risk on radiotherapy planning CT and MRI scans

Generalizable medical image analysis using segmentation and classification neural networks