Recent progress on Transformers and multilayer perceptron (MLP) models provide new network architectural designs for computer vision tasks. Although these models proved to be effective in many vision tasks such as image recognition, there remain challenges in adapting them for lowlevel vision. The inflexibility to support high-resolution images and limitations of local attention are perhaps the main bottlenecks. In this work, we present a multi-axis MLP based architecture called MAXIM, that can serve as an efficient and flexible general-purpose vision backbone for image processing tasks. MAXIM uses a UNet-shaped hierarchical structure and supports long-range interactions enabled by spatially-gated MLPs. Specifically, MAXIM contains two MLP-based building blocks: a multi-axis gated MLP that allows for efficient and scalable spatial mixing of local and global visual cues, and a cross-gating block, an alternative to cross-attention, which accounts for cross-feature conditioning. Both these modules are exclusively based on MLPs, but also benefit from being both global and ‘fully-convolutional’, two properties that are desirable for image processing. Our extensive experimental results show that the proposed MAXIM model achieves state-of-the-art performance on more than ten benchmarks across a range of image processing tasks, including denoising, deblurring, de raining, dehazing, and enhancement while requiring fewer or comparable numbers of parameters and FLOPs than competitive models. The source code and trained models will be available at https://github.com/google-research/maxim.

MAXIM: Multi-Axis MLP for Image Processing

We propose a weakly supervised approach to semantic segmentation using bounding box annotations. Bounding boxes are treated as noisy labels for the foreground objects. We predict a per-class attention map that saliently guides the per-pixel cross entropy loss to focus on foreground pixels and refines the segmentation boundaries. This avoids propagating erroneous gradients due to incorrect foreground labels on the background. Additionally, we learn pixel embeddings to simultaneously optimize for high intra-class feature affinity while increasing discrimination between features across different classes. Our method, Box2Seg, achieves state-of-the-art segmentation accuracy on PASCAL VOC 2012 by significantly improving the mIOU metric by \(2.1\%\) compared to previous weakly supervised approaches. Our weakly supervised approach is comparable to the recent fully supervised methods when fine-tuned with limited amount of pixel-level annotations. Qualitative results and ablation studies show the benefit of different loss terms on the overall performance.

/pdf/box2seg-attention-weighted-loss-and-discriminative-feature-25shjcu327.pdf

Box2Seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation

Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

Recent progress on Transformers and multilayer perceptron (MLP) models provide new network architectural designs for computer vision tasks. Although these models proved to be effective in many vision tasks such as image recognition, there remain challenges in adapting them for lowlevel vision. The inflexibility to support high-resolution images and limitations of local attention are perhaps the main bottlenecks. In this work, we present a multi-axis MLP based architecture called MAXIM, that can serve as an efficient and flexible general-purpose vision backbone for image processing tasks. MAXIM uses a UNet-shaped hierarchical structure and supports long-range interactions enabled by spatially-gated MLPs. Specifically, MAXIM contains two MLP-based building blocks: a multi-axis gated MLP that allows for efficient and scalable spatial mixing of local and global visual cues, and a cross-gating block, an alternative to cross-attention, which accounts for cross-feature conditioning. Both these modules are exclusively based on MLPs, but also benefit from being both global and ‘fully-convolutional’, two properties that are desirable for image processing. Our extensive experimental results show that the proposed MAXIM model achieves state-of-the-art performance on more than ten benchmarks across a range of image processing tasks, including denoising, deblurring, de raining, dehazing, and enhancement while requiring fewer or comparable numbers of parameters and FLOPs than competitive models. The source code and trained models will be available at https://github.com/google-research/maxim. 

High-level Prior-based Loss Functions for Medical Image Segmentation: A Survey

Medical image segmentation remains particularly challenging for complex and low-contrast anatomical structures. In this paper, we introduce the U-Transformer network, which combines a U-shaped architecture for image segmentation with self-and cross-attention from Transformers. U-Transformer overcomes the inability of U-Nets to model long-range contextual interactions and spatial dependencies, which are arguably crucial for accurate segmentation in challenging contexts. To this end, attention mechanisms are incorporated at two main levels: a self-attention module leverages global interactions between encoder features, while cross-attention in the skip connections allows a fine spatial recovery in the U-Net decoder by filtering out non-semantic features. Experiments on two abdominal CT-image datasets show the large performance gain brought out by U-Transformer compared to U-Net and local Attention U-Nets. We also highlight the importance of using both self-and cross-attention, and the nice interpretability features brought out by U-Transformer.

https://hal.archives-ouvertes.fr/hal-03337089/document

U-Net Transformer: Self and Cross Attention for Medical Image Segmentation

Annotation of medical images for semantic segmentation is a very time consuming and difficult task Moreover, clinical experts often focus on specific anatomical structures and thus, produce partially annotated images In this paper, we introduce SMILE, a new deep convolutional neural network which addresses the issue of learning with incomplete ground truth SMILE aims to identify ambiguous labels in order to ignore them during training, and don’t propagate incorrect or noisy information A second contribution is SMILEr which uses SMILE as initialization for automatically relabeling missing annotations, using a curriculum strategy Experiments on 3 organ classes (liver, stomach, pancreas) show the relevance of the proposed approach for semantic segmentation: with 70% of missing annotations, SMILEr performs similarly as a baseline trained with complete ground truth annotations

/pdf/handling-missing-annotations-for-semantic-segmentation-with-4zdqpc3pv4.pdf

Handling Missing Annotations for Semantic Segmentation with Deep ConvNets

https://hal.archives-ouvertes.fr/hal-03243619/document

Iterative confidence relabeling with deep ConvNets for organ segmentation with partial labels.

Organ segmentation in CT-scans is a major issue in medical image analysis. Recently deep learning methods have brought impressive results for the task of semantic segmentation (Chen et al., 2018; Ronneberger et al., 2015; Long et al., 2015). However it remains very challenging especially due to the visual ambiguities (Kohl et al., 2018). The local visual context is insufficient and including external knowledge could help obtaining a better segmentation. In this paper we address the problem of including prior information about the shape and spatial position of the organs to improve the performance of semantic segmentation. It is particularly relevant in medical imaging where there is conventions on the structure of the images. For example, the position of the patient is always the same (spine at the bottom). The main changes reside on the patient anatomy which slightly varies from one to another. Including such a knowledge by biasing the predictions of Fully Convolutional Networks (FCN) is a difficult problem. Due to weight sharing and pooling operations, FCN models focus on local structures and loose the input position information. Moreover, it has also been highlighted that ConvNets have difficulties succeeding at simple spatial prediction tasks like coordinate transform (Liu et al., 2018). In the medical imaging literature, some methods have been proposed to incorporate such kind of information. For example cascaded networks (Roth et al., 2018) rely on selecting a Region of Interest (RoI) by a first model, which is then finely segmented by a second model. Those methods are efficient but rely on the good localization of the RoI. As a consequence, each step limits the overall performances. Other methods try to incorporate spatial information implicitly. For instance by generating distance maps with a GAN (Trullo et al., 2019) or by learning attention maps through an attention mechanism (Oktay et al., 2018). In those approaches, the spatial information is learned during training so we have no control of what is actually learned and how it biased the prediction.

/pdf/biasing-deep-convnets-for-semantic-segmentation-of-medical-1st24mrwmf.pdf

Biasing Deep ConvNets for Semantic Segmentation of Medical Images with a Prior-driven Prediction Function

Medical image segmentation remains particularly challenging for complex and low-contrast anatomical structures. In this paper, we introduce the U-Transformer network, which combines a U-shaped architecture for image segmentation with self- and cross-attention from Transformers. U-Transformer overcomes the inability of U-Nets to model long-range contextual interactions and spatial dependencies, which are arguably crucial for accurate segmentation in challenging contexts. To this end, attention mechanisms are incorporated at two main levels: a self-attention module leverages global interactions between encoder features, while cross-attention in the skip connections allows a fine spatial recovery in the U-Net decoder by filtering out non-semantic features. Experiments on two abdominal CT-image datasets show the large performance gain brought out by U-Transformer compared to U-Net and local Attention U-Nets. We also highlight the importance of using both self- and cross-attention, and the nice interpretability features brought out by U-Transformer.

Olivier Petit

Papers

U-Net Transformer: Self and Cross Attention for Medical Image Segmentation

Handling Missing Annotations for Semantic Segmentation with Deep ConvNets

Iterative confidence relabeling with deep ConvNets for organ segmentation with partial labels.

Biasing Deep ConvNets for Semantic Segmentation of Medical Images with a Prior-driven Prediction Function

U-Net Transformer: Self and Cross Attention for Medical Image Segmentation