Transformers have dominated the field of natural language processing and have recently made an impact in the area of computer vision. In the field of medical image analysis, transformers have also been successfully used in to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. This paper aims to promote awareness of the applications of transformers in medical image analysis. Specifically, we first provide an overview of the core concepts of the attention mechanism built into transformers and other basic components. Second, we review various transformer architectures tailored for medical image applications and discuss their limitations. Within this review, we investigate key challenges including the use of transformers in different learning paradigms, improving model efficiency, and coupling with other techniques. We hope this review will provide a comprehensive picture of transformers to readers with an interest in medical image analysis. 

Transformers in Medical Image Analysis: A Review

Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

SSoB: searching a scene-oriented architecture for underwater object detection

Transformer-based models, capable of learning better global dependencies, have recently demonstrated exceptional representation learning capabilities in computer vision and medical image analysis. Transformer reformats the image into separate patches and realize global communication via the self-attention mechanism. However, positional information between patches is hard to preserve in such 1D sequences, and loss of it can lead to sub-optimal performance when dealing with large amounts of heterogeneous tissues of various sizes in 3D medical image segmentation. Additionally, current methods are not robust and efficient for heavy-duty medical segmentation tasks such as predicting a large number of tissue classes or modeling globally inter-connected tissues structures. Inspired by the nested hierarchical structures in vision transformer, we proposed a novel 3D medical image segmentation method (UNesT), employing a simplified and faster-converging transformer encoder design that achieves local communication among spatially adjacent patch sequences by aggregating them hierarchically. We extensively validate our method on multiple challenging datasets, consisting anatomies of 133 structures in brain, 14 organs in abdomen, 4 hierarchical components in kidney, and inter-connected kidney tumors). We show that UNesT consistently achieves state-of-the-art performance and evaluate its generalizability and data efficiency. Particularly, the model achieves whole brain segmentation task complete ROI with 133 tissue classes in single network, outperforms prior state-of-the-art method SLANT27 ensembled with 27 network tiles, our model performance increases the mean DSC score of the publicly available Colin and CANDI dataset from 0.7264 to 0.7444 and from 0.6968 to 0.7025, respectively.

UNesT: Local Spatial Representation Learning with Hierarchical Transformer for Efficient Medical Segmentation

A BSTRACT The transformer has dominated the natural language processing (NLP) ﬁeld for a long time. Recently, the transformer-based method is adopt into the computer vision (CV) ﬁeld and shows promising results. As an important branch of the CV ﬁeld, medical image analysis joins the wave of the transformer-based method rightfully. In this paper, we illustrate the principle of the attention mechanism, and the detailed structures of the transformer, and depict how the transformer is adopted into the CV ﬁeld. We organize the transformer-based medical image analysis applications in the sequence of different CV tasks, including classiﬁcation, segmentation, synthesis, registration, localization, detection, captioning, and denoising. For the mainstream classiﬁcation and segmentation tasks, we further divided the corresponding works based on different medical imaging modalities. We include thirteen modalities and more than twenty objects in our work. We also visualize the proportion that each modality and object occupy to give the readers an intuitive impression. We hope our work can contribute to the development of transformer-based medical image analysis in the future.

Medical image analysis based on transformer: A Review

Multiscale transunet +  + : dense hybrid U-Net with transformer for medical image segmentation

TransRA: transformer and residual attention fusion for single remote sensing image dehazing

Learning to remove sandstorm for image enhancement

Bistatic SAR uses separated transmitter and receiver flying on different platforms, and has the ability of the exploitation of additional information contained in the bistatic reflectivity of targets. Besides of technical problems-like the synchronization of the oscillators-the processing of bistatic raw data imaging is not well studied theoretically all the same, which is a major area in bistatic SAR. This paper firstly describes the imaging of bistatic SAR with special case of parallel flight paths of transmitter and receiver, and then provides an efficient extended chirp scaling (ECS) algorithm for arbitrary bistatic SAR configure, which is based on the bistatic-to-monostatic application (BTMA). The method described is a feasible solution for the image formation of bistatic SAR with arbitrary bistatic configure. The problem of Doppler ambiguity during the ECS processing is also discussed at last. This ECS imaging algorithm is verified with a simulation experiment.

Peng-shu Dong

Papers

Multiscale transunet + + : dense hybrid U-Net with transformer for medical image segmentation

TransRA: transformer and residual attention fusion for single remote sensing image dehazing

Learning to remove sandstorm for image enhancement

An Extended Chirp Scaling imaging algorithm for bistatic SAR with unparallel flight paths

Underwater image enhancement by maximum-likelihood based adaptive color correction and robust scattering removal