In this article, we consider deep learning strategies in ultrasound systems, from the front end to advanced applications. Our goal is to provide the reader with a broad understanding of the possible impact of deep learning methodologies on many aspects of ultrasound imaging. In particular, we discuss methods that lie at the interface of signal acquisition and machine learning, exploiting both data structure (e.g., sparsity in some domain) and data dimensionality (big data) already at the raw radio-frequency channel stage. As some examples, we outline efficient and effective deep learning solutions for adaptive beamforming and adaptive spectral Doppler through artificial agents, learn compressive encodings for the color Doppler, and provide a framework for structured signal recovery by learning fast approximations of iterative minimization problems, with applications to clutter suppression and super-resolution ultrasound. These emerging technologies may have a considerable impact on ultrasound imaging, showing promise across key components in the receive processing chain.

Deep Learning in Ultrasound Imaging

In ultrasound (US) imaging, various types of adaptive beamforming techniques have been investigated to improve the resolution and the contrast-to-noise ratio of the delay and sum (DAS) beamformers. Unfortunately, the performance of these adaptive beamforming approaches degrades when the underlying model is not sufficiently accurate and the number of channels decreases. To address this problem, here, we propose a deep-learning-based beamformer to generate significantly improved images over widely varying measurement conditions and channel subsampling patterns. In particular, our deep neural network is designed to directly process full or subsampled radio frequency (RF) data acquired at various subsampling rates and detector configurations so that it can generate high-quality US images using a single beamformer. The origin of such input-dependent adaptivity is also theoretically analyzed. Experimental results using the B-mode focused US confirm the efficacy of the proposed methods.

Adaptive and Compressive Beamforming Using Deep Learning for Medical Ultrasound

Single plane wave transmissions are promising for automated imaging tasks requiring high ultrasound frame rates over an extended field of view. However, a single plane wave insonification typically produces suboptimal image quality. To address this limitation, we are exploring the use of deep neural networks (DNNs) as an alternative to delay-and-sum (DAS) beamforming. The objectives of this work are to obtain information directly from raw channel data and to simultaneously generate both a segmentation map for automated ultrasound tasks and a corresponding ultrasound B-mode image for interpretable supervision of the automation. We focus on visualizing and segmenting anechoic targets surrounded by tissue and ignoring or deemphasizing less important surrounding structures. DNNs trained with Field II simulations were tested with simulated, experimental phantom, and in vivo data sets that were not included during training. With unfocused input channel data (i.e., prior to the application of receive time delays), simulated, experimental phantom, and in vivo test data sets achieved mean ± standard deviation Dice similarity coefficients of 0.92 ± 0.13, 0.92 ± 0.03, and 0.77 ± 0.07, respectively, and generalized contrast-to-noise ratios (gCNRs) of 0.95 ± 0.08, 0.93 ± 0.08, and 0.75 ± 0.14, respectively. With subaperture beamformed channel data and a modification to the input layer of the DNN architecture to accept these data, the fidelity of image reconstruction increased (e.g., mean gCNR of multiple acquisitions of two in vivo breast cysts ranged 0.89–0.96), but DNN display frame rates were reduced from 395 to 287 Hz. Overall, the DNNs successfully translated feature representations learned from simulated data to phantom and in vivo data, which is promising for this novel approach to simultaneous ultrasound image formation and segmentation.

/pdf/deep-learning-to-obtain-simultaneous-image-and-segmentation-2zgmdnxarv.pdf

Deep Learning to Obtain Simultaneous Image and Segmentation Outputs From a Single Input of Raw Ultrasound Channel Data

Optical coherence tomography (OCT) images of the retina are a powerful tool for diagnosing and monitoring eye disease. However, they are plagued by speckle noise, which reduces image quality and reliability of assessment. This paper introduces a novel speckle reduction method inspired by the recent successes of deep learning in medical imaging. We present two versions of the network to reflect the needs and preferences of different end-users. Specifically, we train a convolution neural network to denoise cross-sections from OCT volumes of healthy eyes using either (1) mean-squared error, or (2) a generative adversarial network (GAN) with Wasserstein distance and perceptual similarity. We then interrogate the success of both methods with extensive quantitative and qualitative metrics on cross-sections from both healthy and glaucomatous eyes. The results show that the former approach provides state-of-the-art improvement in quantitative metrics such as PSNR and SSIM, and aids layer segmentation. However, the latter approach, which puts more weight on visual perception, outperformed for qualitative comparisons based on accuracy, clarity, and personal preference. Overall, our results demonstrate the effectiveness and efficiency of a deep learning approach to denoising OCT images, while maintaining subtle details in the images.

Retinal optical coherence tomography image enhancement via deep learning.

By localizing microbubbles (MBs) in the vasculature, ultrasound localization microscopy (ULM) has recently been proposed, which greatly improves the spatial resolution of ultrasound (US) imaging and will be helpful for clinical diagnosis. Nevertheless, several challenges remain in fast ULM imaging. The main problems are that current localization methods used to implement fast ULM imaging, e.g., a previously reported localization method based on sparse recovery (CS-ULM), suffer from long data-processing time and exhaustive parameter tuning (optimization). To address these problems, in this paper, we propose a ULM method based on deep learning, which is achieved by using a modified sub-pixel convolutional neural network (CNN), termed as mSPCN-ULM. Simulations and in vivo experiments are performed to evaluate the performance of mSPCN-ULM. Simulation results show that even if under high-density condition (6.4 MBs/mm2), a high localization precision ( $\sim 28\mu \text{m}$ in the lateral direction and $\sim 24\mu \text{m}$ in the axial direction) and a high localization reliability (Jaccard index of 0.66) can be obtained by mSPCN-ULM, compared to CS-ULM. The in vivo experimental results indicate that with plane wave scan at a transmit center frequency of 15.625 MHz, microvessels with diameters of $\sim 17\mu \text{m}$ can be detected and adjacent microvessels with a distance of $\sim 42\mu \text{m}$ can be separated. Furthermore, when using GPU acceleration, the data-processing time of mSPCN-ULM can be shortened to ~6 sec/frame in the simulations and ~23 sec/frame in the in vivo experiments, which is 3–4 orders of magnitude faster than CS-ULM. Finally, once the network is trained, mSPCN-ULM does not need parameter tuning to implement ULM. As a result, mSPCN-ULM opens the door to implement ULM with fast data-processing speed, high imaging accuracy, short data-acquisition time, and high flexibility (robustness to parameters) characteristics.

Deep Learning for Ultrasound Localization Microscopy

Compressed sensing (CS) has drawn many interest in the field ultrasound (US) image recovery. It has demonstrated promising results in the recovery of radio-frequency element raw-data [Liebgott et. al. ULTRAS13, Besson et. al. SPARS17]. The objective of such approaches is to recover the raw-data from undersampled random measurements. It is achieved by means of convex optimization or greedy methods which require a high number of iterations to converge and whose accuracy highly depends on hyper-parameters fine-tuning. Recently, deep-neural networks (DNN) has redefined the paradigm of signal recovery, leading to remarkable results for CS reconstruction of natural images [Mousavi et. al. ALBERTON15].

A deep learning approach to ultrasound image recovery

Conventional ultrasound (US) image reconstruction methods rely on delay-and-sum (DAS) beamforming, which is a relatively poor solution to the image reconstruction problem. An alternative to DAS consists in using iterative techniques, which require both an accurate measurement model and a strong prior on the image under scrutiny. Toward this goal, much effort has been deployed in formulating models for US imaging, which usually require a large amount of memory to store the matrix coefficients. We present two different techniques, which take advantage of fast and matrix-free formulations derived for the measurement model and its adjoint, and rely on sparsity of US images in well-chosen models. Sparse regularization is used for enhanced image reconstruction. Compressed beamforming exploits the compressed sensing framework to restore high-quality images from fewer raw data than state-of-the-art approaches. Using simulated data and in vivo experimental acquisitions, we show that the proposed approach is three orders of magnitude faster than non-DAS state-of-the-art methods, with comparable or better image quality.

/pdf/ultrafast-ultrasound-imaging-as-an-inverse-problem-matrix-20px53ozy6.pdf

Ultrafast Ultrasound Imaging as an Inverse Problem: Matrix-Free Sparse Image Reconstruction

The problem of improving image quality in ultrafast ultrasound (US) imaging by means of regularized iterative algorithms has raised a vast interest in the US community. These approaches usually rely on standard image processing priors, such as wavelet sparsity, which are of limited efficacy in the context of US imaging. Moreover, the high computational complexity of iterative approaches make them difficult to deploy in real-time applications. We propose an approach which relies on a convolutional neural network trained exclusively on a simulated dataset for the purpose of improving images reconstructed from a single plane wave (PW) insonification. We provide extensive results on numerical and in vivo data from the plane wave imaging challenge (PICMUS). We show that the proposed approach can be applied in real-time settings, with an increase in contrast-to-noise ratio of more than 8.4 dB and an improvement of the lateral resolution by at least 25 %.

Deep Convolutional Neural Network for Ultrasound Image Enhancement

Ultrafast ultrasound (US) revolutionized biomedical imaging with its capability of acquiring full-view frames at over 1 kHz, unlocking breakthrough modalities such as shear-wave elastography and functional US neuroimaging. Yet, it suffers from strong diffraction artifacts, mainly caused by grating lobes, side lobes, or edge waves. Multiple acquisitions are typically required to obtain a sufficient image quality, at the cost of a reduced frame rate. To answer the increasing demand for high-quality imaging from single-shot acquisitions, we propose a two-step convolutional neural network (CNN)-based image reconstruction method, compatible with real-time imaging. A low-quality estimate is obtained by means of a backprojection-based operation, akin to conventional delay-and-sum beamforming, from which a high-quality image is restored using a residual CNN with multi-scale and multi-channel filtering properties, trained specifically to remove the diffraction artifacts inherent to ultrafast US imaging. To account for both the high dynamic range and the radio frequency property of US images, we introduce the mean signed logarithmic absolute error (MSLAE) as training loss function. Experiments were conducted with a linear transducer array, in single plane wave (PW) imaging. Trainings were performed on a simulated dataset, crafted to contain a wide diversity of structures and echogenicities. Extensive numerical evaluations demonstrate that the proposed approach can reconstruct images from single PWs with a quality similar to that of gold-standard synthetic aperture imaging, on a dynamic range in excess of 60 dB. In vitro and in vivo experiments show that trainings performed on simulated data translate well to experimental settings.

CNN-Based Image Reconstruction Method for Ultrafast Ultrasound Imaging

While restoration methods have been extensively studied in ultrasound (US) imaging, only few recent works have focused on modeling and understanding the blur from a physical point of view even in simple configurations, such as lossless homogeneous media. Despite a highly nonstationary blur due to diffraction effects, many techniques rely on simplistic approximations based on shift-invariant models or sectional methods and their efficiency has not been demonstrated for plane-wave (PW) and diverging-wave (DW) imaging. In this paper, we first propose a physical model of nonstationary blur in the context of PW and DW imaging. The blur operation is expressed as a composition of a U.S. propagation operator and a delay-and-sum operator, each of which has derived in previous works. We show that such a composition leads to a standard model of nonstationary blur as a Fredholm integral of the first kind. Second, we describe an approximation of the blur in the discrete domain based on the above decomposition coupled with an appropriate discretization of the latent element-raw-data space. We show theoretically and empirically that its evaluation, using such an approximation, scales linearly instead of quadratically with respect to the grid size, better than shift-invariant approaches. Through simulations and in vivo experimental data, we demonstrate that using the proposed model in the context of maximum-a-posteriori image restoration results in a higher image quality than using state-of-the-art shift-invariant models.

/pdf/a-physical-model-of-nonstationary-blur-in-ultrasound-imaging-49nkp7ppz6.pdf

Dimitris Perdios

Papers

A deep learning approach to ultrasound image recovery

Ultrafast Ultrasound Imaging as an Inverse Problem: Matrix-Free Sparse Image Reconstruction

Deep Convolutional Neural Network for Ultrasound Image Enhancement

CNN-Based Image Reconstruction Method for Ultrafast Ultrasound Imaging

A Physical Model of Nonstationary Blur in Ultrasound Imaging