Foundations Of Image Science

A variety of deep neural network (DNN)-based image denoising methods have been proposed for use with medical images. Traditional measures of image quality (IQ) have been employed to optimize and evaluate these methods. However, the objective evaluation of IQ for the DNN-based denoising methods remains largely lacking. In this work, we evaluate the performance of DNN-based denoising methods by use of task-based IQ measures. Specifically, binary signal detection tasks under signal-known-exactly (SKE) with background-known-statistically (BKS) conditions are considered. The performance of the ideal observer (IO) and common linear numerical observers are quantified and detection efficiencies are computed to assess the impact of the denoising operation on task performance. The numerical results indicate that, in the cases considered, the application of a denoising network can result in a loss of task-relevant information in the image. The impact of the depth of the denoising networks on task performance is also assessed. The presented results highlight the need for the objective evaluation of IQ for DNN-based denoising technologies and may suggest future avenues for improving their effectiveness in medical imaging applications.

/pdf/assessing-the-impact-of-deep-neural-network-based-image-2apv7yc3is.pdf

Assessing the Impact of Deep Neural Network-Based Image Denoising on Binary Signal Detection Tasks

Artificial intelligence-based methods are showing promise in medical imaging applications. There is substantial interest in clinical translation of these methods, requiring that they be evaluated rigorously. We lay out a framework for objective task-based evaluation of artificial intelligence methods. We provide a list of available tools to conduct this evaluation. We outline the important role of physicians in conducting these evaluation studies. The examples in this article are proposed in the context of PET scans with a focus on evaluating neural network-based methods. However, the framework is also applicable to evaluate other medical imaging modalities and other types of artificial intelligence methods.

Objective Task-Based Evaluation of Artificial Intelligence-Based Medical Imaging Methods:: Framework, Strategies, and Role of the Physician.

The ideal observer (IO) sets an upper performance limit among all observers and has been advocated for use in assessing and optimizing imaging systems. For joint detection-estimation tasks, the estimation ROC (EROC) curve has been proposed for evaluating the performance of observers. However, in practice, it is generally difficult to accurately approximate the IO that maximizes the area under the EROC curve (AEROC) for a general detection-estimation task. In this study, a hybrid method that employs machine learning is proposed to accomplish this. Specifically, a hybrid approach is developed that combines a multi-task convolutional neural network (CNN) and a Markov-Chain Monte Carlo (MCMC) method in order to approximate the IO for detectionestimation tasks. The multi-task CNN is designed to estimate the likelihood ratio and the parameter vector, while the MCMC method is employed to compute the utility-weighted posterior mean of the parameter vector. The IO test statistic is subsequently formed as the product of the likelihood ratio and the posterior mean of the parameter vector. Computer simulation studies were conducted to validate the proposed method, which include backgroundknown-exactly (BKE) and background-known-statistically (BKS) tasks. The proposed method provides a new approach for approximating the IO and may enable the application of EROC analysis for optimizing imaging systems.

Supervised learning-based ideal observer approximation for joint detection and estimation tasks

Multiple objective assessment of image-quality (OAIQ)-based studies have reported that several deep-learning (DL)-based denoising methods show limited performance on signal-detection tasks. Our goal was to investigate the reasons for this limited performance. To achieve this goal, we conducted a task-based characterization of a DLbased denoising approach for individual signal properties. We conducted this study in the context of evaluating a DL-based approach for denoising single photon-emission computed tomography (SPECT) images. The training data consisted of signals of different sizes and shapes within a clustered-lumpy background, imaged with a 2D parallel-hole-collimator SPECT system. The projections were generated at normal and 20% low-dose level, both of which were reconstructed using an ordered-subset-expectation-maximization (OSEM) algorithm. A convolutional neural network (CNN)-based denoiser was trained to denoise the low-dose images. The performance of this CNN was characterized for five different signal sizes and four different signal-to-background ratio (SBRs) by designing each evaluation as a signal-known-exactly/background-known-statistically (SKE/BKS) signal-detection task. Performance on this task was evaluated using an anthropomorphic channelized Hotelling observer (CHO). We observed that the DL-based denoising approach did not improve performance on the signal-detection task for any of the signal types. None of signal types did the DL-based denoising approach improve performance. Further investigations revealed a drop in the mean of the differences between the signal-present and the signal-absent images, resulting in this limited performance on detection tasks. Additionally, it was observed that evaluation with fidelity-based figures of merit (root mean square error and structural similarity index) directly contradicted the observer study findings for all signals. Overall, these results provide new insights on the performance of the DL-based denoising approach as a function of signal size and contrast. More generally, the observer study-based characterization provides a mechanism to evaluate the sensitivity of the method to specific object properties, and may be explored as analogous to characterizations such as modulation transfer function for linear systems. Finally, this work underscores the need for objective task-based evaluation of DL-based denoising approaches.

/pdf/investigating-the-limited-performance-of-a-deep-learning-1y72uyr2.pdf

Investigating the limited performance of a deep-learning-based SPECT denoising approach: an observer-study-based characterization

Deep neural network (DNN)-based image denoising methods have been proposed for use with medical images. These methods are commonly optimized and evaluated by use of traditional physical measures of image quality (IQ). However, the objective evaluation of IQ for such methods remains largely lacking. In this study, task-based IQ measures are used to evaluate the performance of DNN-based denoising methods. Specifically, we consider signal detection tasks under background-known-statistically conditions. The performance of the ideal observer (IO) and the Hotelling observer (HO) are quantified and detection efficiencies are computed to investigate the impact of the denoising operation on task performance. The experimental results show that, in the cases considered, the application of a denoising network generally results in a loss of task-relevant information. The impact of the depth of the denoising networks on task performance is also assessed. While mean squared error improved as the network depths were increased, signal detection performance degraded. These results highlight the need for the objective evaluation of IQ for DNN-based denoising technologies and may suggest future avenues for improving their effectiveness in medical imaging applications.

Task-based performance evaluation of deep neural network-based image denoising

A variety of deep neural network (DNN)-based image denoising methods have been proposed for use with medical images. These methods are typically trained by minimizing loss functions that quantify a distance between the denoised image and the defined target image (e.g., a noise-free or low noise image). They have demonstrated high performance in terms of traditional image quality metrics such as root mean square error (RMSE), structural similarity index metric (SSIM), or peak signal-to-noise ratio (PSNR). However, it has been reported that these denoising methods may not improve the objective measures of image quality (IQ). In this work, a task-informed model training method that preserves task-specific information is established and systematically evaluated with clinical realistic simulated low-dose X-ray computed tomography (CT) images. Specifically, binary signal detection tasks under signal-known-statistically (SKS) with background-known-statistically (BKS) conditions are considered. The low-dose CT denoising networks are first pretrained by use of a mean-square-error (MSE) loss function. A fully connected layer with a sigmoid activation function is subsequently appended to the denoising network, which can be interpreted as a single layer neural network-based numerical observer (SLNN-NO). A hybrid loss function consisting of a binary cross-entropy loss function and mean square loss function is employed to jointly fine-tune the denoising network and train the SLNN-NO. The performance of the SLNN-NO on denoised data is quantified to evaluate the impact of the task-informed training procedure on the denoising network. The presented results indicate that the task-informed training method can improve observer performance while providing control over the trade off between traditional and task-based measures of image quality.

A task-informed model training method for deep neural network-based image denoising

The use of convolutional neural networks (CNNs) for establishing anthropomorphic numerical observers (ANOs) is being actively explored. In these data-driven approaches, CNNs are trained in a standard supervised way with human-labeled training data; hence, the anthropomorphic component of this procedure resides only in the training labels. However, it is well-known that such traditionally trained CNNs can rely on image features that are highly specific to the training distribution and may not align with features exploited by human perception. While being able to predict human observer performance under certain specified conditions, traditionally-trained CNNs lack the interpretability and robustness that may be desired for an ANO. To address this, in this work we investigate the use of an adversarial robust training strategy for training CNN-based observers. As recently demonstrated in the computer vision literature, this training strategy can result in CNNs that exploit more human-interpretable features than would be employed by a standard CNN. Robustly trained CNNs are systematically investigated for performing a signal-known-exactly (SKE) and background-known-statistically (BKS) binary detection task. Additionally, a differential evolution-based optimization procedure is developed to establish robustly trained CNNs that achieve a specified performance, which may provide a new approach to establishing ANOs.

Kaiyan Li

Papers

Assessing the Impact of Deep Neural Network-Based Image Denoising on Binary Signal Detection Tasks

Supervised learning-based ideal observer approximation for joint detection and estimation tasks

Task-based performance evaluation of deep neural network-based image denoising

A task-informed model training method for deep neural network-based image denoising

Investigation of adversarial robust training for establishing interpretable CNN-based numerical observers