Review of medical image quality assessment

Medical imaging used to be primarily within the domain of radiology, but with the advent of virtual pathology slides and telemedicine, imaging technology is expanding in the healthcare enterprise. As new imaging technologies are developed, they must be evaluated to assess the impact and benefit on patient care. The authors review the hierarchical model of the efficacy of diagnostic imaging systems by Fryback and Thornbury Med. Decis. Making 11, 88‐94 1991 as a guiding principle for system evaluation. Evaluation of medical imaging systems encompasses everything from the hardware and software used to acquire, store, and transmit images to the presentation of images to the interpreting clinician. Evaluation of medical imaging systems can take many forms, from the purely technical e.g., patient dose measurement to the increasingly complex e.g., determining whether a new imaging method saves lives and benefits society. Evaluation methodologies cover a broad range, from receiver operating characteristic ROC techniques that measure diagnostic accuracy to timing studies that measure image-interpretation workflow efficiency. The authors review briefly the history of the development of evaluation methodologies and review ROC methodology as well as other types of evaluation methods. They discuss unique challenges in system evaluation that face the imaging community today and opportunities for future advances. © 2008 American Association of Physicists in Medicine. DOI: 10.1118/1.2830376

Anniversary paper: evaluation of medical imaging systems.

A machine-learning framework for automatic reference-free quality assessment in MRI.

With medical imaging technologies growth, the question of their assessment on the impact and benefit on patient care is rising. Development and design of those medical imaging technologies should take into account the concept of image quality as it might impact the ability of practicians while they are using image information. Towards that goal, one should consider several human factors involved in image analysis and interpretation, e.g. image perception issues, decision process, image analysis pipeline (detection, localization, characterization…). While many efforts have been dedicated to objectively assess the value of imaging system in terms of ideal decision process, new trends have recently emerged to deal with human observer perfomances. This task effort is huge considering the variability of imaging acquisition methods and the possible pathologies. This paper proposes a survey of some key issues and results associated to this effort. We first outline the wide range of medical images with their own specific features. Next, we review the main methodologies to evaluate diagnostic quality of medical images from subjective assessment including ROC analysis, and diagnostic criteria quality analysis, to objective assessment including metrics based on the HVS, and model observers. At last, we present another evaluation method: eye-tracking studies to gain basic understanding of the visual search and decision-making process.

Diagnostic quality assessment of medical images: Challenges and trends

The channelized Hotelling observer (CHO) has become a widely used approach for evaluating medical image quality, acting as a surrogate for human observers in early-stage research on assessment and optimization of imaging devices and algorithms. The CHO is typically used to measure lesion detectability. Its popularity stems from experiments showing that the CHO's detection performance can correlate well with that of human observers. In some cases, CHO performance overestimates human performance; to counteract this effect, an internal-noise model is introduced, which allows the CHO to be tuned to match human-observer performance. Typically, this tuning is achieved using example data obtained from human observers. We argue that this internal-noise tuning step is essentially a model training exercise; therefore, just as in supervised learning, it is essential to test the CHO with an internal-noise model on a set of data that is distinct from that used to tune (train) the model. Furthermore, we argue that, if the CHO is to provide useful insights about new imaging algorithms or devices, the test data should reflect such potential differences from the training data; it is not sufficient simply to use new noise realizations of the same imaging method. Motivated by these considerations, the novelty of this paper is the use of new model selection criteria to evaluate ten established internal-noise models, utilizing four different channel models, in a train-test approach. Though not the focus of the paper, a new internal-noise model is also proposed that outperformed the ten established models in the cases tested. The results, using cardiac perfusion SPECT data, show that the proposed train-test approach is necessary, as judged by the newly proposed model selection criteria, to avoid spurious conclusions. The results also demonstrate that, in some models, the optimal internal-noise parameter is very sensitive to the choice of training data; therefore, these models are prone to overfitting, and will not likely generalize well to new data. In addition, we present an alternative interpretation of the CHO as a penalized linear regression wherein the penalization term is defined by the internal-noise model.

http://iopscience.iop.org/article/10.1088/0031-9155/58/20/7159/pdf

Evaluation of the channelized Hotelling observer with an internal-noise model in a train-test paradigm for cardiac SPECT defect detection

Pixel size is of great interest in the flat-panel detector design because of its potential impact on image quality. In the particular case of angiographic x-ray fluoroscopy, small pixels are required in order to adequately visualize interventional devices such as guidewires and stents which have wire diameters as small as 200 and 50 {mu}m, respectively. We used quantitative experimental and modeling techniques to investigate the optimal pixel size for imaging stents. Image quality was evaluated by the ability of subjects to perform two tasks: detect the presence of a stent and discriminate a partially deployed stent from a fully deployed one in synthetic images. With measurements at 50, 100, 200, and 300 {mu}m, the 100 {mu}m pixel size gave the maximum contrast sensitivity for the detection experiment with the idealized direct detector. For an idealized indirect detector with a scintillating layer, an optimal pixel size was obtained at 200 {mu}m pixel size. A channelized human observer model predicted a peak at 150 and 170 {mu}m, for the idealized direct and indirect detectors, respectively. With regard to the stent deployment task for both detector types, smaller pixel sizes are favored and there is a steep drop in performance with larger pixels.more » In general, with the increasing exposures, the model and measurements give the enhanced contrast sensitivities and a smaller optimal pixel size. The effects of electronic noise and fill factor were investigated using the model. We believe that the experimental results and human observer model predications can help guide the flat-panel detector design. In addition, the human observer model should work on the similar images and be applicable to the future model and actual flat-panel implementations.« less

Optimization of detector pixel size for stent visualization in x-ray fluoroscopy.

Methods for quantitative image quality evaluation of MRI parallel reconstructions: detection and perceptual difference model

Pixel size is of great interest in flat-panel detector design. To visualize small interventional devices such as a stent in angiographic x-ray fluoroscopy, pixels should be small to limit contrast dilution from partial-area and large to collect sufficient x-rays for an adequate signal-to-noise ratio (SNR). Using quantitative experimental and modeling techniques, we investigated the optimal pixel sizes for visualization of a stent created from 50 μm diameter wires. Image quality was evaluated by the ability of subjects to perform two tasks: detect the presence of a stent and discriminate a partially deployed stent from a fully deployed one. With regard to detection, for the idealized direct detector, the 100 μm pixel size resulted in maximum measured contrast sensitivity. For an idealized indirect detector, with a scintillating layer, the maximal measured contrast sensitivity was obtained at 200 μm pixel size. The channelized human observer model predicted a peak at 150 and 170 μm, for idealized direct and indirect detectors, respectively. Stent deployment is more sensitive to pixel size than stent detection, resulting in a steeper drop in performance with large pixels. With regard to stent deployment detection, smaller even pixel sizes are favored for both detector types. With increasing exposures, the model predicts a smaller optimal pixel size because the noise penalty is reduced.

Optimization of detector pixel size for stent visualization in x-ray fluoroscopy

We systematically evaluated a variety of MR spiral imaging acquisition and reconstruction schemes using a computational perceptual difference model (PDM) that models the ability of humans to perceive a visual difference between a degraded “fast” MRI image with subsampling of k-space and a “gold standard” image mimicking full acquisition. Human subject experiments performed using a modified double-stimulus continuous-quality scale (DSCQS) correlated well with PDM, over a variety of images. In a smaller set of conditions, PDM scores agreed very well with human detectability measurements of image quality. Having validated the technique, PDM was used to systematically evaluate 2016 spiral image conditions (six interleave patterns, seven sampling densities, three density compensation schemes, four reconstruction methods, and four noise levels). Voronoi (VOR) with conventional regridding gave the best reconstructions. At a fixed sampling density, more interleaves gave better results. With noise present more interleaves and samples were desirable. With PDM, conditions were determined where equivalent image quality was obtained with 50% sampling in noise-free conditions. We conclude that PDM scoring provides an objective, useful tool for the assessment of fast MR image quality that can greatly aid the design of MR acquisition and signal processing strategies.

/pdf/optimization-of-spiral-mri-using-a-perceptual-difference-4h17kaljqn.pdf

Optimization of Spiral MRI Using a Perceptual Difference Model

Although physical measurements such as detective quantum efficiency (DQE) and modulation transfer function (MTF) provide insights, quantitative optimization of x-ray flat panel detectors requires consideration of image quality as perceived by humans. Using experiments and human observer models, we quantified image quality as the ability to detect targets such as stents and guidewires used in interventional angiography. We realistically simulated direct and indirect flat panel detectors over a range of exposures to create realistic fluoroscopy sequences. We performed objective, m-alternative adaptive forced choice experiments and applied models of human detection to fit almost all experiments and predict results for similar tasks and processing. With regard to pixel size, the best size at low fluoroscopic exposures for detecting a 400 μm guide wire with a realistic, indirect detector was at about 200 μm and depended upon such device parameters as electronic noise. For both indirect and direct detectors at higher exposures, noise was not limiting, and a small pixel was desirable. With regard to binning, we determined that binning is desirable at low exposures even for detection of small objects such as a guide wire. A new alternate binning method was found to have superior image quality by utilizing the ability of humans to temporally fuse alternating images binned at different orientations. Comparing to magnification of analog image intensifiers, we determined that flat panel images can be digitally magnified without loss of image quality at a decreased average exposure.

Yuhao Jiang

Papers

Optimization of detector pixel size for stent visualization in x-ray fluoroscopy.

Methods for quantitative image quality evaluation of MRI parallel reconstructions: detection and perceptual difference model

Optimization of detector pixel size for stent visualization in x-ray fluoroscopy

Optimization of Spiral MRI Using a Perceptual Difference Model

Optimization of flat-panel detector characteristics and processing using quantitative image quality techniques