Diagnostic tests are of crucial importance in health care. They are performed to reduce uncertainty concerning whether a patient has a condition of interest. A thorough evaluation of diagnostic tests is necessary to ensure that only accurate tests are used in practice. Diagnostic accuracy studies are a vital step in this evaluation process. Diagnostic accuracy studies aim to investigate how well the results from a test being evaluated (index test) agree with the results of the reference standard. The reference standard is considered the best available method to establish the presence or absence of a condition (target condition). In a classic diagnostic accuracy study, a consecutive series of patients who are suspected of having the target condition undergo the index test; then, all patients are verified by the same reference standard. The index test and reference standard are then read by persons blinded to the results of each, and various measures of agreement are calculated (for example, sensitivity, specificity, likelihood ratios, and diagnostic odds ratios).

Sources of Variation and Bias in Studies of Diagnostic Accuracy: a Systematic Review

Inflexible use of evidence hierarchies confuses practitioners and irritates researchers. So how can we improve the way we assess research?

/pdf/assessing-the-quality-of-research-4fuei9oubv.pdf

Assessing the quality of research.

http://www.mossmanassociates.com/TB%20Biomarker%20report%20Lancet_2010.pdf

Tuberculosis 4 Biomarkers and diagnostics for tuberculosis: progress, needs, and translation into practice

A systematic review classifies sources of bias and variation in diagnostic test accuracy studies.

Background: There have been few inter-observer studies of diffuse parenchymal lung disease (DPLD), but the recent ATS/ERS consensus classification provides a basis for such a study.

Methods: A method for categorising numerically the percentage likelihood of these differential diagnoses was developed, and the diagnostic confidence of pathologists using this classification and the reproducibility of their diagnoses were assessed.

Results: The overall kappa coefficient of agreement for the first choice diagnosis was 0.38 (n = 133 biopsies), increasing to 0.43 for patients (n = 83) with multiple biopsies. Weighted kappa coefficients of agreement, quantifying the level of probability of individual diagnoses, were moderate to good (mean 0.58, range 0.40–0.75). However, in 18% of biopsy specimens the diagnosis was given with low confidence. Over 50% of inter-observer variation related to the diagnosis of non-specific interstitial pneumonia and, in particular, its distinction from usual interstitial pneumonia.

Conclusion: These results show that the ATS/ERS classification can be applied reproducibly by pathologists who evaluate DPLD routinely, and support the practice of taking multiple biopsy specimens.

https://thorax.bmj.com/content/thoraxjnl/59/6/500.full.pdf

Inter-observer variation between pathologists in diffuse parenchymal lung disease

Although cervical-vaginal telecytology is a promising tool, diagnostic accuracy has not been extensively evaluated. The authors examined the accuracy of five cytotechnologists who retrospectively reviewed 50 cervical-vaginal smears using the video monitor, and 2 months later, using the light microscope. Accuracy was expressed in terms of crude agreement with the original diagnosis and number of false positives (FPs) and false negatives (FNs). With a greater than one step difference as discrepant, the group crude agreement using the video monitor and the light microscope was 85.6% and 95.6%, respectively. The group number of FNs and FPs for the light microscope was 8 and 7, respectively, and for the video monitor was 34 and 7, respectively. There was a wide range of individual performance. We conclude that accuracy of telecytology is high, but less than that of light microscopy. The major reason for lower telecytologic accuracy was undercalling dysplasia.

/pdf/telecytology-diagnostic-accuracy-in-cervical-vaginal-smears-2a7dqsimas.pdf

Telecytology: diagnostic accuracy in cervical-vaginal smears.

Objective To evaluate the effectiveness of 3-D vs. 2-D virtual microscopy as adjuncts to education and assessment in cervical cytology. Study Design Five cervical cytology slides were acquired in 2-D; then the identical area of the slide was acquired in 3-D, resulting in 2 sets of virtual slides for comparison with the original glass slide. Seventy-nine paid volunteer cytologists and cytotechnology students participated. Approximately half were sent the 2-D set of slides via the Web, and the others a 3-D set of slides on a DVD. Evaluators examined the virtual slides and committed to an interpretation. After receipt of the original glass slides, a second interpretation was made, if different from the virtual slide interpretation. Results Diagnostic accuracy using virtual cytology slides was similar to that for glass slides (94% vs. 96%). There was no difference in diagnostic accuracy between 2-D and 3-D slides (p = 0.28); however, the ability to focus 3-D slides in the z-axis was strongly endorsed by the participants because of the uncertainty and frustration of having some cells out of focus on 2-D virtual slides. Conclusion There was consensus that virtual cervical cytology slides would be a useful augmentation to education and testing.

Utility of 2-D and 3-D virtual microscopy in cervical cytology education and testing.

Context.—The Clinical Laboratory Improvement Amendments of 1988 require that laboratories perform cytologic-histologic correlation, although the optimal methods and the value of performing...

Variability in cytologic-histologic correlation practices and implications for patient safety.

Diagnoses in pathology often are qualitative, such as atypical or suspicious, and consequently are thought to have limited clinical value. To investigate the utility of a qualitative diagnostic system, seven pathologists retrospectively evaluated 100 bronchial brush specimens using the following categories : definitely benign, probably Benign, possibly malignant, probably malignant, and definitely malignant. The likelihood ratio (LR) and receiver operating characteristic (ROC) curve, two statistical probabilistic measurements, were used to calculate diagnostic accuracy among individuals and groups. The results show : (1) the LR for individual diagnostic categories varied among observers, resulting in different clinically malignant probabilities ; (2) observer experience did not appear to play a role in overall diagnostic accuracy, except in the diagnosis of small cell carcinoma ; (3) observers operate at higher levels of diagnostic accuracy with, rather than without, clinical history. The authors conclude that qualitative diagnoses contain important information and can be interpreted effectively with LR and ROC. (Key words : Statistics ; Anatomic pathology ; Cytology ; Diagnosis ; Quality assurance/control) Am J Clin Pathol 1995 ;103 :588-593.

Pathology and probability. Likelihood ratios and receiver operating characteristic curves in the interpretation of bronchial brush specimens.

Turnaround time for Papanicolaou (Pap) tests became an important service quality issue at our institution. We studied Pap test turnaround time using engineering process improvement tools and benchmarked turnaround time against data published as a College of American Pathologists Q-Probes study. An IDEF3 process map revealed the complexity of the Pap test process and the opportunities for process improvement. We used these data and the action-research method to initiate changes in cytopathology laboratory operations with the goal of reducing turnaround time. Before intervention, mean Pap test turnaround time was highly variable; during a 6-month period, monthly means ranged from 2.5 to 10.8 days. A cycle time study conducted over a 2-week period validated these data. After system improvements were implemented, the monthly mean turnaround time decreased and became more consistent, with 11 of 12 months having a mean turnaround time of 3 days or less (range, 1.5-3.9 days). Our study illustrates the value of publishing Q-Probes data for use as external benchmarks and the benefits of using tools from other disciplines to improve laboratory processes.

/pdf/improving-pap-test-turnaround-time-using-external-benchmark-1rbt73mg6z.pdf

M. Sue Zaleski

Papers

Telecytology: diagnostic accuracy in cervical-vaginal smears.

Utility of 2-D and 3-D virtual microscopy in cervical cytology education and testing.

Variability in cytologic-histologic correlation practices and implications for patient safety.

Pathology and probability. Likelihood ratios and receiver operating characteristic curves in the interpretation of bronchial brush specimens.

Improving Pap test turnaround time using external benchmark data and engineering process improvement tools.