Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study
Ju Sun,Le Peng,Taihui Li,Dyah Adila,Z. Zaiman,Genevieve Melton-Meaux,Nicholas E. Ingraham,E Murray,Danielle A. Boley,Sean P. Switzer,John L. Burns,Kun Huang,Tadashi Allen,Scott D. Steenburg,Judy Wawira Gichoya,Erich Kummerfeld,Christopher J. Tignanelli +16 more
Reads0
Chats0
TLDR
AI-based tools have not yet reached full diagnostic potential for COVID-19 and underperform compared with radiologist prediction and the association of race and sex with AI model diagnostic accuracy was evaluated.Abstract:
Purpose To conduct a prospective observational study across 12 U.S. hospitals to evaluate real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 on chest radiographs. Materials and Methods A total of 95 363 chest radiographs were included in model training, external validation, and real-time validation. The model was deployed as a clinical decision support system, and performance was prospectively evaluated. There were 5335 total real-time predictions and a COVID-19 prevalence of 4.8% (258 of 5335). Model performance was assessed with use of receiver operating characteristic analysis, precision-recall curves, and F1 score. Logistic regression was used to evaluate the association of race and sex with AI model diagnostic accuracy. To compare model accuracy with the performance of board-certified radiologists, a third dataset of 1638 images was read independently by two radiologists. Results Participants positive for COVID-19 had higher COVID-19 diagnostic scores than participants negative for COVID-19 (median, 0.1 [IQR, 0.0–0.8] vs 0.0 [IQR, 0.0–0.1], respectively; P < .001). Real-time model performance was unchanged over 19 weeks of implementation (area under the receiver operating characteristic curve, 0.70; 95% CI: 0.66, 0.73). Model sensitivity was higher in men than women (P = .01), whereas model specificity was higher in women (P = .001). Sensitivity was higher for Asian (P = .002) and Black (P = .046) participants compared with White participants. The COVID-19 AI diagnostic system had worse accuracy (63.5% correct) compared with radiologist predictions (radiologist 1 = 67.8% correct, radiologist 2 = 68.6% correct; McNemar P < .001 for both). Conclusion AI-based tools have not yet reached full diagnostic potential for COVID-19 and underperform compared with radiologist prediction. Keywords: Diagnosis, Classification, Application Domain, Infection, Lung Supplemental material is available for this article.. © RSNA, 2022read more
Citations
More filters
Posted ContentDOI
Can Artificial Intelligence Detect Monkeypox from Digital Skin Images?
TL;DR: The study found that deep AI models have great potential in the detection of Monkeypox from digital skin images (precision of 85%).
Posted ContentDOI
A Web-scrapped Skin Image Database of Monkeypox, Chickenpox, Smallpox, Cowpox, and Measles
TL;DR: This project used web-scrapping to collect Monkeypox, Chickenpox, Smallpox, Cowpox and Measles infected skin as well as healthy skin images to build a comprehensive image database and made it publicly available.
Journal ArticleDOI
Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals
Le Peng,Gaoxiang Luo,Andrew Walker,Z. Zaiman,Emma Jones,Hemant Gupta,Kristopher Kersten,John L. Burns,Christopher A. Harle,Tanja Magoc,Benjamin Shickel,Scott D. Steenburg,Tyler J. Loftus,Genevieve B. Melton,Judy Wawira Gichoya,Ju Sun,Christopher J. Tignanelli +16 more
TL;DR: FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity.
Journal ArticleDOI
Kidney Diseases Classification using Hybrid Transfer-Learning DenseNet201-Based and Random Forest Classifier
TL;DR: In this paper , a hybrid technique is used by utilizing both pre-train models for feature extraction and classification using machine learning algorithms for the task of kidney disease image diagnosis, using a dataset of 12,446 CT urogram and whole abdomen images.
Journal ArticleDOI
Artificial Intelligence–enabled Decision Support in Surgery
Tyler J. Loftus,Maria S. Altieri,Jeremy Balch,Kenneth L. Abbott,Je Ho Choi,Jayson S. Marwaha,Daniel A. Hashimoto,Gabriel A. Brat,Ioannis Raftopoulos,Heather L. Evans,Gretchen Purcell Jackson,Danielle S Walsh,Christopher J. Tignanelli +12 more
TL;DR: In this article , the authors summarize state-of-the-art artificial intelligence-enabled decision support in surgery and quantify deficiencies in scientific rigor and reporting, and conclude that researchers should strive to improve scientific quality.
References
More filters
Journal ArticleDOI
Note on the sampling error of the difference between correlated proportions or percentages.
TL;DR: Two formulas are presented for judging the significance of the difference between correlated proportions and the chi square equivalent of one of the developed formulas.
Journal ArticleDOI
Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal
Laure Wynants,Laure Wynants,Ben Van Calster,Ben Van Calster,Gary S. Collins,Gary S. Collins,Richard D Riley,Georg Heinze,Ewoud Schuit,Marc J.M. Bonten,Darren Dahly,Johanna A A G Damen,Thomas P. A. Debray,Valentijn M.T. de Jong,Maarten De Vos,Paula Dhiman,Paula Dhiman,Maria C Haller,Michael O. Harhay,Liesbet Henckaerts,Pauline Heus,Michael Kammer,Nina Kreuzberger,Anna Lohmann,Kim Luijken,Jie Ma,Glen P. Martin,David J. McLernon,Constanza L Andaur Navarro,Johannes B. Reitsma,Jamie C. Sergeant,Chunhu Shi,Nicole Skoetz,Luc J.M. Smits,Kym I E Snell,Matthew Sperrin,René Spijker,René Spijker,Ewout W. Steyerberg,Toshihiko Takada,Ioanna Tzoulaki,Ioanna Tzoulaki,Sander M. J. van Kuijk,Bas C T van Bussel,Bas C T van Bussel,Iwan C. C. van der Horst,Florien S. van Royen,Jan Y Verbakel,Jan Y Verbakel,Christine Wallisch,Christine Wallisch,Jack Wilkinson,Robert Wolff,Lotty Hooft,Karel G.M. Moons,Maarten van Smeden +55 more
TL;DR: Proposed models for covid-19 are poorly reported, at high risk of bias, and their reported performance is probably optimistic, according to a review of published and preprint reports.
Journal ArticleDOI
False Negative Tests for SARS-CoV-2 Infection - Challenges and Implications.
TL;DR: Diagnostic testing for SARS-CoV-2 will help in safely reopening the country, but only if tests are highly accurate, experts say.
Journal ArticleDOI
Racial and Ethnic Disparities in COVID-19-Related Infections, Hospitalizations, and Deaths : A Systematic Review.
Katherine Mackey,Chelsea Ayers,Karli Kondo,Somnath Saha,Shailesh Advani,Sarah Young,Hunter C. Spencer,Max Rusek,Johanna Anderson,Stephanie Veazie,Mia Smith,Devan Kansagara +11 more
TL;DR: A systematic review evaluating racial/ethnic disparities in SARS-CoV-2 infection rates and COVID-19 outcomes, factors contributing to disparities, and interventions to reduce them suggests that impacts of CO VID-19 differ among U.S. racial/ ethnic groups.
Related Papers (5)
Diagnostic Accuracy of the Aldosterone–to–Active Renin Ratio for Detecting Primary Aldosteronism
Stefan Pilz,Martin H. Keppel,Christian Trummer,Verena Theiler-Schwetz,Marlene Pandis,Valentin Borzan,Matthias Pittrof,Barbara Obermayer-Pietsch,Martin R. Grübler,Nicolas Verheyen,Vinzenz Stepan,Andreas Meinitzer,Jakob Voelkl,Jakob Voelkl,Winfried März,Winfried März,Winfried März,Andreas Tomaschitz +17 more