http://www.cell.com/article/S2666379122004244/pdf

Evidence synthesis, digital scribes, and translational challenges for artificial intelligence in healthcare

A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews

We aimed to evaluate the performance of supervised machine learning algorithms in predicting articles relevant for full‐text review in a systematic review. Overall, 16,430 manually screened titles/abstracts, including 861 references identified relevant for full‐text review were used for the analysis. Of these, 40% (n = 6573) were sub‐divided for training (70%) and testing (30%) the algorithms. The remaining 60% (n = 9857) were used as a validation set. We evaluated down‐ and up‐sampling methods and compared unigram, bigram, and singular value decomposition (SVD) approaches. For each approach, Naïve Bayes, Support Vector Machines (SVM), regularized logistic regressions, neural networks, random forest, Logit boost, and XGBoost were implemented using simple term frequency or Tf‐Idf feature representations. Performance was evaluated using sensitivity, specificity, precision and area under the Curve. We combined predictions of the best‐performing algorithms (Youden Index ≥0.3 with sensitivity/specificity≥70/60%). In a down‐sample unigram approach, Naïve Bayes, SVM/quanteda text models with Tf‐Idf, and linear SVM e1071 package with Tf‐Idf achieved >90% sensitivity at specificity >65%. Combining the predictions of the 10 best‐performing algorithms improved the performance to reach 95% sensitivity and 64% specificity in the validation set. Crude screening burden was reduced by 61% (5979) (adjusted: 80.3%) with 5% (27) false negativity rate. All the other approaches yielded relatively poorer performances. The down‐sampling unigram approach achieved good performance in our data. Combining the predictions of algorithms improved sensitivity while screening burden was reduced by almost two‐third. Implementing machine learning approaches in title/abstract screening should be investigated further toward refining these tools and automating their implementation.

https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/jrsm.1589

In‐depth evaluation of machine learning methods for semi‐automating article screening in a systematic review of mechanistic literature

Machine learning (ML) tools exist that can reduce or replace human activities in repetitive or complex tasks. Yet, ML is underutilized within evidence synthesis, despite the steadily growing rate of primary study publication and the need to periodically update reviews to reflect new evidence. Underutilization may be partially explained by a paucity of evidence on how ML tools can reduce resource use and time-to-completion of reviews.This protocol describes how we will answer two research questions using a retrospective study design: Is there a difference in resources used to produce reviews using recommended ML versus not using ML, and is there a difference in time-to-completion? We will also compare recommended ML use to non-recommended ML use that merely adds ML use to existing procedures. We will retrospectively include all reviews conducted at our institute from 1 August 2020, corresponding to the commission of the first review in our institute that used ML.The results of this study will allow us to quantitatively estimate the effect of ML adoption on resource use and time-to-completion, providing our organization and others with better information to make high-level organizational decisions about ML. 

/pdf/the-effect-of-machine-learning-tools-for-evidence-synthesis-1grah6x4.pdf

The effect of machine learning tools for evidence synthesis on resource use and time-to-completion: protocol for a retrospective pilot study

Hip arthroplasty dislocation risk calculator: evaluation of one million primary implants and twenty-five thousand dislocations with deep learning artificial intelligence in a systematic review of reviews

The exponential increase in published articles makes a thorough and expedient review of literature increasingly challenging. This review delineated automated tools and platforms that employ artificial intelligence (AI) approaches and evaluated the reported benefits and challenges in using such methods. A search was conducted in 4 databases (Medline, Embase, CDSR, and Epistemonikos) up to April 2021 for systematic reviews and other related reviews implementing AI methods. To be included, the review must use any form of AI method, including machine learning, deep learning, neural network, or any other applications used to enable the full or semi‐autonomous performance of one or more stages in the development of evidence synthesis. Twelve reviews were included, using nine different tools to implement 15 different AI methods. Eleven methods were used in the screening stages of the review (73%). The rest were divided: two in data extraction (13%) and two in risk of bias assessment (13%). The ambiguous benefits of the data extractions, combined with the reported advantages from 10 reviews, indicating that AI platforms have taken hold with varying success in evidence synthesis. However, the results are qualified by the reliance on the self‐reporting of the review authors. Extensive human validation still appears required at this stage in implementing AI methods, though further evaluation is required to define the overall contribution of such platforms in enhancing efficiency and quality in evidence synthesis.

/pdf/using-artificial-intelligence-methods-for-systematic-review-2fepfb5f.pdf

Aymeric Blaizot

Papers

Using artificial intelligence methods for systematic review in health sciences: A systematic review