In this paper we introduce OperA, a transformer-based model that accurately predicts surgical phases from long video sequences. A novel attention regularization loss encourages the model to focus on high-quality frames during training. Moreover, the attention weights are utilized to identify characteristic high attention frames for each surgical phase, which could further be used for surgery summarization. OperA is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos, outperforming various state-of-the-art temporal refinement approaches.

OperA: Attention-Regularized Transformers for Surgical Phase Recognition

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.PROSPERO: CRD42020226071. 

/pdf/machine-learning-for-technical-skill-assessment-in-surgery-a-170qrey0.pdf

Machine learning for technical skill assessment in surgery: a systematic review

Telesurgical robotics, as a technical solution for robot-assisted minimally invasive surgery (RAMIS), has become the first domain within medicosurgical robotics that achieved a true global clinical adoption. Its relative success (still at a low single-digit percentile total market penetration) roots in the particular human-in-the-loop control, in which the trained surgeon is always kept responsible for the clinical outcome achieved by the robot-actuated invasive tools. Nowadays, this paradigm is challenged by the need for improved surgical performance, traceability, and safety reaching beyond the human capabilities. Partially due to the technical complexity and the financial burden, the adoption of telesurgical robotics has not reached its full potential, by far. Apart from the absolutely market-dominating da Vinci surgical system, there are already 60+ emerging RAMIS robot types, out of which 15 have already achieved some form of regulatory clearance. This article aims to connect the technological advancement with the principles of commercialization, particularly looking at engineering components that are under development and have the potential to bring significant advantages to the clinical practice. Current RAMIS robots often do not exceed the functionalities deriving from their mechatronics, due to the lack of data-driven assistance and smart human–machine collaboration. Computer assistance is gradually gaining more significance within emerging RAMIS systems. Enhanced manipulation capabilities, refined sensors, advanced vision, task-level automation, smart safety features, and data integration mark together the inception of a new era in telesurgical robotics, infiltrated by machine learning (ML) and artificial intelligence (AI) solutions. Observing other domains, it is definite that a key requirement of a robust AI is the good quality data, derived from proper data acquisition and sharing to allow building solutions in real time based on ML. Emerging RAMIS technologies are reviewed both in a historical and a future perspective.

Robot-Assisted Minimally Invasive Surgery—Surgical Robotics in the Data Age

The growing interest in analysis of surgical video through machine learning has led to increased research efforts; however, common methods of annotating video data are lacking. There is a need to establish recommendations on the annotation of surgical video data to enable assessment of algorithms and multi-institutional collaboration. Four working groups were formed from a pool of participants that included clinicians, engineers, and data scientists. The working groups were focused on four themes: (1) temporal models, (2) actions and tasks, (3) tissue characteristics and general anatomy, and (4) software and data structure. A modified Delphi process was utilized to create a consensus survey based on suggested recommendations from each of the working groups. After three Delphi rounds, consensus was reached on recommendations for annotation within each of these domains. A hierarchy for annotation of temporal events in surgery was established. While additional work remains to achieve accepted standards for video annotation in surgery, the consensus recommendations on a general framework for annotation presented here lay the foundation for standardization. This type of framework is critical to enabling diverse datasets, performance benchmarks, and collaboration.

SAGES consensus recommendations on an annotation framework for surgical video

Aim: Artificial intelligence (AI) is rapidly evolving in healthcare worldwide, especially in surgery. This article reviews important terms used in machine learning and the challenges of deep learning in surgery. Methods: A review of the English literature was carried out focused on the terms “challenges of deep learning” and “surgery” using Medline and PubMed between 2018 and 2022. Results: In total, 54 articles discussed the challenges of deep learning in general. We include 25 articles from various surgical specialties discussing challenges corresponding to their respective specialties. Conclusion: The increased utilization of AI in surgery is faced with a wide variety of technical, ethical, clinical, and business-related challenges. The best way to expedite its expansion in surgery in the safest and most cost-efficient manner is by ensuring that as many surgeons as possible have a clear understanding of basic AI concepts and how they can be applied to the preoperative, intraoperative, postoperative, and long-term follow-up phases of the surgical patient care.

The challenges of deep learning in artificial intelligence and autonomous actions in surgery: a literature review

Objective To provide an overview of ML models and data streams utilized for automated surgical phase recognition. Background Phase recognition identifies different steps and phases of an operation. ML is an evolving technology that allows analysis and interpretation of huge data sets. Automation of phase recognition based on data inputs is essential for optimization of workflow, surgical training, intraoperative assistance, patient safety, and efficiency. Methods A systematic review was performed according to the Cochrane recommendations and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. PubMed, Web of Science, IEEExplore, GoogleScholar, and CiteSeerX were searched. Literature describing phase recognition based on ML models and the capture of intraoperative signals during general surgery procedures was included. Results A total of 2254 titles/abstracts were screened, and 35 full-texts were included. Most commonly used ML models were Hidden Markov Models and Artificial Neural Networks with a trend towards higher complexity over time. Most frequently used data types were feature learning from surgical videos and manual annotation of instrument use. Laparoscopic cholecystectomy was used most commonly, often achieving accuracy rates over 90%, though there was no consistent standardization of defined phases. Conclusions ML for surgical phase recognition can be performed with high accuracy, depending on the model, data type, and complexity of surgery. Different intraoperative data inputs such as video and instrument type can successfully be used. Most ML models still require significant amounts of manual expert annotations for training. The ML models may drive surgical workflow towards standardization, efficiency, and objectiveness to improve patient outcome in the future. Registration prospero CRD42018108907.

Machine Learning for Surgical Phase Recognition: A Systematic Review.

Previous studies have developed some blood‐based biomarker algorithms such as the Doylestown algorithm and aMAP score to improve the detection of Hepatocellular carcinoma (HCC). However, no one has studied the application of the Doylestown algorithm in the Chinese. Meanwhile, which of these two screening models is more suitable for people with liver cirrhosis remains to be investigated.

Validation of the hepatocellular carcinoma early detection screening algorithm Doylestown and aMAP in a cohort of Chinese with cirrhosis

Machine learning for surgical phase recognition

Identifying BRCA mutations and homologous recombination deficiency (HRD) is the key to choosing patients for poly (ADP-ribose) polymerase inhibitor (PARPi) therapy. At present, a large amount of research focuses on the application of HRD detection in ovarian cancer. However, few studies have discussed the relationship between HRD detection and postoperative survival in patients with epithelial ovarian cancer (EOC). This study included 38 consecutive patients with EOC who underwent cytoreduction surgery. Owing to tissue availability, only 29 patients underwent molecular profiling and survival analysis. Overall, 21 (72.4%) tumors had HRD scores of ≥42. Mutations in BRCA were observed in 5/29 (17.2%) patients. In this cohort, an HRD score of ≥42 was more common in serous ovarian tumors. We found no statistically significant association between homologous recombination repair (HRR) genes and HRD scores except for tumor protein P53 (TP53) mutation. We also found a strong positive association between HRD scores and chromosomal instability (CIN). In the survival analysis, an HRD score of >23 was correlated with better postoperative progression-free survival (pPFS). With increased depth of research, an appropriate HRD score threshold may serve as a prognostic tool and should be assessed in future studies to predict the clinical value of PARPi.

/pdf/biomarker-assessment-of-homologous-recombination-deficiency-l8m0wrkv.pdf

Linhong Li

Papers

Machine Learning for Surgical Phase Recognition: A Systematic Review.

Validation of the hepatocellular carcinoma early detection screening algorithm Doylestown and aMAP in a cohort of Chinese with cirrhosis

Machine learning for surgical phase recognition

Biomarker Assessment of Homologous Recombination Deficiency in Epithelial Ovarian Cancer: Association With Progression-Free Survival After Surgery