In the past decade, advances in precision oncology have resulted in an increased demand for predictive assays that enable the selection and stratification of patients for treatment. The enormous divergence of signalling and transcriptional networks mediating the crosstalk between cancer, stromal and immune cells complicates the development of functionally relevant biomarkers based on a single gene or protein. However, the result of these complex processes can be uniquely captured in the morphometric features of stained tissue specimens. The possibility of digitizing whole-slide images of tissue has led to the advent of artificial intelligence (AI) and machine learning tools in digital pathology, which enable mining of subvisual morphometric phenotypes and might, ultimately, improve patient management. In this Perspective, we critically evaluate various AI-based computational approaches for digital pathology, focusing on deep neural networks and 'hand-crafted' feature-based methodologies. We aim to provide a broad framework for incorporating AI and machine learning tools into clinical oncology, with an emphasis on biomarker development. We discuss some of the challenges relating to the use of AI, including the need for well-curated validation datasets, regulatory approval and fair reimbursement strategies. Finally, we present potential future opportunities for precision oncology.

/pdf/artificial-intelligence-in-digital-pathology-new-tools-for-3tat3v4mlq.pdf

Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology

Medical image analysis

Multi-grade brain tumor classification using deep CNN with extensive data augmentation

Automatic recognition of different tissue types in histological images is an essential part in the digital pathology toolbox. Texture analysis is commonly used to address this problem; mainly in the context of estimating the tumour/stroma ratio on histological samples. However, although histological images typically contain more than two tissue types, only few studies have addressed the multi-class problem. For colorectal cancer, one of the most prevalent tumour types, there are in fact no published results on multiclass texture separation. In this paper we present a new dataset of 5,000 histological images of human colorectal cancer including eight different types of tissue. We used this set to assess the classification performance of a wide range of texture descriptors and classifiers. As a result, we found an optimal classification strategy that markedly outperformed traditional methods, improving the state of the art for tumour-stroma separation from 96.9% to 98.6% accuracy and setting a new standard for multiclass tissue separation (87.4% accuracy for eight classes). We make our dataset of histological images publicly available under a Creative Commons license and encourage other researchers to use it as a benchmark for their studies.

Multi-class texture analysis in colorectal cancer histology

Conventional methods for intraoperative histopathologic diagnosis are labour- and time-intensive, and may delay decision-making during brain-tumour surgery. Stimulated Raman scattering (SRS) microscopy, a label-free optical process, has been shown to rapidly detect brain-tumour infiltration in fresh, unprocessed human tissues. Here, we demonstrate the first application of SRS microscopy in the operating room using a portable fibre-laser-based microscope and unprocessed specimens from 101 neurosurgical patients. We also introduce an image-processing method—stimulated Raman histology (SRH)—that leverages SRS images to create virtual haematoxylin-and-eosin-stained slides, revealing essential diagnostic features. In a simulation of intraoperative pathologic consultation in 30 patients, we found a remarkable concordance of SRH and conventional histology for predicting diagnosis (Cohen’s kappa, κ > 0.89), with accuracy exceeding 92%. We also built and validated a multilayer perceptron based on quantified SRH image attributes that predicts brain-tumour subtype with 90% accuracy. Our findings provide insight into how SRH can now be used to improve the surgical care of brain-tumour patients. By taking advantage of stimulated Raman spectroscopy and fibre-laser technology, virtual histology images can be obtained in real time in the operating room, with diagnostic quality comparable with that achieved via conventional histopathology.

/pdf/rapid-intraoperative-histology-of-unprocessed-surgical-289fe2dq53.pdf

Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy.

Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles.

Purpose:
With the advent of digital slide scanning technologies and the potential proliferation of large repositories of digital pathology images, many research studies can leverage these data for biomedical discovery and to develop clinical applications. However, quantitative analysis of digital pathology images is impeded by batch effects generated by varied staining protocols and staining conditions of pathological slides.

Methods:
To overcome this problem, this paper proposes a novel, fully automated stain normalization method to reduce batch effects and thus aid research in digital pathology applications. Their method, intensity centering and histogram equalization (ICHE), normalizes a diverse set of pathology images by first scaling the centroids of the intensity histograms to a common point and then applying a modified version of contrast-limited adaptive histogram equalization. Normalization was performed on two datasets of digitized hematoxylin and eosin (H&E) slides of different tissue slices from the same lung tumor, and one immunohistochemistry dataset of digitized slides created by restaining one of the H&E datasets.

Results:
The ICHE method was evaluated based on image intensity values, quantitative features, and the effect on downstream applications, such as a computer aided diagnosis. For comparison, three methods from the literature were reimplemented and evaluated using the same criteria. The authors found that ICHE not only improved performance compared with un-normalized images, but in most cases showed improvement compared with previous methods for correcting batch effects in the literature.

Conclusions:
ICHE may be a useful preprocessing step a digital pathology image processing pipeline.

A method for normalizing pathology images to improve feature extraction for quantitative pathology.

Provided are automated (computerized) methods and systems for analyzing digitized pathology images in a variety of tissues potentially containing diseased or neoplastic cells. The method utilizes a coarse-to-fine analysis, in which an entire image is tiled and shape, color, and texture features are extracted in each tile, as primary features. A representative subset of tiles is determined within a cluster of similar tiles. A statistical analysis (e.g. principal component analysis) reduces the substantial number of "coarse" features, decreasing computational complexity of the classification algorithm. Afterwards, a fine stage provides a detailed analysis of a single representative tile from each group. A second statistical step uses a regression algorithm (e.g. elastic net classifier) to produce a diagnostic decision value for each representative tile. A weighted voting scheme aggregates the decision values from these tiles to obtain a diagnosis at the whole slide level.

Profiling of pathology images for clinical applications

ABSTRACT Temporally locating and classifying instruments in surgical video is useful for the analysis and comparison of surgical techniques. This paper aims to apply action segmentation techniques to temporally segment and classify surgical instruments, and to highlight the utility of this modelling approach through example applications. This paper shows that the action segmentation transformer (ASFormer) architecture with an EfficientNetV2 featurizer performs significantly better in mean average precision than any previous approaches to this task on the Cholec80 dataset. The ASFormer also outperforms Long Short-Term Memory (LSTM) and Multi-Stage Temporal Convolutional Network (MS-TCN) architectures with the same featurizer. This model reduces the need for costly human labelling of surgical video, driving the development of indexed surgical video libraries and instrument usage tracking applications. Examples of these applications are included after the results.

Surgical instrument recognition for instrument usage documentation and surgical video library indexing

Mastering the technical skills required to perform surgery is an extremely challenging task. Video-based assessment allows surgeons to receive feedback on their technical skills to facilitate learning and development. Currently, this feedback comes primarily from manual video review, which is time-intensive and limits the feasibility of tracking a surgeon's progress over many cases. In this work, we introduce a motion-based approach to automatically assess surgical skills from surgical case video feed. The proposed pipeline first tracks surgical tools reliably to create motion trajectories and then uses those trajectories to predict surgeon technical skill levels. The tracking algorithm employs a simple yet effective re-identification module that improves ID-switch compared to other state-of-the-art methods. This is critical for creating reliable tool trajectories when instruments regularly move on- and off-screen or are periodically obscured. The motion-based classification model employs a state-of-the-art self-attention transformer network to capture short- and long-term motion patterns that are essential for skill evaluation. The proposed method is evaluated on an in-vivo (Cholec80) dataset where an expert-rated GOALS skill assessment of the Calot Triangle Dissection is used as a quantitative skill measure. We compare transformer-based skill assessment with traditional machine learning approaches using the proposed and state-of-the-art tracking. Our result suggests that using motion trajectories from reliable tracking methods is beneficial for assessing surgeon skills based solely on video streams.

Jocelyn Barker

Papers

Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles.

A method for normalizing pathology images to improve feature extraction for quantitative pathology.

Profiling of pathology images for clinical applications

Surgical instrument recognition for instrument usage documentation and surgical video library indexing

Video-based Surgical Skills Assessment using Long term Tool Tracking