Showing papers by "Mert R. Sabuncu published in 2023"

UniverSeg: Universal Medical Image Segmentation

Abstract: While deep learning models have become the predominant method for medical image segmentation, they are typically not capable of generalizing to unseen segmentation tasks involving new anatomies, image modalities, or labels. Given a new segmentation task, researchers generally have to train or fine-tune models, which is time-consuming and poses a substantial barrier for clinical researchers, who often lack the resources and expertise to train neural networks. We present UniverSeg, a method for solving unseen medical segmentation tasks without additional training. Given a query image and example set of image-label pairs that define a new segmentation task, UniverSeg employs a new Cross-Block mechanism to produce accurate segmentation maps without the need for additional training. To achieve generalization to new tasks, we have gathered and standardized a collection of 53 open-access medical segmentation datasets with over 22,000 scans, which we refer to as MegaMedical. We used this collection to train UniverSeg on a diverse set of anatomies and imaging modalities. We demonstrate that UniverSeg substantially outperforms several related methods on unseen tasks, and thoroughly analyze and draw insights about important aspects of the proposed system. The UniverSeg source code and model weights are freely available at https://universeg.csail.mit.edu

Modulating human brain responses via optimal natural image selection and synthetic image generation

Abstract: One of the main goals of neuroscience is to understand how biological brains interpret and process incoming environmental information. Building computational encoding models that map images to neural responses is one way to pursue this goal. Moreover, generating or selecting visual stimuli designed to achieve specific patterns of responses allows exploration and control of neuronal firing rates or regional brain activity responses. Here, we investigated the brain’s regional activation selectivity and inter-individual differences in human brain responses to various sets of natural and synthetic (generated) images via two functional MRI (fMRI) studies. For our first fMRI study, we used a pre-trained group-level neural model for selecting or synthesizing images that are predicted to maximally activate targeted brain regions. We then presented these images to subjects while collecting their fMRI data. Our results show that optimized images indeed evoke larger magnitude responses than other images predicted to achieve average levels of activation.Furthermore, the activation gain is positively associated with the encoding model accuracy. While most regions’ activations in response to maximal natural images and maximal synthetic images were not different, two regions, namely anterior temporal lobe faces (aTLfaces) and fusiform body area 1 (FBA1), had significantly higher activation in response to maximal synthetic images compared to maximal natural images. On the other hand, three regions; medial temporal lobe face area (mTLfaces), ventral word form area 1 (VWFA1) and ventral word form area 2 (VWFA2), had higher activation in response to maximal natural images compared to maximal synthetic images. In our second fMRI experiment, we focused on probing inter-individual differences in face regions’ responses and found that individual-specific synthetic (and not natural) images derived using a personalized encoding model elicited significantly higher responses compared to synthetic images derived from the group-level or other subjects’ encoding models. Finally, we replicated the finding showing synthetic images elicited larger activation responses in the aTLfaces region compared to natural image responses in that region. Here, for the first time, we leverage our data-driven and generative modeling framework NeuroGen to probe inter-individual differences in and functional specialization of the human visual system. Our results indicate that NeuroGen can be used to modulate macro-scale brain regions in specific individuals using synthetically generated visual stimuli.

Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources

Abstract: Machine learning (ML) in healthcare presents numerous opportunities for enhancing patient care, population health, and healthcare providers' workflows. However, the real-world clinical and cost benefits remain limited due to challenges in data privacy, heterogeneous data sources, and the inability to fully leverage multiple data modalities. In this perspective paper, we introduce"patchwork learning"(PL), a novel paradigm that addresses these limitations by integrating information from disparate datasets composed of different data modalities (e.g., clinical free-text, medical images, omics) and distributed across separate and secure sites. PL allows the simultaneous utilization of complementary data sources while preserving data privacy, enabling the development of more holistic and generalizable ML models. We present the concept of patchwork learning and its current implementations in healthcare, exploring the potential opportunities and applicable data sources for addressing various healthcare challenges. PL leverages bridging modalities or overlapping feature spaces across sites to facilitate information sharing and impute missing data, thereby addressing related prediction tasks. We discuss the challenges associated with PL, many of which are shared by federated and multimodal learning, and provide recommendations for future research in this field. By offering a more comprehensive approach to healthcare data integration, patchwork learning has the potential to revolutionize the clinical applicability of ML models. This paradigm promises to strike a balance between personalization and generalizability, ultimately enhancing patient experiences, improving population health, and optimizing healthcare providers' workflows.

Learning to Compare Longitudinal Images

Abstract: Longitudinal studies, where a series of images from the same set of individuals are acquired at different time-points, represent a popular technique for studying and characterizing temporal dynamics in biomedical applications. The classical approach for longitudinal comparison involves normalizing for nuisance variations, such as image orientation or contrast differences, via pre-processing. Statistical analysis is, in turn, conducted to detect changes of interest, either at the individual or population level. This classical approach can suffer from pre-processing issues and limitations of the statistical modeling. For example, normalizing for nuisance variation might be hard in settings where there are a lot of idiosyncratic changes. In this paper, we present a simple machine learning-based approach that can alleviate these issues. In our approach, we train a deep learning model (called PaIRNet, for Pairwise Image Ranking Network) to compare pairs of longitudinal images, with or without supervision. In the self-supervised setup, for instance, the model is trained to temporally order the images, which requires learning to recognize time-irreversible changes. Our results from four datasets demonstrate that PaIRNet can be very effective in localizing and quantifying meaningful longitudinal changes while discounting nuisance variation. Our code is available at \url{https://github.com/heejong-kim/learning-to-compare-longitudinal-images.git}

Neural Pre-Processing: A Learning Framework for End-to-end Brain MRI Pre-processing

Abstract: Head MRI pre-processing involves converting raw images to an intensity-normalized, skull-stripped brain in a standard coordinate space. In this paper, we propose an end-to-end weakly supervised learning approach, called Neural Pre-processing (NPP), for solving all three sub-tasks simultaneously via a neural network, trained on a large dataset without individual sub-task supervision. Because the overall objective is highly under-constrained, we explicitly disentangle geometric-preserving intensity mapping (skull-stripping and intensity normalization) and spatial transformation (spatial normalization). Quantitative results show that our model outperforms state-of-the-art methods which tackle only a single sub-task. Our ablation experiments demonstrate the importance of the architecture design we chose for NPP. Furthermore, NPP affords the user the flexibility to control each of these tasks at inference time. The code and model are freely-available at \url{https://github.com/Novestars/Neural-Pre-processing}.

327 Radiopathomics: Integration of Advanced Neuroimaging and Molecular Pathology Features in Meningiomas

Abstract: OBJECTIVES/GOALS: Our overall objective is to investigate the relationship between radiologic features of meningioma with recently identified histopathological and molecular biomarkers, and to apply a machine learning (ML) approach to further demonstrate their utility in predicting clinical outcomes. METHODS/STUDY POPULATION: We have enrolled a cohort of 84 patients with meningioma diagnosed on the basis of conventional gadolinium-enhanced MRI imaging features since September 2019. Each patient has demographic and clinical data, Ga-68-DOTATATE MRI/PET SUV and dynamic metrics, DCE-MRI perfusion parameters, and histopathologic data. Various tumor subregions will be segmented semi-automatically and later confirmed by experienced neuroradiologist. Histopathologic data will include histologic grade, mitotic rate, Ki67 proliferative index, and presence of WHO established atypical histologic features, immunohistochemical parameters, and established high-grade molecular features. We will use supervised learning techniques to develop algorithms for predicting molecular features from imaging phenotypes. RESULTS/ANTICIPATED RESULTS: Anticipated results - advancements in understanding the molecular biomarkers of meningiomas has uncovered genetic alterations and epigenetic changes that more accurately determine tumor behavior. Currently, the imaging correlates of these molecular biomarkers are unknown, and utilizing radiographic data to predict prognosis and imaging-based classifications of meningiomas have not yet been investigated. Validated imaging correlates of molecular biomarkers not only provide an in-vivo assessment of tumor biology, but can also be integrated with histopathologic features ( radiopathomics models’) for more accurate disease prognostication. We anticipate that our results will identify surrogate imaging features for some of the recently emerged molecular biomarkers of meningioma. DISCUSSION/SIGNIFICANCE: There is a paucity of data on the importance of imaging phenotypes in determining tumor biology. This work has the potential of significant clinical impact by enabling a priori molecular characterization of meningiomas at the time of new diagnosis or recurrence, thereby allowing a personalized medicine approach to treatment planning.

A Coordinate-Regression-Based Deep Learning Model for Catheter Detection during Structural Heart Interventions

Abstract: With a growing geriatric population estimated to triple by 2050, minimally invasive procedures that are image-guided are becoming both more popular and necessary for treating a variety of diseases. To lower the learning curve for new procedures, it is necessary to develop better guidance systems and methods to analyze procedure performance. Since fluoroscopy remains the primary mode of visualizations, the ability to perform catheter tracking from fluoroscopic images is an important part of this endeavor. This paper explores the use of deep learning to perform the landmark detection of a catheter from fluoroscopic images in 3D-printed heart models. We show that a two-stage deep-convolutional-neural-network-based model architecture can provide improved performance by initially locating a region of interest before determining the coordinates of the catheter tip within the image. This model has an average error of less than 2% of the image resolution and can be performed within 4 milliseconds, allowing for its potential use for real-time intraprocedural tracking. Coordinate regression models have the advantage of directly outputting values that can be used for quantitative tracking in future applications and are easier to create ground truth values (~50× faster), as compared to semantic segmentation models that require entire masks to be made. Therefore, we believe this work has better long-term potential to be used for a broader class of cardiac devices, catheters, and guidewires.

Optic nerve diameter on non-contrast computed tomography and intracranial hypertension in patients with acute brain injury: A validation study.

Abstract: Intracranial hypertension is a feared complication of acute brain injury that can cause ischemic stroke, herniation, and death. Identifying those at risk is difficult, and the physical examination is often confounded. Given the widespread availability and use of computed tomography (CT) in patients with acute brain injury, prior work has attempted to use optic nerve diameter measurements to identify those at risk of intracranial hypertension. We aimed to validate the use of optic nerve diameter measurements on CT as a screening tool for intracranial hypertension in a large cohort of brain injured patients. We performed a retrospective observational cohort study in a single tertiary referral Neurosciences Intensive Care Unit. We identified patients with documented intracranial pressure measures as part of their routine clinical care who had non-contrast CT head scans collected within 24 hours, and then measured the optic nerve diameters and explored the relationship and test characteristics of these measures to identify those at risk of intracranial hypertension. In a cohort of 314 patients, optic nerve diameter on CT was linearly but weakly associated with intracranial pressure. When used to identify those with intracranial hypertension (>20 mm Hg), the area under the receiver operator curve was 0.68. Using a previously proposed threshold of 0.6 cm, the sensitivity was 81%, specificity 43%, positive likelihood ratio 1.4, and negative likelihood ratio 0.45. CT-derived optic nerve diameter using a threshold of 0.6 cm is sensitive but not specific for intracranial hypertension, and the overall correlation is weak.

Disentangling the distal association between β-Amyloid and tau pathology at varying stages of tau deposition

Abstract: Studies on the histopathology of Alzheimer's disease (AD) strongly suggest that extracellular {beta}-amyloid (A{beta}) plaques promote the spread of neurofibrillary tau tangles. Despite well-documented spatial discrepancies between these two proteinopathies, their association remains elusive. In this study, we aimed to investigate the distal (non-local) association between tau and A{beta} deposition by studying the A{beta}, and tau positron emission tomography (PET) scans of 572 elderly subjects with an average age of 67.11 years old (476 healthy controls (HC), 14 with mild cognitive impairment (MCI), 82 mild AD). We also leveraged 47 tau-PET and 97 A{beta}-PET scans of healthy young individuals (aged 20-40) to find regional cut-points for tau- and A{beta}-positivity in 68 cortical regions in the brain. Based on these cut-points, we implemented a pseudo longitudinal technique to categorize the elderly subjects into four pathologic phases of AD progression: a no-tau phase, a pre-acceleration phase, an acceleration phase, and a post-acceleration phase. We then assessed the distal association between tau and A{beta} in each phase using multiple linear regression models. First, we show that the association between tau and A{beta} starts distally in medial temporal lobe (MTL) regions of tau (e.g., left and right entorhinal cortex and right parahippocampal gyrus) in the early stage of tau aggregation (pre-acceleration phase). We then show that tau in several bilateral brain regions (particularly the entorhinal cortex and parahippocampal gyrus) exhibits strong distal associations with A{beta} in several cortical brain regions during the acceleration phase. We found a weak distal association in the post-acceleration phase, comprising 96% of MCI or mild AD and A{beta}+ subjects. Most importantly, we show that the HC A{beta}+ subjects have the highest degree of distal association between tau and A{beta} of all the subjects in the acceleration phase. The results of this study characterize the distal association between the two key proteinopathies of AD. This information has potential use for understanding disease progression in the brain and for the development of anti-tau therapeutic agents.

A robust and interpretable deep learning framework for multi-modal registration via keypoints

Abstract: We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/evanmy/keymorph.

Empirical Analysis of a Segmentation Foundation Model in Prostate Imaging

Abstract: Most state-of-the-art techniques for medical image segmentation rely on deep-learning models. These models, however, are often trained on narrowly-defined tasks in a supervised fashion, which requires expensive labeled datasets. Recent advances in several machine learning domains, such as natural language generation have demonstrated the feasibility and utility of building foundation models that can be customized for various downstream tasks with little to no labeled data. This likely represents a paradigm shift for medical imaging, where we expect that foundation models may shape the future of the field. In this paper, we consider a recently developed foundation model for medical image segmentation, UniverSeg. We conduct an empirical evaluation study in the context of prostate imaging and compare it against the conventional approach of training a task-specific segmentation model. Our results and discussion highlight several important factors that will likely be important in the development and adoption of foundation models for medical image segmentation.