Human cardiac function is characterized by a complex interplay of mechanical deformation and electrophysiological conduction. Similar to the underlying cardiac anatomy, these interconnected physiological patterns vary considerably across the human population with important implications for the effectiveness of clinical decision-making and the accuracy of computerized heart models. While many previous works have investigated this variability separately for either cardiac anatomy or physiology, this work aims to combine both aspects in a single data-driven approach and capture their intricate interdependencies in a multi-domain setting. To this end, we propose a novel multi-domain Variational Autoencoder (VAE) network to capture combined Electrocardiogram (ECG) and Magnetic Resonance Imaging (MRI)-based 3D anatomy information in a single model. Each VAE branch is specifically designed to address the particular challenges of the respective input domain, enabling efficient encoding, reconstruction, and synthesis of multi-domain cardiac signals. Our method achieves high reconstruction accuracy on a United Kingdom Biobank dataset, with Chamfer Distances between reconstructed and input anatomies below the underlying image resolution and ECG reconstructions outperforming multiple single-domain benchmarks by a considerable margin. The proposed VAE is capable of generating realistic virtual populations of arbitrary size with good alignment in clinical metrics between the synthesized and gold standard anatomies and Maximum Mean Discrepancy (MMD) scores of generated ECGs below those of comparable single-domain approaches. Furthermore, we observe the latent space of our VAE to be highly interpretable with separate components encoding different aspects of anatomical and ECG variability. Finally, we demonstrate that the combined anatomy and ECG representation improves the performance in a cardiac disease classification task by 3.9% in terms of Area Under the Receiver Operating Characteristic (AUROC) curve over the best corresponding single-domain modeling approach.

/pdf/multi-domain-variational-autoencoders-for-combined-modeling-36wqf2mp.pdf

Multi-Domain Variational Autoencoders for Combined Modeling of MRI-Based Biventricular Anatomy and ECG-Based Cardiac Electrophysiology

Understanding population-wide variability of the human heart is crucial to detect abnormalities and improve the assessment of both cardiac anatomy and function. While many computational modeling approaches have been developed to capture this variability separately for either cardiac anatomy or physiology, their complex interconnections have rarely been explored together. In this work, we propose a novel multi-modal variational autoencoder (VAE) capable of processing combined physiology and bitemporal anatomy information in the form of electrocardiograms (ECG) and 3D biventricular point clouds. Our method achieves high reconstruction accuracy on a UK Biobank dataset with Chamfer distances between predicted and input anatomies below the underlying image resolution and the ECG reconstructions outperforming a state-of-the-art benchmark approach specialized in ECG generation. We also evaluate its generative ability and find comparable populations of generated and gold standard anatomies, ECGs, and combined anatomy-ECG data in terms of common clinical metrics and maximum mean discrepancies.

Combined Generation of Electrocardiogram and Cardiac Anatomy Models Using Multi-Modal Variational Autoencoders

Understanding population-wide variability of the human heart is crucial to detect abnormalities and improve the assessment of both cardiac anatomy and function. While many computational modeling approaches have been developed to capture this variability separately for either cardiac anatomy or physiology, their complex interconnections have rarely been explored together. In this work, we propose a novel multi-modal variational autoencoder (VAE) capable of processing combined physiology and bitemporal anatomy information in the form of electrocardiograms (ECG) and 3D biventricular point clouds. Our method achieves high reconstruction accuracy on a UK Biobank dataset with Chamfer distances between predicted and input anatomies below the underlying image resolution and the ECG reconstructions outperforming a state-of-the-art benchmark approach specialized in ECG generation. We also evaluate its generative ability and find comparable populations of generated and gold standard anatomies, ECGs, and combined anatomy-ECG data in terms of common clinical metrics and maximum mean discrepancies. 

/pdf/combined-generation-of-electrocardiogram-and-cardiac-anatomy-i7l19bh1.pdf

Cardiac anatomy and function vary considerably across the human population with important implications for clinical diagnosis and treatment planning. Consequently, many computer-based approaches have been developed to capture this variability for a wide range of applications, including explainable cardiac disease detection and prediction, dimensionality reduction, cardiac shape analysis, and the generation of virtual heart populations. In this work, we propose a variational mesh autoencoder (mesh VAE) as a novel geometric deep learning approach to model such population-wide variations in cardiac shapes. It embeds multi-scale graph convolutions and mesh pooling layers in a hierarchical VAE framework to enable direct processing of surface mesh representations of the cardiac anatomy in an efficient manner. The proposed mesh VAE achieves low reconstruction errors on a dataset of 3D cardiac meshes from over 1,000 patients with acute myocardial infarction, with mean surface distances between input and reconstructed meshes below the underlying image resolution. We also find that it outperforms a voxelgrid-based deep learning benchmark in terms of both mean surface distance and Hausdorff distance while requiring considerably less memory. Furthermore, we explore the quality and interpretability of the mesh VAE's latent space and showcase its ability to improve the prediction of major adverse cardiac events over a clinical benchmark. Finally, we investigate the method's ability to generate realistic virtual populations of cardiac anatomies and find good alignment between the synthesized and gold standard mesh populations in terms of multiple clinical metrics.

Interpretable cardiac anatomy modeling using variational mesh autoencoders

Cine magnetic resonance imaging (MRI) is the gold standard modality for the assessment of cardiac anatomy and function. However, a standard cine acquisition typically consists of only a set of intersecting 2D image slices to represent the true 3D geometry of the human heart, thus limiting its utility in various clinical and research settings. In this work, we present a novel geometric deep learning method, Point2Mesh-Net, to directly and efficiently transform a set of 2D MRI slices into 3D cardiac surface meshes. Its architecture consists of an encoder and a decoder, which are based on recent advances in point cloud and mesh-based deep learning, respectively. This allows the network to not only directly process point cloud data, which represents the sparse MRI contours obtained from image segmentation, but also to output 3D triangular surface meshes, which are highly suitable for a variety of follow-up tasks. Furthermore, the Point2Mesh-Net’s hierarchical setup with multiple downsampling and upsampling steps enables multi-scale feature learning and helps the network to successfully overcome the two main challenges of cardiac surface reconstruction: data sparsity and slice misalignment. We evaluate the model on a synthetic dataset derived from a 3D MRI-based statistical shape model and find surface distances between reconstructed and gold standard meshes below the underlying image resolution for multiple anatomical substructures of the heart. In addition, we apply the pre-trained Point2Mesh-Net as part of a multi-step pipeline to cine MRI acquisitions of the UK Biobank dataset and observe realistic mesh reconstructions with various clinical metrics in line with corresponding findings of large-scale population studies. 

Point2Mesh-Net: Combining Point Cloud and Mesh-Based Deep Learning for Cardiac Shape Reconstruction

Many important cardiac biomarkers used in clinical practice describe cardiac anatomy and function in three dimensions (3D). However, common cardiac magnetic resonance imaging (MRI) protocols often only generate two-dimensional (2D) image slices of the underlying 3D anatomy and are susceptible to various types of motion artifacts causing slice misalignment. In this paper, we propose a deep learning method acting directly on point clouds to reconstruct a dense 3D biventricular heart model from misaligned 2D cardiac MR image contours. The method is able to reduce mild, medium, and strong slice misalignments (mean translation $\sim 3.5$ mm; mean rotation $\sim 2.5^{\circ})$ to a Chamfer distance below image resolution (1.25 mm) with high robustness (standard deviation 0.18 mm) on a statistical shape model dataset. It also manages to reconstruct smooth 3D shapes with accurate left ventricular volumes from cine MR images of the UK Biobank study.

/pdf/biventricular-surface-reconstruction-from-cine-mri-contours-38xjic176t.pdf

Biventricular Surface Reconstruction From Cine Mri Contours Using Point Completion Networks

Generative statistical models have a wide variety of applications in modelling of cardiac anatomy and function, including disease diagnosis and prediction, personalized shape analysis, and generation of population cohorts for electrophysiological and mechanical computer simulations. In this work, we propose a novel geometric deep learning method based on the variational autoencoder (VAE) framework capable of accurately encoding, reconstructing, and synthesizing 3D surface models of the biventricular anatomy. Our non-linear approach works directly with memory-efficient point clouds and is able to process multiple substructures of the cardiac anatomy at the same time in a multi-class setting. Furthermore, we introduce subpopulation-specific characteristics as additional conditional inputs to allow the generation of new personalized anatomies. Our method achieves high reconstruction quality on a dataset derived from the UK Biobank study with average Chamfer distances between reconstructed and gold standard point clouds below the underlying image pixel resolution, for all anatomical substructures and combinations of conditional inputs. We investigate our method’s generative capabilities and show that it is able to synthesize virtual populations of realistic hearts with volumetric measurements in line with established clinical precedent. We also analyse the effects of variations in the latent space of the autoencoder on the generated anatomies and find interpretable changes in cardiac shapes and sizes. 

/pdf/generating-subpopulation-specific-biventricular-anatomy-3n0epok9.pdf

Generating Subpopulation-Specific Biventricular Anatomy Models Using Conditional Point Cloud Variational Autoencoders

Mechanical contraction and relaxation of the heart play an important role in evaluating healthy and diseased cardiac function. Mechanical patterns consist of complex non-linear 3D deformations that vary considerably between subjects and are difficult to observe on 2D images, which impacts the prediction accuracy of cardiac outcomes. In this work, we aim to capture 3D biventricular deformations at the end-diastolic (ED) and end-systolic (ES) phases of the cardiac cycle with a novel geometric deep learning approach. Our network consists of an encoder-decoder structure that works directly with light-weight point cloud data. We initially train our network on pairs of ED and ES point clouds stemming from a mixed population of subjects with the aim of accurately predicting ED outputs from ES inputs as well as ES outputs from ED inputs. We validate our network’s performance using the Chamfer distance (CD) and find that ED and ES predictions can be achieved with an average CD of 1.66 ± 0.62 mm on a dataset derived from the UK Biobank cohort with an underlying voxel size of $$1.8 \times 1.8 \times 8.0$$ mm [8]. We derive structural and functional clinical metrics such as myocardial mass, ventricular volume, ejection fraction, and stroke volume from the predictions and find an average mean deviation from their respective gold standards of 1.6% and comparable standard deviations. Finally, we show our method’s ability to capture deformation differences between specific subpopulations in the dataset. 

/pdf/predicting-3d-cardiac-deformations-with-point-cloud-2kl16shr.pdf

Predicting 3D Cardiac Deformations with Point Cloud Autoencoders

Cardiac image segmentation is a crucial step in clinical practice as it allows for the assessment of cardiac morphology and the quantification of image-based biomarkers. While deep learning methods have recently achieved near human-level performance on large, single-domain cine MRI datasets, their accuracy decreases considerably in more complex multi-domain settings, limiting their clinical applicability. To this end, we propose a novel multi-view crossover cascade approach combined with both shape and appearance augmentations for effective multi-domain cardiac image segmentation. Our cascade consists of two Attention U-Net paths that share information across different views and an intermediate heart location crop to reduce variance and improve label balance. In addition to multiple shape augmentations (scaling, elastic deformations, grid distortions, etc.) and histogram matching, we introduce multi-scale Fourier Domain Adaptation to cardiac image analysis. We evaluate both the crossover cascade and the augmentations on the cine MRI dataset of the M&Ms-2 challenge and outperform a U-Net benchmark by respective Dice score increases of $$\sim $$ 0.02 and $$\sim $$ 0.03. 

https://zenodo.org/record/6324370/files/A%20Multi-view%20Crossover%20Attention%20U-Net%20Cascade%20with%20Fourier%20Domain%20Adaptation%20for%20Multi-domain%20Cardiac.pdf

Marcel Beetz

Papers

Biventricular Surface Reconstruction From Cine Mri Contours Using Point Completion Networks

Generating Subpopulation-Specific Biventricular Anatomy Models Using Conditional Point Cloud Variational Autoencoders

Predicting 3D Cardiac Deformations with Point Cloud Autoencoders

A Multi-view Crossover Attention U-Net Cascade with Fourier Domain Adaptation for Multi-domain Cardiac MRI Segmentation