Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity, these real-world distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated collection of 8 benchmark datasets that reflect a diverse range of distribution shifts which naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training results in substantially lower out-of-distribution than in-distribution performance, and that this gap remains even with models trained by existing methods for handling distribution shifts. This underscores the need for new training methods that produce models which are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at this https URL.

WILDS: A Benchmark of in-the-Wild Distribution Shifts

Image-based cell phenotyping with deep learning.

Cell biology is fundamentally limited in its ability to collect complete data on cellular phenotypes and the wide range of responses to perturbation. Areas such as computer vision and speech recognition have addressed this problem of characterizing unseen or unlabeled conditions with the combined advances of big data, deep learning, and computing resources in the past 5 years. Similarly, recent advances in machine learning approaches enabled by single-cell data start to address prediction tasks in perturbation response modeling. We first define objectives in learning perturbation response in single-cell omics; survey existing approaches, resources, and datasets (https://github.com/theislab/sc-pert); and discuss how a perturbation atlas can enable deep learning models to construct an informative perturbation latent space. We then examine future avenues toward more powerful and explainable modeling using deep neural networks, which enable the integration of disparate information sources and an understanding of heterogeneous, complex, and unseen systems.

Machine learning for perturbational single-cell omics

The elucidation of a compound's Mechanism of Action (MoA) is a challenging task in the drug discovery process, but it is important in order to rationalise phenotypic findings and to anticipate potential side-effects. Bioinformatic approaches, advances in machine learning techniques and the increasing deposition of high-throughput data in public databases have significantly contributed to recent advances in the field, but it is not straightforward to decide which data and methods are most suitable to use in a given case. In this review, we focus on these methods and data and their applications in generating MoA hypotheses for subsequent experimental validation. We discuss compound-specific data such as -omics, cell morphology and bioactivity data, as well as commonly used supplementary prior knowledge such as network and pathway data, and provide information on databases where this data can be accessed. In terms of methodologies, we discuss both well-established methods (connectivity mapping, pathway enrichment) as well as more developing methods (neural networks and multi-omics integration). Finally, we review case studies where the MoA of a compound was successfully suggested from computational analysis by incorporating multiple data modalities and/or methodologies. Our aim for this review is to provide researchers with insights into the benefits and drawbacks of both the data and methods in terms of level of understanding, biases and interpretation - and to highlight future avenues of investigation which we foresee will improve the field of MoA elucidation, including greater public access to -omics data and methodologies which are capable of data integration. 

/pdf/computational-analyses-of-mechanism-of-action-moa-data-37silped.pdf

Computational analyses of mechanism of action (MoA): data, methods and integration

Identifying genetic and chemical perturbations with similar impacts on cell morphology can reveal compounds’ mechanisms of action or novel regulators of genetic pathways. Research on methods for identifying such similarities has lagged due to a lack of carefully designed and well-annotated image sets of cells treated with chemical and genetic perturbations. Here, we create such a Resource dataset, CPJUMP1, where each perturbed gene is a known target of at least two chemical compounds in the dataset. We systematically explore the directionality of correlations among perturbations that target the same gene, and we find that identifying matches between chemical perturbations and genetic perturbations is a challenging task. Our dataset and baseline analyses provide a benchmark for evaluating methods that measure perturbation similarities and impact, and more generally, learn effective representations of cellular state from microscopy images. Such advancements would accelerate the applications of image-based profiling, such as functional genomics and drug discovery.

/pdf/three-million-images-and-morphological-profiles-of-cells-29bj2axv.pdf

Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations

Development of accurate disease models and discovery of immune-modulating drugs is challenged by the immune system’s highly interconnected and context-dependent nature. Here we apply deep-learning-driven analysis of cellular morphology to develop a scalable “phenomics” platform and demonstrate its ability to identify dose-dependent, high-dimensional relationships among and between immunomodulators, toxins, pathogens, genetic perturbations, and small and large molecules at scale. High-throughput screening on this platform demonstrates rapid identification and triage of hits for TGF-β- and TNF-α-driven phenotypes. We deploy the platform to develop phenotypic models of active SARS-CoV-2 infection and of COVID-19-associated cytokine storm, surfacing compounds with demonstrated clinical benefit and identifying several new candidates for drug repurposing. The presented library of images, deep learning features, and compound screening data from immune profiling and COVID-19 screens serves as a deep resource for immune biology and cellular-model drug discovery with immediate impact on the COVID-19 pandemic.

/pdf/functional-immune-mapping-with-deep-learning-enabled-5g637gahip.pdf

Michael A. Statnick

Papers

Functional immune mapping with deep-learning enabled phenomics applied to immunomodulatory and COVID-19 drug discovery