Comprehensive, high-quality reference genomes are required for functional characterization and taxonomic assignment of the human gut microbiota. We present the Unified Human Gastrointestinal Genome (UHGG) collection, comprising 204,938 nonredundant genomes from 4,644 gut prokaryotes. These genomes encode >170 million protein sequences, which we collated in the Unified Human Gastrointestinal Protein (UHGP) catalog. The UHGP more than doubles the number of gut proteins in comparison to those present in the Integrated Gene Catalog. More than 70% of the UHGG species lack cultured representatives, and 40% of the UHGP lack functional annotations. Intraspecies genomic variation analyses revealed a large reservoir of accessory genes and single-nucleotide variants, many of which are specific to individual human populations. The UHGG and UHGP collections will enable studies linking genotypes to phenotypes in the human gut microbiome.

/pdf/a-unified-catalog-of-204-938-reference-genomes-from-the-4hl9uvu58y.pdf

A unified catalog of 204,938 reference genomes from the human gut microbiome.

The gut microbiome is shaped by diet and influences host metabolism; however, these links are complex and can be unique to each individual. We performed deep metagenomic sequencing of 1,203 gut microbiomes from 1,098 individuals enrolled in the Personalised Responses to Dietary Composition Trial (PREDICT 1) study, whose detailed long-term diet information, as well as hundreds of fasting and same-meal postprandial cardiometabolic blood marker measurements were available. We found many significant associations between microbes and specific nutrients, foods, food groups and general dietary indices, which were driven especially by the presence and diversity of healthy and plant-based foods. Microbial biomarkers of obesity were reproducible across external publicly available cohorts and in agreement with circulating blood metabolites that are indicators of cardiovascular disease risk. While some microbes, such as Prevotella copri and Blastocystis spp., were indicators of favorable postprandial glucose metabolism, overall microbiome composition was predictive for a large panel of cardiometabolic blood markers including fasting and postprandial glycemic, lipemic and inflammatory indices. The panel of intestinal species associated with healthy dietary habits overlapped with those associated with favorable cardiometabolic and postprandial markers, indicating that our large-scale resource can potentially stratify the gut microbiome into generalizable health levels in individuals without clinically manifest disease.

/pdf/microbiome-connections-with-host-metabolism-and-habitual-47b6kl2fxz.pdf

Microbiome connections with host metabolism and habitual diet from 1,098 deeply phenotyped individuals.

Microbial and viral communities transform the chemistry of Earth's ecosystems, yet the specific reactions catalyzed by these biological engines are hard to decode due to the absence of a scalable, metabolically resolved, annotation software. Here, we present DRAM (Distilled and Refined Annotation of Metabolism), a framework to translate the deluge of microbiome-based genomic information into a catalog of microbial traits. To demonstrate the applicability of DRAM across metabolically diverse genomes, we evaluated DRAM performance on a defined, in silico soil community and previously published human gut metagenomes. We show that DRAM accurately assigned microbial contributions to geochemical cycles and automated the partitioning of gut microbial carbohydrate metabolism at substrate levels. DRAM-v, the viral mode of DRAM, established rules to identify virally-encoded auxiliary metabolic genes (AMGs), resulting in the metabolic categorization of thousands of putative AMGs from soils and guts. Together DRAM and DRAM-v provide critical metabolic profiling capabilities that decipher mechanisms underpinning microbiome function.

/pdf/dram-for-distilling-microbial-metabolism-to-automate-the-31m4c5a2dg.pdf

DRAM for distilling microbial metabolism to automate the curation of microbiome function

Gene Expression Changes and Community Turnover Differentially Shape the Global Ocean Metatranscriptome

Our current knowledge about nucleocytoplasmic large DNA viruses (NCLDVs) is largely derived from viral isolates that are co-cultivated with protists and algae. Here we reconstructed 2,074 NCLDV genomes from sampling sites across the globe by building on the rapidly increasing amount of publicly available metagenome data. This led to an 11-fold increase in phylogenetic diversity and a parallel 10-fold expansion in functional diversity. Analysis of 58,023 major capsid proteins from large and giant viruses using metagenomic data revealed the global distribution patterns and cosmopolitan nature of these viruses. The discovered viral genomes encoded a wide range of proteins with putative roles in photosynthesis and diverse substrate transport processes, indicating that host reprogramming is probably a common strategy in the NCLDVs. Furthermore, inferences of horizontal gene transfer connected viral lineages to diverse eukaryotic hosts. We anticipate that the global diversity of NCLDVs that we describe here will establish giant viruses-which are associated with most major eukaryotic lineages-as important players in ecosystems across Earth's biomes.

/pdf/giant-virus-diversity-and-host-interactions-through-global-i2x9zbl54c.pdf

Giant virus diversity and host interactions through global metagenomics

We previously reported on MetaBAT, an automated metagenome binning software tool to reconstruct single genomes from microbial communities for subsequent analyses of uncultivated microbial species. MetaBAT has become one of the most popular binning tools largely due to its computational efficiency and ease of use, especially in binning experiments with a large number of samples and a large assembly. MetaBAT requires users to choose parameters to fine-tune its sensitivity and specificity. If those parameters are not chosen properly, binning accuracy can suffer, especially on assemblies of poor quality. Here, we developed MetaBAT 2 to overcome this problem. MetaBAT 2 uses a new adaptive binning algorithm to eliminate manual parameter tuning. We also performed extensive software engineering optimization to increase both computational and memory efficiency. Comparing MetaBAT 2 to alternative software tools on over 100 real world metagenome assemblies shows superior accuracy and computing speed. Binning a typical metagenome assembly takes only a few minutes on a single commodity workstation. We therefore recommend the community adopts MetaBAT 2 for their metagenome binning experiments. MetaBAT 2 is open source software and available at https://bitbucket.org/berkeleylab/metabat.

/pdf/metabat-2-an-adaptive-binning-algorithm-for-robust-and-3770aaeh8l.pdf

MetaBAT 2: an adaptive binning algorithm for robust and efficient genome reconstruction from metagenome assemblies.

Metagenomics, the study of all microbial species cohabitants in an environment, often produces large amount of sequence data varying from several GBs to a few TBs. Analyzing metagenomics data includes both data-intensive and compute-intensive steps, making the entire process hard to scale. Here we aim to optimize a metagenomics application that partitions the shortgun metagenomics sequences based on their species of origin. Our solution combines MapReduce-based BioPig analytic toolkit with MPI to provide scalability in respective to both data and compute. We also made some improvements to the existing BioPig toolkit by using simplified data types and compressed k-mer storage. These optimizations leads up to 193
 $$\times $$
 speedup for the computing-intensive step and 9.6
 $$\times $$
 speedup over the entire pipeline. Our optimized application is also capable of processing datasets that are 16 times larger on the same hardware platform. These results suggest integrating heterogeneous technologies such as Hadoop and MPI is quite efficient to solve large genomics problems that are both data-intensive and compute-intensive.

/pdf/combining-hadoop-with-mpi-to-solve-metagenomics-problems-481p86t7b0.pdf

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

ABSTRACT With the aid of the newly developed ‘Sunway’ heterogeneous-architecture supercomputer, which has world-leading HPC (high-performance computer) capability, a series of high-resolution coupled Earth system models (SW-HRESMs) with up to 5 km of atmosphere and 3 km of ocean have been developed. These models can meet the needs of multiscale interaction studies with different computational costs. Here we describe the progress of SW-HRESMs development, with an overview of the major advancements made by the international Earth science community in HR-ESMs. We also show the preliminary results of SW-HRESMs with regard to capturing major weather-climate extremes in the atmosphere and ocean, stressing the importance of permitted clouds and ocean submesoscale eddies in modeling tropical cyclones and eddy-mean flow interactions, and paving the way for further model development to resolve finer scales with even higher resolution and more realistic physics. Finally, in addition to increasing model resolution, the development procedure for a non-hydrostatic cloud and ocean submesoscale resolved ESM is discussed, laying out the major scientific directions of such a huge modeling advancement.

/pdf/toward-earth-system-modeling-with-resolved-clouds-and-ocean-290llaxi.pdf

Toward Earth system modeling with resolved clouds and ocean submesoscales on heterogeneous many-core HPCs

As innovations in AI/ML heterogeneous systems and DNN models continue to grow, there is a need for accurate performance analysis methodology to understand and navigate this complex SW-HW interplay to support efficient running of DNN models for future systems. The great success of dataflow paradigm for representing the neural network computation with dataflow graph has drawn attentions from both industry and academia. For the given deep learning workloads, to organize the corresponding directed acyclic graphs (DAGs) on the dedicated heterogeneous environment involves a complex interplay between DNN algorithms, hardware, algorithm-to-hardware mapping. In this work, we propose a fine-grained dataflow runtime simulator for achieving highly accurate performance modeling towards the deep learning workloads. It makes two major contributions. (i) The end-to-end ML method is used to realize the performance modeling of DNN operator, task-based modeling with cycle-accurate simulation. (ii) Research on task scheduling and task division of heterogeneous data flow runtime system. Based on our dataflow runtime simulator, we systematically analyze the computational graph scheduling and data flow granularity of typical AI workloads in complex heterogeneous hardware environments from the dataflow perspective. As an example of the dataflow method in Huawei Ascend 910 scenarios, it provides a reference build performance model for fast and accurate performance (e.g. iteration training time) prediction without actually executing DNN models on the dedicated hardware can better realize software and hardware co-design on heterogeneous systems.

A Systematic Methodology for performance characterizing of Heterogeneous Systems with a dataflow runtime simulator

High performance computing has been developing rapidly over the past decades. Nowadays, most super computers are still using traditional pure CPU architecture or very limited heterogeneous devices like Nvidia GPU. Many other vendor's devices are also very reliable, such as Intel GPU, which was first introduced few years ago. As programming methods on most heterogeneous devices are always very strongly restricted by vendors, porting from one platform to another is usually difficult and time-costing. With the great help of ONEAPI, applications can be ported to Intel GPU and Intel FPGA with almost zero cost, rather than rebuild kernel codes in CUDA. This paper demonstrates the porting work of an important and widely used application, the GW approximation in the First Principle. We introduce the whole workflow of how to port applications to Intel ONEAPI, which allows us to execute application on all platforms. To fully use all devices, we introduce an automatic heterogeneous device sneaking and workload balance method in this paper. Moreover, we develop a series of methods of how to achieve workload balance for each single core in GPU, and design a better reduction method than all previous works. The tests are taken on 2 Intel Xe-HP ATS-P GPUs, and the results show that our method achieves an another 65x acceleration over traditional OpenMP + MPI implementation.

Hong An

Papers

MetaBAT 2: an adaptive binning algorithm for robust and efficient genome reconstruction from metagenome assemblies.

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Toward Earth system modeling with resolved clouds and ocean submesoscales on heterogeneous many-core HPCs

A Systematic Methodology for performance characterizing of Heterogeneous Systems with a dataflow runtime simulator

An Accelerated First Principle Method Implemented on IntelGPU