Modeling & Simulation has played an essential role in supporting the decision-making activities of policymakers for COVID-19. However, a proliferation of models has been noted in the literature, and new models are only more likely to emerge given the shift to long-term management of the disease and the call for highly tailored tools. Having a multiplicity of models can have benefits, for example when contributing to ensembles of models. However, if each model is created from scratch, there is significant redundancy in efforts hence time inefficiency and a heightened risk of bugs. Our study examines the naturally occurring practices of modelers who wrote COVID-19 models in NetLogo to identify redundancy in code and thus suggest reusable ‘building blocks’ that would speed-up the process of model development as well as improving code quality. Based on 28 models, we identified five themes and discussed their transformation into potential building blocks for simulation.

Towards Reusable Building Blocks to Develop COVID-19 Simulation Models

The COVID-19 pandemic has infected over 200 million people worldwide and killed more than 4 million as of August 2021. Many intervention strategies have been utilized by governments around the world, including masks, social distancing, and vaccinations. However, officials making decisions regarding interventions may have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also have limitations due to requirements on processing power or time. This paper examines whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories very close to the original simulation results. Using four previously published agent-based models for COVID-19, this paper analyzes the predictions of decision tree regression machine learning models and compares them to the results of the original simulations. The results indicate that accurate machine learning meta-models can be generated from simulation models with no strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data. However, meta-models for simulation models that include strong interventions required much more training data to achieve a similar accuracy. This indicates that machine learning meta-models could be used in some scenarios to assist in faster decision making.

/pdf/when-do-we-need-massive-computations-to-perform-detailed-1m5mgqcaau.pdf

When do we need massive computations to perform detailed COVID-19 simulations?

Agent-based modeling (ABM) is a well-established computational paradigm for simulating complex systems in terms of the interactions between individual entities that comprise the system’s population. Machine learning (ML) refers to computational approaches whereby algorithms use statistical methods to “learn” from data on their own, i.e., without imposing any a priori model/theory onto a system or its behavior. Biological systems—ranging from molecules, to cells, to entire organisms, to whole populations and even ecosystems—consist of vast numbers of discrete entities, governed by complex webs of interactions that span various spatiotemporal scales and exhibit nonlinearity, stochasticity, and variable degrees of coupling between entities. For these reasons, the macroscopic properties and collective dynamics of biological systems are generally difficult to accurately model or predict via continuum modeling techniques and mean-field formalisms. ABM takes a “bottom-up” approach that obviates common difficulties of other modeling approaches by enabling one to relatively easily create (or at least propose, for testing) a set of well-defined “rules” to be applied to the individual entities (agents) in a system. Quantitatively evaluating a system and propagating its state over a series of discrete time-steps effectively simulates the system, allowing various observables to be computed and the system’s properties to be analyzed. Because the rules that govern an ABM can be difficult to abstract and formulate from experimental data, at least in an unbiased way, there is a uniquely synergistic opportunity to employ ML to help infer optimal, system-specific ABM rules. Once such rule-sets are devised, running ABM calculations can generate a wealth of data, and ML can be applied in that context too—for example, to generate statistical measures that accurately and meaningfully describe the stochastic outputs of a system and its properties. As an example of synergy in the other direction (from ABM to ML), ABM simulations can generate plausible (realistic) datasets for training ML algorithms (e.g., for regularization, to mitigate overfitting). In these ways, one can envision a variety of synergistic ABM⇄ML loops. After introducing some basic ideas about ABMs and ML, and their limitations, this Review describes examples of how ABM and ML have been integrated in diverse contexts, spanning spatial scales that include multicellular and tissue-scale biology to human population-level epidemiology. In so doing, we have used published studies as a guide to identify ML approaches that are well-suited to particular types of ABM applications, based on the scale of the biological system and the properties of the available data.

Innovations in integrating machine learning and agent-based modeling of biomedical systems

The virus that causes COVID-19 changes over time, occasionally leading to Variants of Interest (VOIs) and Variants of Concern (VOCs) that can behave differently with respect to detection kits, treatments, or vaccines. For instance, two vaccination doses were 61% effective against the BA.1 predominant variant, but only 24% effective when BA.2 became predominant. While doses still confer protection against severe disease outcomes, the BA.5 variant demonstrates the possibility that individuals who have received a few doses built for previous variants can still be infected with newer variants. As previous vaccines become less effective, new ones will be released to target specific variants and the whole process of vaccinating the population will restart. While previous models have detailed logistical aspects and disease progression, there are three additional key elements to model COVID-19 vaccination coverage in the long term. First, the willingness of the population to participate in regular vaccination campaigns is essential for long-term effective COVID-19 vaccination coverage. Previous research has shown that several categories of variables drive vaccination status: sociodemographic, health-related, psychological, and information-related constructs. However, the inclusion of these categories in future models raises questions about the identification of specific factors (e.g., which sociodemographic aspects?) and their operationalization (e.g., how to initialize agents with a plausible combination of factors?). While previous models separately accounted for natural- and vaccine-induced immunity, the reality is that a significant fraction of individuals will be both vaccinated and infected over the coming years. Modeling the decay in immunity with respect to new VOCs will thus need to account for hybrid immunity. Finally, models rarely assume that individuals make mistakes, even though this over-reliance on perfectly rational individuals can miss essential dynamics. Using the U.S. as a guiding example, our scoping review summarizes these aspects (vaccinal choice, immunity, and errors) through ten recommendations to support the modeling community in developing long-term COVID-19 vaccination models.

/pdf/a-scoping-review-of-three-dimensions-for-long-term-covid-19-2knees4k.pdf

A Scoping Review of Three Dimensions for Long-Term COVID-19 Vaccination Models: Hybrid Immunity, Individual Drivers of Vaccinal Choice, and Human Errors

Modeling & Simulation has played an essential role in supporting the decision-making activities of policymakers for COVID-19. However, a proliferation of models has been noted in the literature, and new models are only more likely to emerge given the shift to long-term management of the disease and the call for highly tailored tools. Having a multiplicity of models can have benefits, for example when contributing to ensembles of models. However, if each model is created from scratch, there is significant redundancy in efforts hence time inefficiency and a heightened risk of bugs. Our study examines the naturally occurring practices of modelers who wrote COVID-19 models in NetLogo to identify redundancy in code and thus suggest reusable ‘building blocks’ that would speed-up the process of model development as well as improving code quality. Based on 28 models, we identified five themes and discussed their transformation into potential building blocks for simulation. 

The COVID-19 pandemic has infected over 250 million people worldwide and killed more than 5 million as of November 2021. Many intervention strategies are utilized (e.g., masks, social distancing, vaccinations), but officials making decisions have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also require significant processing power or time. It is examined whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories resembling the original simulation results. Using four previously published agent-based models (ABMs) for COVID-19, a decision tree regression for each ABM is built and its predictions are compared to the corresponding ABM. Accurate machine learning meta-models are generated from ABMs without strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data: the root-mean-square error (RMSE) with 25% of the data is close to the RMSE for the full dataset (0.15 vs 0.14 in one model;0.07 vs 0.06 in another). However, meta-models for ABMs employing strong interventions require much more training data (at least 60%) to achieve a similar accuracy. In conclusion, machine learning meta-models can be used in some scenarios to assist in faster decision-making. © 2021 Wiley-VCH GmbH

When Do We Need Massive Computations to Perform Detailed COVID-19 Simulations?

Computer simulations are used in precision medicine to assist in adapting treatment plans for varying patient characteristics, especially for diseases like HIV where these characteristics have a major impact on disease trajectory. However, simulations are computationally intensive, which can be prohibitive at scale. Meta-models for HIV progression have been developed previously to approximate these simulation results more efficiently, but we are interested in determining how much data is required to build an accurate meta-model. Using many different amounts of data from two HIV simulation models, we build machine learning classification meta-models to predict if an HIV patient is at risk for AIDS based on treatment parameters. Our findings indicate that the amount required to achieve high meta-model accuracy varies for different computer simulations. We are able to achieve near-perfect accuracy with one of our models using limited data, while the other model requires more extensive data to achieve stable accuracy.

How Many Costly Simulations Do we Need to Create Accurate Metamodels? A Case Study on Predicting HIV Viral Load in Response to Clinically Relevant Intervention Scenarios

Christopher B Lutz

Papers

When Do We Need Massive Computations to Perform Detailed COVID-19 Simulations?

When do we need massive computations to perform detailed COVID-19 simulations?

How Many Costly Simulations Do we Need to Create Accurate Metamodels? A Case Study on Predicting HIV Viral Load in Response to Clinically Relevant Intervention Scenarios