In order to use environmental models effectively for management and decision-making, it is vital to establish an appropriate level of confidence in their performance. This paper reviews techniques available across various fields for characterising the performance of environmental models with focus on numerical, graphical and qualitative methods. General classes of direct value comparison, coupling real and modelled values, preserving data patterns, indirect metrics based on parameter values, and data transformations are discussed. In practice environmental modelling requires the use and implementation of workflows that combine several methods, tailored to the model purpose and dependent upon the data and information available. A five-step procedure for performance evaluation of models is suggested, with the key elements including: (i) (re)assessment of the model's aim, scale and scope; (ii) characterisation of the data for calibration and testing; (iii) visual and other analysis to detect under- or non-modelled behaviour and to gain an overview of overall performance; (iv) selection of basic performance criteria; and (v) consideration of more advanced methods to handle problems such as systematic divergence between modelled and observed values.

Position paper: Characterising performance of environmental models

The coronavirus disease 2019 (COVID-19) pandemic has placed epidemic modeling at the forefront of worldwide public policy making. Nonetheless, modeling and forecasting the spread of COVID-19 remains a challenge. Here, we detail three regional-scale models for forecasting and assessing the course of the pandemic. This work demonstrates the utility of parsimonious models for early-time data and provides an accessible framework for generating policy-relevant insights into its course. We show how these models can be connected to each other and to time series data for a particular region. Capable of measuring and forecasting the impacts of social distancing, these models highlight the dangers of relaxing nonpharmaceutical public health interventions in the absence of a vaccine or antiviral therapies.

/pdf/the-challenges-of-modeling-and-forecasting-the-spread-of-3pp2vhrpx7.pdf

The challenges of modeling and forecasting the spread of COVID-19.

Analysing the properties of a biological system through in silico experimentation requires a satisfactory mathematical representation of the system including accurate values of the model parameters. Fortunately, modern experimental techniques allow obtaining time-series data of appropriate quality which may then be used to estimate unknown parameters. However, in many cases, a subset of those parameters may not be uniquely estimated, independently of the experimental data available or the numerical techniques used for estimation. This lack of identifiability is related to the structure of the model, i.e. the system dynamics plus the observation function. Despite the interest in knowing a priori whether there is any chance of uniquely estimating all model unknown parameters, the structural identifiability analysis for general non-linear dynamic models is still an open question. There is no method amenable to every model, thus at some point we have to face the selection of one of the possibilities. This work presents a critical comparison of the currently available techniques. To this end, we perform the structural identifiability analysis of a collection of biological models. The results reveal that the generating series approach, in combination with identifiability tableaus, offers the most advantageous compromise among range of applicability, computational complexity and information provided.

/pdf/structural-identifiability-of-systems-biology-models-a-4w6riqjlgb.pdf

Structural Identifiability of Systems Biology Models: A Critical Comparison of Methods

DAISY: a new software tool to test global identifiability of biological and physiological systems.

A growing number of population pharmacokinetic analyses of therapeutic monoclonal antibodies (mAbs) have been published in the scientific literature. The aims of this article are to summarize the findings from these studies and to relate the findings to the general pharmacokinetic and structural characteristics of therapeutic mAbs. A two-compartment model was used in the majority of the population analyses to describe the disposition of the mAb. Population estimates of the volumes of distribution in the central (V1) and peripheral (V2) compartments were typically small, with median (range) values of 3.1 (2.4–5.5) L and 2.8 (1.3–6.8) L, respectively. The estimated between-subject variability in the V1 was usually moderate, with a median (range) coefficient of variation (CV) of 26% (12–84%). Between-subject variability in other distribution-related parameters such as the V2 and intercompartmental clearance were often not estimated. Although the pharmacokinetic models used most frequently in the population analyses were models with linear clearance, other models with nonlinear, or parallel linear and nonlinear clearance pathways were also applied, as many therapeutic mAbs are eliminated via saturable target-mediated mechanisms. Population estimates of the maximum elimination rate (Vmax) and the mAb concentration at which elimination was at half maximum for Michaelis-Menten-type elimination pathways varied considerably among the different therapeutic mAbs. However, estimates of the total clearance (CL) of mAbs with linear clearance characteristics and of the clearance of mAbs via the linear clearance pathway (CLL) with parallel linear and nonlinear clearance were quite similar for the different mAbs and typically ranged from 0.2 to 0.5 L/day, which is relatively close to the estimated clearance of endogenous IgG of 0.21 L/day. The between-subject variability in the Vmax, CL and CLL was moderate to high, with estimated CVs ranging from 15% to 65%. Measures of body size were the covariates most commonly identified as influencing the pharmacokinetics of therapeutic mAbs. In summary, many features of the population pharmacokinetics of currently used therapeutic mAbs are similar, despite differences in their pharmacological targets and studied patient populations.

Population pharmacokinetics of therapeutic monoclonal antibodies.

Identifiability of uncontrolled nonlinear rational systems

In this paper, it is shown that the SIR epidemic model, with the force of infection subject to seasonal variation, and a proportion of either the prevalence or the incidence measured, is unidentifiable unless certain key system parameters are known, or measurable. This means that an uncountable number of different parameter vectors can, theoretically, give rise to the same idealised output data. Any subsequent parameter estimation from real data must be viewed with little confidence as a result. The approach adopted for the structural identifiability analysis utilises the existence of an infinitely differentiable transformation that connects the state trajectories corresponding to parameter vectors that give rise to identical output data. When this approach proves computationally intractable, it is possible to use the converse idea that the existence of a coordinate transformation between states for particular parameter vectors implies indistinguishability between these vectors from the corresponding model outputs.

The structural identifiability of the susceptible infected recovered model with seasonal forcing.

In this paper a compartmental modelling approach is applied to provide a mathematical description of the activity of the anti-cancer agent topotecan, and delivery to its nuclear DNA target following administration. The activity of topotecan in defined buffers is first modelled using a linear two compartment model that then forms the basis of a cell based model for drug activity in live cell experiments. An identifiability analysis is performed before parameter estimation to ensure that the model output (i.e., continuous, perfect and noise-free data) uniquely determines the parameters. Parameter estimation is performed using experimental data which offers concentrations of active and inactive forms of topotecan from high performance liquid chromatography methods.

A mathematical model for the in vitro kinetics of the anti-cancer agent topotecan

Under certain controllability and observability restrictions, two different parameterisations for a non-linear compartmental model can only have the same input-output behaviour if they differ by a locally diffeomorphic change of basis for the state space. With further restrictions, it is possible to gain valuable information with respect to identifiability via a linear analysis. Examples are presented where non-linear identifiability analyses are substantially simplified by means of an initial linear analysis. For complex models, with four or more compartments, this linear analysis can prove lengthy to perform by hand and so symbolic computation has been employed to aid this procedure.

Structural identifiability for a class of non-linear compartmental systems using linear/non-linear splitting and symbolic computation.

The paper considers the structural identifiability of a parent–metabolite pharmacokinetic model for ivabradine and one of its metabolites. The model, which is linear, is considered initially for intravenous administration of ivabradine, and then for a combined intravenous and oral administration. In both cases, the model is shown to be unidentifiable. Simplification of the model (for both forms of administration) to that proposed by Duffull et al. (1) results in a globally structurally identifiable model. The analysis could be applied to the modeling of any drug undergoing first-pass metabolism, with plasma concentrations available from drug and metabolite.

M.J. Chapman

Papers

Identifiability of uncontrolled nonlinear rational systems

The structural identifiability of the susceptible infected recovered model with seasonal forcing.

A mathematical model for the in vitro kinetics of the anti-cancer agent topotecan

Structural identifiability for a class of non-linear compartmental systems using linear/non-linear splitting and symbolic computation.

An identifiability analysis of a parent-metabolite pharmacokinetic model for ivabradine.