Clinical practice involves measuring quantities for a variety of purposes, such as aiding diagnosis, predicting future patient outcomes, and serving as endpoints in studies or randomized trials. Measurements are almost always prone to various sorts of errors, which cause the measured value to differ from the true value; accordingly, studies investigating measurement error frequently appear in this and other journals. The importance of measurement error depends upon the context in which the measurements in question are to be used. For example, a certain degree of measurement error may be acceptable if measurements are to be used as an outcome in a comparative study such as a clinical trial, but the same measurement errors may be unacceptably large to make measurements usable in individual patient management, such as screening or risk prediction. In the past 20 years many papers have been published advocating how studies of measurement error should be analyzed, with a paper by Bland and Altman1 being one of the most cited and well known examples. There has been much controversy concerning the choice of parameter to be estimated and reported, and consequently confusion surrounding the meaning and interpretation of results from studies investigating measurement error. In this paper we first distinguish between the general concepts of agreement and reliability to aid researchers in considering which are relevant for their particular application. We then review the statistical methods that can be used to investigate and quantify agreement and reliability, dealing separately with the different types of measurement error study, while emphasizing the largely common techniques that should be used for data analysis. We reiterate that the judgment of whether agreement or reliability are acceptable must be related to the clinical application, and cannot be proven by a statistical test. We highlight the fact that reliability depends on the population in which measurements are made, and not just on the measurement errors of the measurement method. We discuss the advantages of method comparison studies making at least two measurements with each measurement method on each subject. A key advantage is that the cause of a correlation between paired differences and means in the so-called Bland–Altman plot can be determined, in contrast to when only a single measurement is made with each method. Throughout the paper, we try to emphasize that calculated values of agreement and reliability from measurement error studies are estimates of parameters, and as such we should report such estimates with CIs to indicate the uncertainty with which they have been estimated. We restrict our attention to measurements of a continuous quantity; alternative methods are required for categorical data2.

https://obgyn.onlinelibrary.wiley.com/doi/pdf/10.1002/uog.5256

Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables

https://nutritionmodels.com/papers/Tedeschi2004.pdf

Assessment of the adequacy of mathematical models

Understanding the community assembly mechanisms controlling biodiversity patterns is a central issue in ecology. Although it is generally accepted that both deterministic and stochastic processes play important roles in community assembly, quantifying their relative importance is challenging. Here we propose a general mathematical framework to quantify ecological stochasticity under different situations in which deterministic factors drive the communities more similar or dissimilar than null expectation. An index, normalized stochasticity ratio (NST), was developed with 50% as the boundary point between more deterministic ( 50%) assembly. NST was tested with simulated communities by considering abiotic filtering, competition, environmental noise, and spatial scales. All tested approaches showed limited performance at large spatial scales or under very high environmental noise. However, in all of the other simulated scenarios, NST showed high accuracy (0.90 to 1.00) and precision (0.91 to 0.99), with averages of 0.37 higher accuracy (0.1 to 0.7) and 0.33 higher precision (0.0 to 1.8) than previous approaches. NST was also applied to estimate stochasticity in the succession of a groundwater microbial community in response to organic carbon (vegetable oil) injection. Our results showed that community assembly was shifted from more deterministic (NST = 21%) to more stochastic (NST = 70%) right after organic carbon input. As the vegetable oil was consumed, the community gradually returned to be more deterministic (NST = 27%). In addition, our results demonstrated that null model algorithms and community similarity metrics had strong effects on quantifying ecological stochasticity.

/pdf/a-general-framework-for-quantitatively-assessing-ecological-rj4z6z343m.pdf

A general framework for quantitatively assessing ecological stochasticity

Although investigators in the imaging community have been active in developing and evaluating quantitative imaging biomarkers (QIBs), the development and implementation of QIBs have been hampered by the inconsistent or incorrect use of terminology or methods for technical performance and statistical concepts. Technical performance is an assessment of how a test performs in reference objects or subjects under controlled conditions. In this article, some of the relevant statistical concepts are reviewed, methods that can be used for evaluating and comparing QIBs are described, and some of the technical performance issues related to imaging biomarkers are discussed. More consistent and correct use of terminology and study design principles will improve clinical research, advance regulatory science, and foster better care for patients who undergo imaging studies.

Metrology Standards for Quantitative Imaging Biomarkers.

Validity, reliability, and reproducibility of plaster vs digital study models: Comparison of peer assessment rating and Bolton analysis and their constituent measurements

Measurements of agreement are needed to assess the acceptability of a new or generic process, methodology, and formulation in areas of laboratory performance, instrument or assay validation, method comparisons, statistical process control, goodness of fit, and individual bioequivalence. In all of these areas, one needs measurements that capture a large proportion of data that are within a meaningful boundary from target values. Target values can be considered random (measured with error) or fixed (known), depending on the situation. Various meaningful measures to cope with such diverse and complex situations have become available only in the last decade. These measures often assume that the target values are random. This article reviews the literature and presents methodologies in terms of “coverage probability.” In addition, analytical expressions are introduced for all of the aforementioned measurements when the target values are fixed and when the error structure is homogenous or heterogeneous (proport...

/pdf/statistical-methods-in-assessing-agreement-models-issues-and-21wcznp471.pdf

Statistical methods in assessing agreement: Models, issues, and tools

Introduction. - Basic approach for paired continuous data when target values are random or fixed. - Sample size and power. - Unified approach for continuous and categorical data.

/pdf/statistical-tools-for-measuring-agreement-4dab4nzm74.pdf

Statistical Tools for Measuring Agreement

This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches infinity, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.

A Unified Approach for Assessing Agreement for Continuous and Categorical Data

This paper is an overview that summarizes recently developed tools in assessing agreement for methods comparison and instrument/assay validation in medical devices. This paper emphasizes concept, sample sizes, and examples more than analytical formulas. We have considered a unified approach of evaluating agreement among multiple instruments (k), each with multiple replicates (m) for both continuous and categorical data. We start with the basic scenario of two instruments (k = 2), each with only one measurement (m = 1). In this basic scenario for continuous data, we also consider if the target values are considered random (values of a gold standard instrument) or fixed (known values). In the more general case, we will not consider when the target values are fixed. We discuss the simplified sample size calculations. When there is a disagreement between methods, one needs to know if the source of the disagreement was due to a systematic shift (bias) or random error. The coefficients of accuracy and precision will be discussed to characterize these sources. This is important because a systematic shift usually can be easily fixed through calibration, while a random error usually is a more cumbersome variation reduction exercise. For categorical variables, we consider scaled agreement statistics. For continuous variables, we use scaled or unscaled agreement statistics. For variables with proportional error, we can simply apply a log transformation to the data. Finally, three examples are given: one for assay validation, one for a lab proficiency assessment, and one for a lab comparison on categorical assay.

Overview of agreement statistics for medical devices

This article proposes a general comparison model for assessing individual agreement of k  ≥  2 raters evaluating n subjects with m  ≥  2 replicated readings. Users can explore total-rater agreement relative to intrarater agreement where any subset of the k raters can be selected in the numerator and denominator. Users are also allowed to compare intrarater agreement among selected raters. Based on the ratio of mean squared deviations (MSDs), two comparative agreement indices, total–intra ratio (TIR) and intra–intra ratio (IIR), are proposed. The TIR is a noninferiority assessment such that the differences of individual readings from different raters cannot be inferior by a prespecified margin to the differences of the replicated readings within raters. TIR can be used whether a reference exists or not. The method used by the Food and Drug Administration (FDA) for evaluating individual bioequivalence under relative scale becomes the special case of our approach. The IIR is a classical assessment such that ...

Lawrence Lin

Papers

Statistical methods in assessing agreement: Models, issues, and tools

Statistical Tools for Measuring Agreement

A Unified Approach for Assessing Agreement for Continuous and Categorical Data

Overview of agreement statistics for medical devices

A comparison model for measuring individual agreement.