Quality criteria were proposed for measurement properties of health status questionnaires

TL;DR: The criteria can be used in systematic reviews of health status questionnaires, to detect shortcomings and gaps in knowledge of measurement properties, and to design validation studies.

Abstract: Objectives: Recently, an increasing number of systematic reviews have been published in which the measurement properties of health status questionnaires are compared. For a meaningful comparison, quality criteria for measurement properties are needed. Our aim was to develop quality criteria for design, methods, and outcomes of studies on the development and evaluation of health status questionnaires. Study Design and Setting: Quality criteria for content validity, internal consistency, criterion validity, construct validity, reproducibility, longitudinal validity, responsiveness, floor and ceiling effects, and interpretability were derived from existing guidelines and consensus within our research group. Results: For each measurement property a criterion was defined for a positive, negative, or indeterminate rating, depending on the design, methods, and outcomes of the validation study. Conclusion: Our criteria make a substantial contribution toward defining explicit quality criteria for measurement properties of health status questionnaires. Our criteria can be used in systematic reviews of health status questionnaires, to detect shortcomings and gaps in knowledge of measurement properties, and to design validation studies. The future challenge will be to refine and complete the criteria and to reach broad consensus, especially on quality criteria for good measurement properties. 2006 Elsevier Inc. All rights reserved.

Summary

1. Introduction

• The number of available health status questionnaires has increased dramatically over the past decades.
• Within each of these attributes, specific criteria were defined by which instruments should be reviewed.
• The authors criteria can also be used to detect shortcomings and gaps in knowledge of measurement properties, and to design validation studies.
• The authors aim is to contribute to the development of explicit quality criteria for the design, methods, and outcomes of studies on the development and evaluation of health status questionnaires.

2. Content validity

• Content validity examines the extent to which the concepts of interest are comprehensively represented by the items in the questionnaire [16].
• The relevance of items may also depend on disease severity.
• Items in the questionnaire must reflect areas that are important to the target population that is being studied.
• This strategy, however, does not guarantee a better content validity, because a comprehensive set of items can also be achieved without item reduction.
• The authors give a positive rating for content validity if a clear description is provided of the measurement aim, the target population, the concepts that are being measured, and the item selection.

3. Internal consistency

• Internal consistency is a measure of the extent to which items in a questionnaire (sub)scale are correlated , thus measuring the same concept.
• Internal consistency is an important measurement property for questionnaires that intend to measure a single underlying concept by using multiple items.
• In case that there is no prior hypothesis regarding the dimensionality of a questionnaire, exploratory principal component analysis or factor analyses can be applied.
• Rules-of-thumb vary from four to 10 subjects per variable, with a minimum number of 100 subjects to ensure stability of the varianceecovariance matrix [27].
• Furthermore, a very high Cronbach’s alpha is usually found for scales with a large number of items, because Cronbach’s alpha is dependent upon the number of items in a scale.

5. Construct validity

• Construct validity refers to the extent to which scores on a particular instrument relate to other measures in a manner that is consistent with theoretically derived hypotheses concerning the concepts that are being measured [17,19].
• Construct validity should be assessed by testing predefined hypotheses (e.g., about expected correlations between measures or expected differences in scores between ‘‘known’’ groups).
• Without specific hypotheses, the risk of bias is high because retrospectively it is tempting to think up alternative explanations for low correlations instead of concluding that the questionnaire may not be valid.

6. Reproducibility

• Reproducibility concerns the degree to which repeated measurements in stable persons provide similar answers.
• Reliability concerns the degree to which patients can be distinguished from each other, despite measurement error [19].
• Reliability coefficients (intraclass correlation coefficients (ICC)) concern the variation in the population (interindividual variation) divided by the total variation, which is the interindividual variation plus the intraindividual variation (measurement error), expressed as a ratio between 0 and 1.
• The time period between the repeated administrations should be long enough to prevent recall, though short enough to ensure that clinical change has not occurred.
• Therefore, the authors do not rate the appropriateness of the time period, but only require that this time period is described and justified.

6.1. Agreement

• The measurement error can be adequately expressed as the standard error of measurement (SEM) [30].
• The SEM equals the square root of the error variance of an ANOVA analysis, either including systematic differences or excluding them .
• Another adequate parameter of agreement is described by Bland and Altman [34].
• For evaluative purposes, the absolute measurement error should be smaller than the minimal amount of change in the (sub)scale that is considered to be important (minimal important change (MIC)).

6.2. Reliability

• The ICC is the most suitable and most commonly used reliability parameter for continuous measures.
• Because systematic differences are considered to be part of the measurement error, ICCagreement (two-way random effects model, or ICC (A,1) according to McGraw and Wong [36]) is preferred.
• Floor or ceiling effects are considered to be present if more than 15% of respondents achieved the lowest or highest possible score, respectively [41].
• Investigators should provide information about what (change in) score would be clinically meaningful.
• Various distribution-based and anchor-based methods have been proposed.

10. Population-specific ratings of measurement properties

• Each property is rated as positive, negative, or indeterminate, depending on the design, methods, and outcomes of the study.
• Measurement properties differ between populations and settings.
• The setting refers to the testing conditions, e.g., self-completed or interview, and language.
• A clear description of the design of each individual study has to be provided, including population (diagnosis and clinical features, age, and gender); design (e.g., language version, time between the measurements, completion before or after treatment); testing conditions (e.g., questionnaires completed at home or in a waiting room, self of in interview); and analyses of the data.
• In addition, if any important methodological weakness in the design or execution of the study is found, e.g., selection bias or an extremely heterogeneous study population, the evaluated measurement properties are also rated as indeterminate.

11. Overview table

• In the final comparison of the measurement properties of different questionnaires, one has to consider all ratings together when choosing between different questionnaires.
• In Table 2 the results are presented from their systematic review of all questionnaires measuring disability in patients with shoulder complaints (because there is no gold standard for disability, criterion validity was not assessed) [1].
• In Table 2 all ratings for each questionnaire are presented separately for each specific population or setting.
• Construct validity and responsiveness were rated positively for outpatients (c), but rated as indeterminate for primary care patients (b) and hospital patients (d).
• Taking into account those measurement properties that are most important for a specific application (e.g., reliability when using a questionnaire for discrimination and responsiveness when using it for evaluation of a treatment effect) and the population and setting in which the questionnaire is going to be used.

12. Discussion

• The authors developed quality criteria for the design, methods, and outcomes of studies on the development and evaluation of health status questionnaires.
• Firstly, with their approach, poor quality questionnaires can be given positive ratings for some measurement properties.
• Furthermore, poorly reported validation studies will lead to low ratings for questionnaires that are not necessarily poor in design or performance.
• Practically all authors of instrument evaluation studies conclude that their instrument is valid, whereas objective counting of the number of hypotheses that were confirmed frequently indicates otherwise [1,4].
• Different authors, however, have pointed to the conceptual and algebraic differences between different indices of responsiveness, showing that different indices may lead to different conclusions [53,54].

13. Future challenges

• One might argue that their criteria are not discriminative enough to distinguish between good and very high-quality questionnaires.
• When coefficient alpha does and doesn’t matter, also known as Being inconsistent about consistency.
• New York: Wiley; 1989. [26] de Vet HCW, Ader HJ, Terwee CB, Pouwer F. Evaluating quality of life and health status instruments: development of scientific review criteria.

Frequently asked questions

Q1. What are the criteria needed to legitimize what the questionnaire is?

Explicit quality criteria for studies on the development and evaluation of health status questionnaires are needed to legitimize what the best questionnaire is. 

Q2. How many patients are adequate for the assessment of the agreement parameter?

In both cases, the authors consider a sample size of at least 50 patients adequate for the assessment of the agreement parameter, based on a general guideline by Altman [35]. 

Q3. Why do the authors prefer the SEM to be considered part of the measurement error?

The authors believe that systematic differences should be considered part of the measurement error, because the authors want to distinguish them from ‘‘real’’ changes, e.g., due to treatment. 

Q4. Why is the number of health status questionnaires growing?

Because the number of health status questionnaires is rapidly growing, choosing the right questionnaire for a specific purpose becomes a time-consuming and difficult task. 

Q5. How many patients are in correspondence with the hypotheses?

The authors therefore give a positive rating for construct validity if hypotheses are specified in advance and at least 75% of the results are in correspondence with these hypotheses, in (sub)groups of at least 50 patients. 

Q6. What can be done to make an evidence-based choice for the questionnaire?

With this table one can make an evidencebased choice for the questionnaire with the best measurement properties, taking into account those measurement properties that are most important for a specific application (e.g., reliability when using a questionnaire for discrimination and responsiveness when using it for evaluation of a treatment effect) and the population and setting in which the questionnaire is going to be used. 

Q7. What is the way to measure internal consistency?

The authors give a positive rating for internal consistency when factor analysis was applied and Cronbach’s alpha is between 0.70 and 0.95. 

Q8. What is the way to measure agreement?

The authors give a positive rating for agreement if the SDC (SDCind for application in individuals and SDCgroup for use in groups) or the limits of agreement (upper or lower limit, depending on whether the interest is in improvement or deterioration) are smaller than the MIC.