Principles Underlying the Use of Multiple Informants' Reports
TL;DR: In this review, the authors advance a framework (Operations Triad Model) outlining general principles for using and interpreting informants' reports and provide supportive evidence for this framework and discuss its implications for hypothesis testing, study design, and quantitative review.
Abstract: Researchers use multiple informants' reports to assess and examine behavior. However, informants' reports commonly disagree. Informants' reports often disagree in their perceived levels of a behavior (“low” versus “elevated” mood), and examining multiple reports in a single study often results in inconsistent findings. Although researchers often espouse taking a multi-informant assessment approach, they frequently address informant discrepancies using techniques that treat discrepancies as measurement error. Yet, recent work indicates that researchers in a variety of fields often may be unable to justify treating informant discrepancies as measurement error. In this review, the authors advance a framework (Operations Triad Model) outlining general principles for using and interpreting informants' reports. Using the framework, researchers can test whether or not they can extract meaningful information about behavior from discrepancies among multiple informants' reports. The authors provide supportive evide...
Citations
More filters
TL;DR: A modified version of a 5-item measure of adult psychosocial impairments (i.e., Work and Social Adjustment Scale for Youth [WSASY], which opens doors to new areas of inquiry regarding the developmental psychopathology of impairment, including questions regarding the onset of impairments and their links to mental health.
Abstract: A key component of delivering mental health services involves evaluating psychosocial impairments linked to mental health concerns. Youth may experience these impairments in various ways (e.g., dysfunctional family and/or peer relationships, poor school performance). Importantly, youth may display symptoms of mental illness without co-occurring psychosocial impairments, and the reverse may be true. However, all available instruments for assessing youth psychosocial impairments presume the presence of mental health concerns among those assessed. Consequently, key gaps exist in knowledge about the developmental psychopathology of psychosocial impairments; and thus how to understand impairments in the context of youth mental health. To address these issues we developed a modified version of a 5-item measure of adult psychosocial impairments (i.e., Work and Social Adjustment Scale for Youth [WSASY]) and tested its psychometric properties. A mixed clinical/community sample of adolescents and parents completed parallel versions of the WSASY, along with a multi-domain, multi-method battery of measures of adolescent internalizing and externalizing concerns, parent psychosocial functioning, adolescent-parent conflict, adolescent peer functioning, and observed social skills. On both versions of the WSASY, increased scores related to increased adolescent mental health concerns, adolescent–parent conflict, parent psychosocial dysfunction, and peer-related impairments. WSASY scores also distinguished adolescents who displayed co-occurring mental health concerns from those who did not, and related to observed social skills deficits within social interactions with unfamiliar peers. The WSASY opens doors to new areas of inquiry regarding the developmental psychopathology of impairment, including questions regarding the onset of impairments and their links to mental health.
26 citations
TL;DR: In this article, school systems are the primary providers for the increasing number of children with mental health needs and school-based universal screening offers a valuable way to identify children that would benefit from mental health care.
Abstract: School systems are the primary providers for the increasing number of children with mental health needs. School-based universal screening offers a valuable way to identify children that would benef...
26 citations
Cites methods from "Principles Underlying the Use of Mu..."
...For example, overall behavioral risk data could be compared from the perspective of both the student and teacher thereby potentially identifying students previously missed by teacher report screeners (De Los Reyes et al., 2015)....
[...]
...Although possessing psychometric support for applied use (Lane et al., 2009), the Student Risk Screening Scale (Drummond, 1994) is limited to teacher report and does not take into consideration student reported risk; these data are especially important for accurate identification of risk in later grades (e.g., high school; De Los Reyes et al., 2015)....
[...]
...Yet, given the time-intensive nature of data collection, analysis, and subsequent decision making required of multiinformant ratings, their use has typically been limited to more targeted assessment practices conducted across a smaller number of individuals (e.g., diagnosis and classification; De Los Reyes et al., 2013)....
[...]
...…nature of data collection, analysis, and subsequent decision making required of multiinformant ratings, their use has typically been limited to more targeted assessment practices conducted across a smaller number of individuals (e.g., diagnosis and classification; De Los Reyes et al., 2013)....
[...]
...However, to promote efficiency and effectiveness in MTSS decision making, additional SAEBRS rater forms are necessary to facilitate multi-informant decision making (De Los Reyes, Thomas, Goodman, & Kundey, 2013)....
[...]
TL;DR: While latent scores relate to either observed anxiety measure, parent and child report on the SCARED each provide valuable information that differentially relate to naturalistic social anxiety-related behaviors.
Abstract: Social anxiety typically emerges by adolescence and is one of the most common anxiety disorders. Many clinicians and researchers utilize the Screen for Child Anxiety Related Disorders (SCARED) to quantify anxiety symptoms, including social anxiety, throughout childhood and adolescence. The SCARED can be administered to both children and their parents, though reports from each informant tend to only moderately correlate. Here, we investigated parent–child concordance on the SCARED in a sample of adolescents (N = 360, Mage = 13.2) using a multi-trait multi-method (MTMM) model. Next, in a selected sample of the adolescents, we explored relations among child report, parent report, and latent social anxiety scores with two laboratory tasks known to elicit signs of social anxiety in the presence of unfamiliar peers: a speech task and a “Get to Know You” task. Findings reveal differences in variance of the SCARED accounted for by parent and child report. Parent report of social anxiety is a better predictor of anxiety signs elicited by a structured speech task, whereas child report of social anxiety is a better predictor of anxiety signs during the naturalistic conversation with unfamiliar peers. Moreover, while latent social anxiety scores predict both observed anxiety measures, parent report more closely resembles latent scores in relation to the speech task, whereas child report functions more similarly to latent scores in relation to the peer conversation. Thus, while latent scores relate to either observed anxiety measure, parent and child report on the SCARED each provide valuable information that differentially relate to naturalistic social anxiety-related behaviors.
26 citations
TL;DR: Child perceived anxiety control significantly moderated informant agreement, such that pairs with children who had high levels of perceived control of their anxiety had lower PTSD symptom agreement where children reported lower symptoms relative to their parents.
Abstract: The goal of this study is to examine parent and child agreement of child posttraumatic stress disorder (PTSD) symptoms pre- and posttreatment, as well as potential moderators of agreement including treatment responder status, child anxiety control, and parent self-reported PTSD symptoms. We examined child self-reported and parent-reported child PTSD symptoms from the Diagnostic Interview Schedule for Children. Of the 141 parent-child pairs, the mean age of children was 12.72 (SD = 3.40), 53% were female, and 54% were Black. A subsample of participants (n = 47) was assessed after completion of a cognitive behavioral therapy treatment for PTSD. Moderate levels of agreement were found at baseline, though Criterion D (increased arousal) symptoms had lower levels of agreement than the other symptom clusters. Symptom agreement was lower at posttreatment. Treatment responders had higher levels of baseline informant agreement than treatment nonresponders. Child perceived anxiety control significantly moderated informant agreement, such that pairs with children who had high levels of perceived control of their anxiety had lower PTSD symptom agreement where children reported lower symptoms relative to their parents. Contrary to expectations, parent self-reported PTSD did not moderate parent-child symptom agreement. Factors associated with higher parent-child agreement of child PTSD symptoms were being a PTSD treatment responder and children with lower perceived anxiety control. These findings have potential implications for determining those who may benefit from greater symptom monitoring over the course of intervention and potential alternative intervention approaches.
26 citations
TL;DR: It is demonstrated that trait agreement can be partialled into individual contributions so that it is possible to show how much each individual pair of judges contributes to agreement on a particular trait.
Abstract: It is argued that if we compute self-other agreement on some personality traits then we possess no or very little information about the individuals who are the targets of this judgment. This idea is largely based on two separate ways of computing self-other agreement: trait agreement (rT) and profile agreement (rP), which are typically associated with two different trait-centered and person-centered approaches in personality research. Personality traits of 4,115 targets from Czech, Belgian, Estonian, and German samples were rated by themselves and knowledgeable informants. We demonstrate that trait agreement can be partialled into individual contributions so that it is possible to show how much each individual pair of judges contributes to agreement on a particular trait. Similarly, it is possible to decompose agreement between two personality profiles into the individual contributions of traits from which these profiles are assembled. If normativeness is separated from distinctiveness of personality scores and individual profiles are ipsatized, then mean profile agreement r _P becomes identical to mean trait agreement r _T. The views that trait-by-trait analysis does not provide information regarding accuracy level of a particular pair of judges and profile analysis does not permit assessment of the relative contributions of traits to overall accuracy are not supported.
25 citations
Cites methods from "Principles Underlying the Use of Mu..."
...…to separate some of the different components of ratings, the use of multiple informants has become one of the most valuable tools in personality research (McCrae, 1994; Funder, 1999; Kenny et al., 2006; Vazire, 2006; Borkenau and Zaltauskas, 2009; Kandler et al., 2010; De Los Reyes et al., 2013)....
[...]
References
More filters
11 Jun 2013
113,134 citations
Book•
01 Jan 1983
TL;DR: In this Section: 1. Multivariate Statistics: Why? and 2. A Guide to Statistical Techniques: Using the Book Research Questions and Associated Techniques.
Abstract: In this Section: 1. Brief Table of Contents 2. Full Table of Contents 1. BRIEF TABLE OF CONTENTS Chapter 1 Introduction Chapter 2 A Guide to Statistical Techniques: Using the Book Chapter 3 Review of Univariate and Bivariate Statistics Chapter 4 Cleaning Up Your Act: Screening Data Prior to Analysis Chapter 5 Multiple Regression Chapter 6 Analysis of Covariance Chapter 7 Multivariate Analysis of Variance and Covariance Chapter 8 Profile Analysis: The Multivariate Approach to Repeated Measures Chapter 9 Discriminant Analysis Chapter 10 Logistic Regression Chapter 11 Survival/Failure Analysis Chapter 12 Canonical Correlation Chapter 13 Principal Components and Factor Analysis Chapter 14 Structural Equation Modeling Chapter 15 Multilevel Linear Modeling Chapter 16 Multiway Frequency Analysis 2. FULL TABLE OF CONTENTS Chapter 1: Introduction Multivariate Statistics: Why? Some Useful Definitions Linear Combinations of Variables Number and Nature of Variables to Include Statistical Power Data Appropriate for Multivariate Statistics Organization of the Book Chapter 2: A Guide to Statistical Techniques: Using the Book Research Questions and Associated Techniques Some Further Comparisons A Decision Tree Technique Chapters Preliminary Check of the Data Chapter 3: Review of Univariate and Bivariate Statistics Hypothesis Testing Analysis of Variance Parameter Estimation Effect Size Bivariate Statistics: Correlation and Regression. Chi-Square Analysis Chapter 4: Cleaning Up Your Act: Screening Data Prior to Analysis Important Issues in Data Screening Complete Examples of Data Screening Chapter 5: Multiple Regression General Purpose and Description Kinds of Research Questions Limitations to Regression Analyses Fundamental Equations for Multiple Regression Major Types of Multiple Regression Some Important Issues. Complete Examples of Regression Analysis Comparison of Programs Chapter 6: Analysis of Covariance General Purpose and Description Kinds of Research Questions Limitations to Analysis of Covariance Fundamental Equations for Analysis of Covariance Some Important Issues Complete Example of Analysis of Covariance Comparison of Programs Chapter 7: Multivariate Analysis of Variance and Covariance General Purpose and Description Kinds of Research Questions Limitations to Multivariate Analysis of Variance and Covariance Fundamental Equations for Multivariate Analysis of Variance and Covariance Some Important Issues Complete Examples of Multivariate Analysis of Variance and Covariance Comparison of Programs Chapter 8: Profile Analysis: The Multivariate Approach to Repeated Measures General Purpose and Description Kinds of Research Questions Limitations to Profile Analysis Fundamental Equations for Profile Analysis Some Important Issues Complete Examples of Profile Analysis Comparison of Programs Chapter 9: Discriminant Analysis General Purpose and Description Kinds of Research Questions Limitations to Discriminant Analysis Fundamental Equations for Discriminant Analysis Types of Discriminant Analysis Some Important Issues Comparison of Programs Chapter 10: Logistic Regression General Purpose and Description Kinds of Research Questions Limitations to Logistic Regression Analysis Fundamental Equations for Logistic Regression Types of Logistic Regression Some Important Issues Complete Examples of Logistic Regression Comparison of Programs Chapter 11: Survival/Failure Analysis General Purpose and Description Kinds of Research Questions Limitations to Survival Analysis Fundamental Equations for Survival Analysis Types of Survival Analysis Some Important Issues Complete Example of Survival Analysis Comparison of Programs Chapter 12: Canonical Correlation General Purpose and Description Kinds of Research Questions Limitations Fundamental Equations for Canonical Correlation Some Important Issues Complete Example of Canonical Correlation Comparison of Programs Chapter 13: Principal Components and Factor Analysis General Purpose and Description Kinds of Research Questions Limitations Fundamental Equations for Factor Analysis Major Types of Factor Analysis Some Important Issues Complete Example of FA Comparison of Programs Chapter 14: Structural Equation Modeling General Purpose and Description Kinds of Research Questions Limitations to Structural Equation Modeling Fundamental Equations for Structural Equations Modeling Some Important Issues Complete Examples of Structural Equation Modeling Analysis. Comparison of Programs Chapter 15: Multilevel Linear Modeling General Purpose and Description Kinds of Research Questions Limitations to Multilevel Linear Modeling Fundamental Equations Types of MLM Some Important Issues Complete Example of MLM Comparison of Programs Chapter 16: Multiway Frequency Analysis General Purpose and Description Kinds of Research Questions Limitations to Multiway Frequency Analysis Fundamental Equations for Multiway Frequency Analysis Some Important Issues Complete Example of Multiway Frequency Analysis Comparison of Programs
53,113 citations
TL;DR: This transmutability of the validation matrix argues for the comparisons within the heteromethod block as the most generally relevant validation data, and illustrates the potential interchangeability of trait and method components.
Abstract: Content Memory (Learning Ability) As Comprehension 82 Vocabulary Cs .30 ( ) .23 .31 ( ) .31 .31 .35 ( ) .29 .48 .35 .38 ( ) .30 .40 .47 .58 .48 ( ) As judged against these latter values, comprehension (.48) and vocabulary (.47), but not memory (.31), show some specific validity. This transmutability of the validation matrix argues for the comparisons within the heteromethod block as the most generally relevant validation data, and illustrates the potential interchangeability of trait and method components. Some of the correlations in Chi's (1937) prodigious study of halo effect in ratings are appropriate to a multitrait-multimethod matrix in which each rater might be regarded as representing a different method. While the published report does not make these available in detail because it employs averaged values, it is apparent from a comparison of his Tables IV and VIII that the ratings generally failed to meet the requirement that ratings of the same trait by different raters should correlate higher than ratings of different traits by the same rater. Validity is shown to the extent that of the correlations in the heteromethod block, those in the validity diagonal are higher than the average heteromethod-heterotrait values. A conspicuously unsuccessful multitrait-multimethod matrix is provided by Campbell (1953, 1956) for rating of the leadership behavior of officers by themselves and by their subordinates. Only one of 11 variables (Recognition Behavior) met the requirement of providing a validity diagonal value higher than any of the heterotrait-heteromethod values, that validity being .29. For none of the variables were the validities higher than heterotrait-monomethod values. A study of attitudes toward authority and nonauthority figures by Burwen and Campbell (1957) contains a complex multitrait-multimethod matrix, one symmetrical excerpt from which is shown in Table 6. Method variance was strong for most of the procedures in this study. Where validity was found, it was primarily at the level of validity diagonal values higher than heterotrait-heteromethod values. As illustrated in Table 6, attitude toward father showed this kind of validity, as did attitude toward peers to a lesser degree. Attitude toward boss showed no validity. There was no evidence of a generalized attitude toward authority which would include father and boss, although such values as the VALIDATION BY THE MULTITRAIT-MULTIMETHOD MATRIX
15,795 citations
Book•
01 Jan 1953TL;DR: The psychology classic "Walden Two" as mentioned in this paper is a detailed study of scientific theories of human nature and the possible ways in which human behavior can be predicted and controlled from one of the most influential behaviorists of the twentieth century.
Abstract: The psychology classic-a detailed study of scientific theories of human nature and the possible ways in which human behavior can be predicted and controlled-from one of the most influential behaviorists of the twentieth century and the author of Walden Two. "This is an important book, exceptionally well written, and logically consistent with the basic premise of the unitary nature of science. Many students of society and culture would take violent issue with most of the things that Skinner has to say, but even those who disagree most will find this a stimulating book." -Samuel M. Strong, The American Journal of Sociology "This is a remarkable book-remarkable in that it presents a strong, consistent, and all but exhaustive case for a natural science of human behavior...It ought to be...valuable for those whose preferences lie with, as well as those whose preferences stand against, a behavioristic approach to human activity." -Harry Prosch, Ethics
8,325 citations
"Principles Underlying the Use of Mu..." refers background in this paper
...…which a patient lives may vary as a function of the contingencies (e.g., corporal punishment and praise) that reinforce expressions of her behavior (e.g., Skinner 1953), and as mentioned previously, multiple informants may vary in where they observe a patient’s behavior (e.g., De Los Reyes 2011)....
[...]
TL;DR: Etude de la coherence entre differentes sources (269 echantillons utilisees dans 119 etudes) concernant les evaluations des problemes affectifs et comportementaux d'enfants et d'adolescents âges de 1 1/2 a 19 ans.
Abstract: Etude de la coherence entre differentes sources (269 echantillons utilisees dans 119 etudes) concernant les evaluations des problemes affectifs et comportementaux d'enfants et d'adolescents âges de 1 1/2 a 19 ans
5,254 citations
"Principles Underlying the Use of Mu..." refers background in this paper
...Third, as mentioned previously, low multi-informant correspondence is the norm rather than the exception in assessments of child, adolescent, and adult psychopathology (Achenbach et al. 1987, 2005)....
[...]
...Along these lines, perhaps the investigative team would draw from theory suggesting that parents and teachers provide discrepant reports because they observe children in different contexts or settings (parents at home versus teachers at school; Achenbach et al. 1987, Kraemer et al. 2003)....
[...]
...Achenbach et al. (1987) lucidly reflected this interpretation, in a meta-analysis of correspondence between informants’ reports of child mental health that was published nearly a century after Edgeworth’s (1888) article....
[...]
...However, although multiple informants’ reports typically correspond no higher than low-to-moderate in magnitude (e.g., r’s ranging from 0.20 to 0.40; see Achenbach et al. 1987, 2005), they nonetheless often correspond with each other at statistically significant magnitudes....
[...]
...(e.g., Achenbach et al. 1987, 2005)....
[...]