Self-Reports in Organizational Research: Problems and Prospects

Measurement invariance is usually tested using Multigroup Confirmatory Factor Analysis, which examines the change in the goodness-of-fit index (GFI) when cross-group constraints are imposed on a measurement model. Although many studies have examined the properties of GFI as indicators of overall model fit for single-group data, there have been none to date that examine how GFIs change when between-group constraints are added to a measurement model. The lack of a consensus about what constitutes significant GFI differences places limits on measurement invariance testing. We examine 20 GFIs based on the minimum fit function. A simulation under the two-group situation was used to examine changes in the GFIs (ΔGFIs) when invariance constraints were added. Based on the results, we recommend using Δcomparative fit index, ΔGamma hat, and ΔMcDonald's Noncentrality Index to evaluate measurement invariance. These three ΔGFIs are independent of both model complexity and sample size, and are not correlated with the o...

Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance

Statistical procedures for missing data have vastly improved, yet misconception and unsound practice still abound. The authors frame the missing-data problem, review methods, offer advice, and raise issues that remain unresolved. They clear up common misunderstandings regarding the missing at random (MAR) concept. They summarize the evidence against older procedures and, with few exceptions, discourage their use. They present, in both technical and practical language, 2 general approaches that come highly recommended: maximum likelihood (ML) and Bayesian multiple imputation (MI). Newer developments are discussed, including some for dealing with missing data that are not MAR. Although not yet in the mainstream, these procedures may eventually extend the ML and MI methods that currently represent the state of the art.

/pdf/missing-data-our-view-of-the-state-of-the-art-3xnwc88wmd.pdf

Missing data: Our view of the state of the art.

Despite the concern that has been expressed about potential method biases, and the pervasiveness of research settings with the potential to produce them, there is disagreement about whether they really are a problem for researchers in the behavioral sciences. Therefore, the purpose of this review is to explore the current state of knowledge about method biases. First, we explore the meaning of the terms “method” and “method bias” and then we examine whether method biases influence all measures equally. Next, we review the evidence of the effects that method biases have on individual measures and on the covariation between different constructs. Following this, we evaluate the procedural and statistical remedies that have been used to control method biases and provide recommendations for minimizing method bias.

Sources of Method Bias in Social Science Research and Recommendations on How to Control It

Exploratory factor analysis (EFA) is a complex, multi-step process. The goal of this paper is to collect, in one article, information that will allow researchers and practitioners to understand the various choices available through popular software packages, and to make decisions about ”best practices” in exploratory factor analysis. In particular, this paper provides practical information on making decisions regarding (a) extraction, (b) rotation, (c) the number of factors to interpret, and (d) sample size.

/pdf/best-practices-in-exploratory-factor-analysis-four-3q64vgrpxp.pdf

Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis

In an effort to better understand possible pathways that lead to a relatively high incidence of depressive symptoms among Mexican American youth, an interpersonal stress model of depression was tested using a community sample of 674 Mexican American mothers and their 5th grade children. Structural equation analyses revealed that maternal depression was positively related to mothers’ interpersonal stress in their relationships. Additionally, maternal poor parenting quality was associated with three child outcomes: lower social competence, higher interpersonal stress, and higher depressive symptoms. Child difficulties with peers, however, were not related to child depression. Overall, evidence suggests that familial factors, rather than peer factors, were related to child depression for these study participants.

The Relation Between Maternal and Child Depression in Mexican American Families.

A common criticism of iterative least squares estimates of communality is that method of initial estimation may influence stabilized values. As little systematic research on this topic has been performed, the criticism appears to be based on cumulated experience with empirical data sets. In the present paper, two studies are reported in which four types of initial estimate (unities, squared multiple correlations, highestr, and zeroes) and four levels of convergence criterion were employed using four widely available computer packages (BMDP, SAS, SPSS, and SOUPAC). The results suggest that initial estimates have no effect on stabilized communality estimates when a stringent criterion for convergence is used, whereas initial estimates appear to affect stabilized values employing rather gross convergence criteria. There were no differences among the four computer packages for matrices without Heywood cases.

Iterative least squares estimates of communality: Initial estimate need not affect stabilized value

espanol. Antecedentes: a pesar de la importancia crucial del concepto de formas paralelas en la Teoria Clasica de los Tests, el grado de paralelismo entre dos formas paralelas no puede comprobarse directamente debido al caracter inobservable de las puntuaciones verdaderas. Nuestra propuesta pretende superar algunas de las limitaciones de los metodos tradicionales utilizando el esquema del Funcionamiento Diferencial del Item. Metodo: cambiamos el objeto de la comparacion de las puntuaciones totales a cada uno de los items individuales. Analizamos las puntuaciones de un unico grupo de participantes en items paralelos disenados para medir los mismos criterios comportamentales. Ejemplificamos la propuesta con las respuestas de 527 participantes a las dos formas paralelas de la “Attention Deficit-Hyperactivity Disorder Scale” (Caterino, Gomez-Benito, Balluerka, Amador-Campos, & Stock, 2009).Resultados: 12 de los 18 items (66,6%) muestran valores de probabilidad asociados con el estadistico Mantel χ2 menores de .01. El procedimiento de Estandarizacion muestra que la mitad de los items con DIF favorecen a la Forma A y la otra mitad a la Forma B. Conclusiones: el procedimiento “differential functioning of behavioral indicators” (DFBI) puede aportar informacion unica sobre el paralelismo entre parejas de items complementando el analisis tradicional de la equivalencia de formas paralelas. EnglishBackground: Despite the crucial importance of the notion of parallel forms within Classical Test Theory, the degree of parallelism between two forms of a test cannot be directly verified due to the unobservable nature of true scores. We intend to overcome some of the limitations of traditional approaches to analyzing parallelism by using the Differential Item Functioning framework. Method: We change the focus on comparison from total test scores to each of the items developed during test construction. We analyze the performance of a single group of individuals on parallel items designed to measure the same behavioral criterion by several DIF techniques. The proposed approach is illustrated with a dataset of 527 participants that responded to the two parallel forms of the Attention Deficit-Hyperactivity Disorder Scale (Caterino, Gomez-Benito, Balluerka, Amador-Campos, & Stock, 2009). Results: 12 of the 18 items (66.6%) show probability values associated with the Mantel χ2 statistic of less than .01. The standardization procedure shows that half of DIF items favoured Form A and the other half Form B. Conclusions: The “differential functioning of behavioral indicators” (DFBI) can provide unique information on parallelism between pairs of items to complement traditional analysis of equivalence between parallel test forms based on total scores.

/pdf/detecting-differential-item-functioning-in-behavioral-4kqbcz8p3l.pdf

Detecting differential item functioning in behavioral indicators across parallel forms

We introduce an approach for ensuring empirical identification of the correlated trait–correlated method (CT–CM) model under a variety of conditions. A set of models are referred to as augmented correlated trait–correlated method (ACT–CM) models because they are based on systematically augmenting the multitrait–multimethod matrix put forth by Campbell and Fiske (1959). We show results from a Monte Carlo simulation study in which data characteristics lead to an empirically underidentified standard CT–CM model, but a well-identified fully augmented correlated trait–correlated method (FACT–CM) model. This improved identification occurs even for a model in which equality constraints are imposed on loadings on each trait factor and loadings on each method factor—a specific case shown to lead to an empirically underidentified CT–CM model.

Keith F. Widaman

Papers

The Relation Between Maternal and Child Depression in Mexican American Families.

Iterative least squares estimates of communality: Initial estimate need not affect stabilized value

Detecting differential item functioning in behavioral indicators across parallel forms

Augmenting the Correlated Trait–Correlated Method Model for Multitrait–Multimethod Data

Qualitative transitions amid quantitative development: A challenge for measuring and representing change.