Showing papers in "Psychometrika in 2022"

Possible Futures for Network Psychometrics

Abstract: This commentary reflects on the articles included in the Psychometrika Special Issue on Network Psychometrics in Action. The contributions to the special issue are related to several possible future paths for research in this area. These include the development of models to analyze and represent interventions, improvement in exploratory and inferential techniques in network psychometrics, the articulation of psychometric theories in addition to psychometric models, and extensions of network modeling to novel data sources. Finally, network psychometrics is part of a larger movement in psychology that revolves around the analysis of human beings as complex systems, and it is timely that psychometricians start extending their rich modeling tradition to improve and extend the analysis of systems in psychology.

Modeling Eye Movements During Decision Making: A Review

Abstract: This article reviews recent advances in the psychometric and econometric modeling of eye-movements during decision making. Eye movements offer a unique window on unobserved perceptual, cognitive, and evaluative processes of people who are engaged in decision making tasks. They provide new insights into these processes, which are not easily available otherwise, allow for explanations of fundamental search and choice phenomena, and enable predictions of future decisions. We propose a theoretical framework of the search and choice tasks that people commonly engage in and of the underlying cognitive processes involved in those tasks. We discuss how these processes drive specific eye-movement patterns. Our framework emphasizes the central role of task and strategy switching for complex goal attainment. We place the extant literature within that framework, highlight recent advances in modeling eye-movement behaviors during search and choice, discuss limitations, challenges, and open problems. An agenda for further psychometric modeling of eye movements during decision making concludes the review.

Modeling Conditional Dependence of Response Accuracy and Response Time with the Diffusion Item Response Theory Model

Abstract: In this paper, we propose a model-based method to study conditional dependence between response accuracy and response time (RT) with the diffusion IRT model (Tuerlinckx and De Boeck in Psychometrika 70(4):629–650, 2005, https://doi.org/10.1007/s11336-000-0810-3 ; van der Maas et al. in Psychol Rev 118(2):339–356, 2011, https://doi.org/10.1080/20445911.2011.454498 ). We extend the earlier diffusion IRT model by introducing variability across persons and items in cognitive capacity (drift rate in the evidence accumulation process) and variability in the starting point of the decision processes. We show that the extended model can explain the behavioral patterns of conditional dependency found in the previous studies in psychometrics. Variability in cognitive capacity can predict positive and negative conditional dependency and their interaction with the item difficulty. Variability in starting point can account for the early changes in the response accuracy as a function of RT given the person and item effects. By the combination of the two variability components, the extended model can produce the curvilinear conditional accuracy functions that have been observed in psychometric data. We also provide a simulation study to validate the parameter recovery of the proposed model and present two empirical applications to show how to implement the model to study conditional dependency underlying data response accuracy and RTs.

Possible Futures for Network Psychometrics

Abstract: This commentary reflects on the articles included in the Psychometrika Special Issue on Network Psychometrics in Action. The contributions to the special issue are related to several possible future paths for research in this area. These include the development of models to analyze and represent interventions, improvement in exploratory and inferential techniques in network psychometrics, the articulation of psychometric theories in addition to psychometric models, and extensions of network modeling to novel data sources. Finally, network psychometrics is part of a larger movement in psychology that revolves around the analysis of human beings as complex systems, and it is timely that psychometricians start extending their rich modeling tradition to improve and extend the analysis of systems in psychology.

On the Information Obtainable from Comparative Judgments

Abstract: Abstract Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people’s trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.

Sparse and Simple Structure Estimation via Prenet Penalization.

Abstract: We propose a prenet (product-based elastic net), a novel penalization method for factor analysis models. The penalty is based on the product of a pair of elements in each row of the loading matrix. The prenet not only shrinks some of the factor loadings toward exactly zero but also enhances the simplicity of the loading matrix, which plays an important role in the interpretation of the common factors. In particular, with a large amount of prenet penalization, the estimated loading matrix possesses a perfect simple structure, which is known as a desirable structure in terms of the simplicity of the loading matrix. Furthermore, the perfect simple structure estimation via the proposed penalization turns out to be a generalization of the k-means clustering of variables. On the other hand, a mild amount of the penalization approximates a loading matrix estimated by the quartimin rotation, one of the most commonly used oblique rotation techniques. Simulation studies compare the performance of our proposed penalization with that of existing methods under a variety of settings. The usefulness of the perfect simple structure estimation via our proposed procedure is presented through various real data applications.

DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method

Abstract: Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents’ problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents’ problem-solving competence. The proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.

Bridging Parametric and Nonparametric Methods in Cognitive Diagnosis

Abstract: A number of parametric and nonparametric methods for estimating cognitive diagnosis models (CDMs) have been developed and applied in a wide range of contexts. However, in the literature, a wide chasm exists between these two families of methods, and their relationship to each other is not well understood. In this paper, we propose a unified estimation framework to bridge the divide between parametric and nonparametric methods in cognitive diagnosis to better understand their relationship. We also develop iterative joint estimation algorithms and establish consistency properties within the proposed framework. Lastly, we present comprehensive simulation results to compare different methods and provide practical recommendations on the appropriate use of the proposed framework in various CDM contexts.

Computation and application of generalized linear mixed model derivatives using lme4

Abstract: Maximum likelihood estimation of generalized linear mixed models (GLMMs) is difficult due to marginalization of the random effects. Derivative computations of a fitted GLMM’s likelihood are also difficult, especially because the derivatives are not by-products of popular estimation algorithms. In this paper, we first describe theoretical results related to GLMM derivatives along with a quadrature method to efficiently compute the derivatives, focusing on fitted lme4 models with a single clustering variable. We describe how psychometric results related to item response models are helpful for obtaining the derivatives, as well as for verifying the derivatives’ accuracies. We then provide a tutorial on the many possible uses of these derivatives, including robust standard errors, score tests of fixed effect parameters, and likelihood ratio tests of non-nested models. The derivative computation methods and applications described in the paper are all available in easily obtained R packages.

Objective Bayesian Edge Screening and Structure Selection for Ising Networks

Abstract: Abstract The Ising model is one of the most widely analyzed graphical models in network psychometrics. However, popular approaches to parameter estimation and structure selection for the Ising model cannot naturally express uncertainty about the estimated parameters or selected structures. To address this issue, this paper offers an objective Bayesian approach to parameter estimation and structure selection for the Ising model. Our methods build on a continuous spike-and-slab approach. We show that our methods consistently select the correct structure and provide a new objective method to set the spike-and-slab hyperparameters. To circumvent the exploration of the complete structure space, which is too large in practical situations, we propose a novel approach that first screens for promising edges and then only explore the space instantiated by these edges. We apply our proposed methods to estimate the network of depression and alcohol use disorder symptoms from symptom scores of over 26,000 subjects.

Computation for Latent Variable Model Estimation: A Unified Stochastic Proximal Framework

Abstract: Abstract Latent variable models have been playing a central role in psychometrics and related fields. In many modern applications, the inference based on latent variable models involves one or several of the following features: (1) the presence of many latent variables, (2) the observed and latent variables being continuous, discrete, or a combination of both, (3) constraints on parameters, and (4) penalties on parameters to impose model parsimony. The estimation often involves maximizing an objective function based on a marginal likelihood/pseudo-likelihood, possibly with constraints and/or penalties on parameters. Solving this optimization problem is highly non-trivial, due to the complexities brought by the features mentioned above. Although several efficient algorithms have been proposed, there lacks a unified computational framework that takes all these features into account. In this paper, we fill the gap. Specifically, we provide a unified formulation for the optimization problem and then propose a quasi-Newton stochastic proximal algorithm. Theoretical properties of the proposed algorithms are established. The computational efficiency and robustness are shown by simulation studies under various settings for latent variable model estimation.

The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment

Abstract: Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a "Reliability Factor" that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

Erratum to: Meta-analytic Gaussian Network Aggregation

Abstract: A correction to this paper has been published: 10.1007/s11336-021-09764-3

A Latent Variable Mixed-Effects Location Scale Model with an Application to Daily Diary Data

Abstract: A mixed-effects location scale model allows researchers to study within- and between-person variation in repeated measures. Key components of the model include separate variance models to study predictors of the within-person variance, as well as predictors of the between-person variance of a random effect, such as a random intercept. In this paper, a latent variable mixed-effects location scale model is developed that combines a longitudinal common factor model and a mixed-effects location scale model to characterize within- and between-person variation in a common factor. The model is illustrated using daily reports of positive affect and daily stressors for a large sample of adult women.

Accurate Assessment via Process Data

Abstract: Accurate assessment of a student's ability is the key task of a test. Assessments based on final responses are the standard. As the infrastructure advances, substantially more information is observed. One of such instances is the process data that is collected by computer-based interactive items and contain a student's detailed interactive processes. In this paper, we show both theoretically and with simulated and empirical data that appropriately including such information in the assessment will substantially improve relevant assessment precision.

Erratum to: Meta-analytic Gaussian Network Aggregation

Abstract: A correction to this paper has been published: 10.1007/s11336-021-09764-3

Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models

Abstract: Cognitive Diagnosis Models (CDMs) are a special family of discrete latent variable models that are widely used in educational and psychological measurement. A key component of CDMs is the Q-matrix characterizing the dependence structure between the items and the latent attributes. Additionally, researchers also assume in many applications certain hierarchical structures among the latent attributes to characterize their dependence. In most CDM applications, the attribute–attribute hierarchical structures, the item-attribute Q-matrix, the item-level diagnostic models, as well as the number of latent attributes, need to be fully or partially pre-specified, which however may be subjective and misspecified as noted by many recent studies. This paper considers the problem of jointly learning these latent and hierarchical structures in CDMs from observed data with minimal model assumptions. Specifically, a penalized likelihood approach is proposed to select the number of attributes and estimate the latent and hierarchical structures simultaneously. An expectation-maximization (EM) algorithm is developed for efficient computation, and statistical consistency theory is also established under mild conditions. The good performance of the proposed method is illustrated by simulation studies and real data applications in educational assessment.

Bi-factor and Second-Order Copula Models for Item Response Data

Abstract: Bi-factor and second-order models based on copulas are proposed for item response data, where the items are sampled from identified subdomains of some larger domain such that there is a homogeneous dependence within each domain. Our general models include the Gaussian bi-factor and second-order models as special cases and can lead to more probability in the joint upper or lower tail compared with the Gaussian bi-factor and second-order models. Details on maximum likelihood estimation of parameters for the bi-factor and second-order copula models are given, as well as model selection and goodness-of-fit techniques. Our general methodology is demonstrated with an extensive simulation study and illustrated for the Toronto Alexithymia Scale. Our studies suggest that there can be a substantial improvement over the Gaussian bi-factor and second-order models both conceptually, as the items can have interpretations of discretized maxima/minima or mixtures of discretized means in comparison with discretized means, and in fit to data.

Bayesian Dynamic Borrowing of Historical Information with Applications to the Analysis of Large-Scale Assessments

Abstract: The purpose of this paper is to demonstrate and evaluate the use of Bayesian dynamic borrowing (Viele et al, in Pharm Stat 13:41-54, 2014) as a means of systematically utilizing historical information with specific applications to large-scale educational assessments. Dynamic borrowing via Bayesian hierarchical models is a special case of a general framework of historical borrowing where the degree of borrowing depends on the heterogeneity among historical data and current data. A joint prior distribution over the historical and current data sets is specified with the degree of heterogeneity across the data sets controlled by the variance of the joint distribution. We apply Bayesian dynamic borrowing to both single-level and multilevel models and compare this approach to other historical borrowing methods such as complete pooling, Bayesian synthesis, and power priors. Two case studies using data from the Program for International Student Assessment reveal the utility of Bayesian dynamic borrowing in terms of predictive accuracy. This is followed by two simulation studies that reveal the utility of Bayesian dynamic borrowing over simple pooling and power priors in cases where the historical data is heterogeneous compared to the current data based on bias, mean squared error, and predictive accuracy. In cases of homogeneous historical data, Bayesian dynamic borrowing performs similarly to data pooling, Bayesian synthesis, and power priors. In contrast, for heterogeneous historical data, Bayesian dynamic borrowing performed at least as well, if not better, than other methods of borrowing with respect to mean squared error, percent bias, and leave-one-out cross-validation.

Penalized Estimation and Forecasting of Multiple Subject Intensive Longitudinal Data

Abstract: Intensive longitudinal data (ILD) is an increasingly common data type in the social and behavioral sciences. Despite the many benefits these data provide, little work has been dedicated to realize the potential such data hold for forecasting dynamic processes at the individual level. To address this gap in the literature, we present the multi-VAR framework, a novel methodological approach allowing for penalized estimation of ILD collected from multiple individuals. Importantly, our approach estimates models for all individuals simultaneously and is capable of adaptively adjusting to the amount of heterogeneity present across individual dynamic processes. To accomplish this, we propose a novel proximal gradient descent algorithm for solving the multi-VAR problem and prove the consistency of the recovered transition matrices. We evaluate the forecasting performance of our method in comparison with a number of benchmark methods and provide an illustrative example involving the day-to-day emotional experiences of 16 individuals over an 11-week period.

Item Response Thresholds Models: A General Class of Models for Varying Types of Items

Abstract: A comprehensive class of models is proposed that can be used for continuous, binary, ordered categorical and count type responses. The difficulty of items is described by difficulty functions, which replace the item difficulty parameters that are typically used in item response models. They crucially determine the response distribution and make the models very flexible with regard to the range of distributions that are covered. The model class contains several widely used models as the binary Rasch model and the graded response model as special cases, allows for simplifications, and offers a distribution free alternative to count type items. A major strength of the models is that they can be used for mixed item formats, when different types of items are combined to measure abilities or attitudes. It is an immediate consequence of the comprehensive modeling approach that allows that difficulty functions automatically adapt to the response distribution. Basic properties of the model class are shown. Several real data sets are used to illustrate the flexibility of the models.

Bayesian Forecasting with a Regime-Switching Zero-Inflated Multilevel Poisson Regression Model: An Application to Adolescent Alcohol Use with Spatial Covariates

Abstract: In this paper, we present and evaluate a novel Bayesian regime-switching zero-inflated multilevel Poisson (RS-ZIMLP) regression model for forecasting alcohol use dynamics. The model partitions individuals' data into two phases, known as regimes, with: (1) a zero-inflation regime that is used to accommodate high instances of zeros (non-drinking) and (2) a multilevel Poisson regression regime in which variations in individuals' log-transformed average rates of alcohol use are captured by means of an autoregressive process with exogenous predictors and a person-specific intercept. The times at which individuals are in each regime are unknown, but may be estimated from the data. We assume that the regime indicator follows a first-order Markov process as related to exogenous predictors of interest. The forecast performance of the proposed model was evaluated using a Monte Carlo simulation study and further demonstrated using substance use and spatial covariate data from the Colorado Online Twin Study (CoTwins). Results showed that the proposed model yielded better forecast performance compared to a baseline model which predicted all cases as non-drinking and a reduced ZIMLP model without the RS structure, as indicated by higher AUC (the area under the receiver operating characteristic (ROC) curve) scores, and lower mean absolute errors (MAEs) and root-mean-square errors (RMSEs). The improvements in forecast performance were even more pronounced when we limited the comparisons to participants who showed at least one instance of transition to drinking.