Discovering unobserved heterogeneity in structural equation models to avert validity threats

TL;DR: In this paper, the authors proposed a new method, Prediction-Oriented Segmentation (PLSPOS), to overcome the limitations of FIMIX-PLS and other distance measure-based methods.

Abstract: A large proportion of information systems research is concerned with developing and testing models pertaining to complex cognition, behaviors, and outcomes of individuals, teams, organizations, and other social systems that are involved in the development, implementation, and utilization of information technology. Given the complexity of these social and behavioral phenomena, heterogeneity is likely to exist in the samples used in IS studies. While researchers now routinely address observed heterogeneity by introducing moderators, a priori groupings, and contextual factors in their research models, they have not examined how unobserved heterogeneity may affect their findings. We describe why unobserved heterogeneity threatens different types of validity and use simulations to demonstrate that unobserved heterogeneity biases parameter estimates, thereby leading to Type I and Type II errors. We also review different methods that can be used to uncover unobserved heterogeneity in structural equation models. While methods to uncover unobserved heterogeneity in covariance-based structural equation models (CB-SEM) are relatively advanced, the methods for partial least squares (PLS) path models are limited and have relied on an extension of mixture regression--finite mixture partial least squares (FIMIX-PLS) and distance measure-based methods--that have mismatches with some characteristics of PLS path modeling. We propose a new method--prediction-oriented segmentation (PLSPOS)--to overcome the limitations of FIMIX-PLS and other distance measure-based methods and conduct extensive simulations to evaluate the ability of PLS-POS and FIMIX-PLS to discover unobserved heterogeneity in both structural and measurement models. Our results show that both PLS-POS and FIMIX-PLS perform well in discovering unobserved heterogeneity in structural paths when the measures are reflective and that PLS-POS also performs well in discovering unobserved heterogeneity in formative measures. We propose an unobserved heterogeneity discovery (UHD) process that researchers can apply to (1) avert validity threats by uncovering unobserved heterogeneity and (2) elaborate on theory by turning unobserved heterogeneity into observed heterogeneity, thereby expanding theory through the integration of new moderator or contextual variables.

Summary

Introduction

• Assuming that data in empirical studies are homogeneous and represent a single population is often unrealistic in the social and behavioral sciences, such as information systems, management, and marketing (Rust and Verhoef 2005; Wedel and Kamakura 2000).
• In contrast, inexperienced users show a strong positive relationship between PEOU and IU and a weak, or nonsignificant, relationship between PU and IU .
• (1) While IS studies now routinely address observed heterogeneity by introducing moderators, a priori groupings, contextual factors, and control variables in their research models, they have not considered unobserved heterogeneity in their data.
• Only recently has a method been proposed to detect unobserved heterogeneity in PLS path models: finite mixture partial least squares (FIMIX-PLS;.

Concept of Heterogeneity and its Treatment in IS Research

• Researchers can obtain different parameter estimates when they consider differences among observations relative to when they overlook them.
• Heterogeneity among observations is not necessarily captured by variables that are preconceived by the researcher and specified by existing theory, as it can exist beyond these previously identified variables (Jedidi et al. 1997).
• As a consequence, it is necessary to differentiate between the following two types of heterogeneity: (1) observed heterogeneity when subpopulations are defined a priori based on known variables and (2) unobserved heterogeneity when the subpopulations in the data are unknown (Lubke and Muthén 2005).

Treatment of Heterogeneity in IS Research

• Given the complexity of the social and behavioral phenomena tackled in IS research, heterogeneity is likely to exist in samples that are used to develop, test, and refine models.
• If this heterogeneity is not uncovered and controlled, the heterogeneity can bias results and conclusions (e.g., Ansari et al. 2000; Johns 2006).
• Unobserved heterogeneity is receiving increasing attention in related disciplines (e.g., marketing, where scholars study similar complex phenomena pertaining to consumer choices and preferences, the alignment of firm-level marketing strategies, interorganizational relationships, and the business value of tangible and intangible resources) to safeguard against biases and probe the underlying reasons for unobserved heterogeneity (e.g., Rigdon et al. 2010).
• This enhances the likelihood of obtaining valid results as well as of generating greater theoretical contributions.
• Methodologists in marketing, econometrics, and psychology have proposed advances to uncover unobserved heterogeneity in various approaches—for instance, regression analysis (DeSarbo and Cron 1988; Späth.

Unobserved Heterogeneity in the Structural Model

• In the context of SEM, heterogeneity can affect the structural model, the measurement model (formative and reflective), or both (e.g., Ansari et al. 2000; Qureshi and Compeau 2009).
• 2 (2) If a researcher overlooks unobserved heterogeneity and obtains a nonsignificant relationship between the constructs as the overall sample estimate, this researcher may overgeneralize the nonsignificant finding, which exists only in the second segment, thereby leading to a Type II error with respect to the first segment.
• The R² decreases in all situations, implying an 2This does not mean that there will be a Type I error in general (i.e., for both segments) but only with respect to segment 2 where the true effect is zero.

Unobserved Heterogeneity in the Measurement Model

• Measurement model specification requires the consideration of the nature of the relationship between constructs and measures.
• The direction of causality is from the indicators to the construct and the weights of formative indicators represent the importance of each indicator in explaining the variance of the construct (Edwards and Lambert 2007; Petter et al.
• In contrast, when a construct’s measures are formative, unobserved heterogeneity can lead to differences in the formative indicators’ weights across groups.
• As a consequence, the interpretation of the weights estimated using the overall sample is misleading, and the formative measures based on the overall sample represent neither the first group nor the second.
• Given this bias in the formative measures for service quality, the relationship between service quality and other constructs (e.g., customer satisfaction) is also likely to be biased.

Implications of Unobserved Heterogeneity for Model Validity

• If unobserved heterogeneity characterizes the data and results are based on the overall sample, the estimated model lacks validity because it will not uncover the true effects of the underlying groups.
• Unobserved heterogeneity is a threat to internal validity because contextual or group variables that affect results are overlooked, thereby resulting in an incomplete model.
• In addition, unobserved heterogeneity threatens statistical conclusion validity.
• Experienced users might have a different understanding of a system’s usefulness compared to inexperienced users, thereby leading to different correlation patterns for the PU construct’s indicators.
• Both threats are regularly addressed when testing for ME/I in multigroup models (i.e., observed heterogeneity) (see Steenkamp and Baumgartner 1998; Vandenberg and Lance.

In contrast, unobserved heterogeneity typically does not affect

• Content validity because the constructs’ measures are normally the same across groups and are grounded in theory.
• An increase in the value of a formative measure’s error term due to unobserved heterogeneity can lead to misinterpretations, as a high error term is typically associated with the construct measure’s incompleteness (Diamantopoulos et al. 2008).
• Finally, if unobserved heterogeneity is not uncovered, there is a threat to external validity (i.e., the ability to generalize findings beyond the current population and context) because the overall sample results are not representative of the underlying groups.
• Next, the authors present an overview of methods to uncover unobserved heterogeneity in structural equation models that researchers can apply to overcome threats to validity due to unobserved heterogeneity.

Uncovering Heterogeneity in Structural Equation Models

• The authors first synthesize and compare different methods in SEM (i.e., CB-SEM and PLS path modeling) to uncover observed and unobserved heterogeneity.
• Given the objectives of their paper, the authors focus primarily on methods in SEM to uncover unobserved heterogeneity.

Existing Methods to Uncover Observed Heterogeneity in SEM

• SEM methods to address observed heterogeneity are now commonly applied in the social and behavioral sciences, including information systems.
• The first category of methods identifies homogenous groups of observations (e.g., individuals) a priori based on grouping variables (e.g., psychographic or socio-demographic).
• The second category of methods aims at identifying moderating factors that explain heterogeneity in specific structural model relationships.
• Uncovering observed heterogeneity with both types of methods requires a priori knowledge about differences across groups.

CB-SEM Methods to Uncover Unobserved Heterogeneity

• In CB-SEM, the following two primary methods have been developed to uncover unobserved heterogeneity: (1) finite mixture models that extend multigroup CB-SEM (Arminger et al.
• Several applications and simulation studies (e.g., Arminger et al.
• In contrast to finite mixture models, hierarchical Bayesian models for CB-SEM, which were developed by Ansari et al. (2000), do not assume heterogeneity among a defined number of groups of individuals but estimate unobserved heterogeneity at the individual4 level using a random coefficients model.

2003) to extend the method to dichotomous variables and

• Missing data and evaluate the performance of these methods.
• While both the finite mixture and the hierarchical Bayesian CB-SEM models have been the subject of extensive methodological research, finite mixture models have been applied in empirical CB-SEM research to a greater extent.
• An increasing number of applications, especially in the marketing, econometrics, and sociology literatures, have utilized finite mixture models to uncover unobserved heterogeneity, thereby improving theoretical and practical implications (e.g., Bart et al.

PLS Path Modeling Methods to Uncover Unobserved Heterogeneity

• Multiple PLS segmentation methods have been proposed.
• The authors draw on Sarstedt’s (2008) review of these methods to identify the following key PLS segmentation methods:.

1. The PATHMOX (path modeling segmentation tree) algorithm (Sánchez 2009; Sánchez and Aluja 2006).5

• This algorithm requires the a priori specification of explanatory variables that are not used as indicators in the PLS path model to discover segments.
• It captures heterogeneity by estimating the probabilities of segment memberships for each observation in order to optimize the likelihood function.
• While FIMIX-PLS is generally applicable to PLS path models regardless of whether the latent variables are measured reflectively or formatively, it does not account for the heterogeneity in the measurement models.
• First, based on an assessment of the benefits and limitations of these methods, Sarstedt (2008, p. 152) concludes: “To sum up, FIMIX-PLS can presently be viewed as the most comprehensive and commonly used approach to capture heterogeneity in PLS path modeling.”.

Partial Least Squares–Prediction-Oriented Segmentation (PLS-POS)

• The latter feature of PLS-POS ensures continuous improvement of the objective criterion throughout the iterations of the algorithm (hill-climbing approach) and provides the ability to uncover very small niche segments.
• The segmentation objective in a PLS path model is to form homogenous groups of observations with increased predictive power (R² of the endogenous latent variables) of the group-specific path model estimates (compared to the overall sample model).
• The extension will be made available with the next release of SmartPLS.
• In the next section, the authors detail the comprehensive simulation experiments they conducted to evaluate whether the differences in the capabilities of FIMIX-PLS and PLS-POS noted in Table 4 hold empirically.

Simulations of PLS-POS and FIMIX-PLS Performance

• The authors conducted experiments with simulated data that define the true group-specific PLS parameters a priori.
• The authors assessed the performance of PLS-POS and FIMIX-PLS based on the differences between the true parameters and those estimated by each method.
• Subsequently, the authors compared the performance of PLS-POS and FIMIX-PLS in recovering the true parameter estimates.

Model Specification

• Consistent with most simulation studies on PLS path models (e.g., Chin et al. 2003), the authors specified a direct effects path model that includes four exogenous latent variables and one endogenous variable.
• The authors specified two versions of the path model: model 1 uses reflective measures for the exogenous and endogenous latent variables , while model 2 uses formative measures for the exogenous latent variables and reflective measures for the endogenous latent variables .
• The authors results were generally stable for these more complex models as well.
• Conversely, for group 2, p4 has a high true parameter value, while the path coefficients p1 to p3 have lower true values.
• For group 1, the measurement weights w1 and w3 have high true values, while weights w2 and w4 have low true values.

Factor Design of the Simulations

• The authors selection of experimental factors and their levels was informed by criteria that were shown to influence PLS path modeling or segmentation results in prior simulation studies.
• When the R² value in both groups is .85, the overall sample that combines the two groups has a R² value of .425 because of unobserved heterogeneity.
• (7) Measurement model heterogeneity—that is, the groupspecific differences in formative measurement weights (.25, .50, .75).
• The number of factors and the number of factor levels systematically increase the complexity of the PLS segmentation task.

Data Generation

• Simulation studies in PLS path modeling require that data generated for the indicators (manifest variables) match the true values of the model.
• Previous studies on PLS path modeling (e.g., Chin et al.
• 12The unequal condition has one segment with 80% and one with 20% of the total sample size.
• Becker et al./Discovering Unobserved Heterogeneity in SEM in the measurement model.
• Data for the formative indicators must first be generated to compute the latent variable scores for formative constructs.

Performance Assessment

• The objectives of their simulation experiments were to (1) assess PLS-POS and FIMIX-PLS in terms of their respective abilities to recover true group-specific parameters, (2) compare PLS-POS and FIMIX-PLS based on the assessment of their parameter recovery, and (3) identify the relative effects of the design factors on the parameter recovery of PLS-POS and FIMIX-PLS.
• The authors knew the true parameters of each factorial combination (i.e., the R², path coefficients, outer weights, and loadings) a priori based on the parameter settings for the data generation.
• The smaller the differences between the true values and the segmentation method’s parameter estimates, the better the parameter recovery.
• MAB values close to zero indicate near perfect parameter recovery.
• Finally, to understand the relative importance of the design factors, the authors evaluated parameter recovery (i.e., the path coefficient’s MAB) using a mixed-effects ANOVA model with the two segmentation methods (PLS-POS and FIMIX-PLS; within-subjects factor) and the eight design factors (between-subjects factors).

Results for Model 1: Reflective Measures

• Table 5 presents the results for the ANOVA with MAB as the dependent variable.
• The direct effect and all of the interaction effects of reliability are nonsignificant.
• The between-subjects effects identify the factors that influenced MAB for both segmentation methods.
• In general, the method has a significant and substantial impact on the parameter recovery for the reflective model.

WithinSubjects

• Df = degrees of freedom well as between measurement model heterogeneity and multicollinearity are significant and substantial but have very little impact compared to the factors discussed earlier, also known as Note.
• For the within-subjects effects, the method’s effect on MAB is significant and substantial.
• The method also significantly and substantially interacts with heterogeneity in both the structural model and the measurement model.
• While the MAB for PLS-POS is always below .05, thereby indicating good parameter recovery, the MAB for FIMIX-PLS increases when measurement model heterogeneity becomes higher and structural model heterogeneity becomes lower.
• Becker et al./Discovering Unobserved Heterogeneity in SEM.

Results for Model 2: Formative Measures

• Table 7 presents the results for the ANOVA in model 2 (formative measures) with MAB as the dependent variable.
• Again, for the sake of space and simplicity, Table 7 presents the direct effects, all two-way interactions with the method, and all other interactions that have significant and substantial effects (partial η² of more than .02).
• For the between-subjects effects, all of the direct effects on MAB are significant, but again, the effect of relative segment size (partial η² = .012) on MAB is not substantial.
• Moreover, the interaction effects between the structural model and measurement model heterogeneity as 680 MIS Quarterly Vol. 37 No. 3/September 2013.
• Becker et al./Discovering Unobserved Heterogeneity in SEM.

Segmentation

• Method Desired Criteria for a PLS Segmentation Method Ability to detect heterogeneity in reflective measures Ability to detect heterogeneity in formative measures Ability to detect heterogeneity in the structural model Maximizes group-specific R² of endogenous latent variables (prediction orientation).
• Ability to handle nonnormal data FIMIX-PLS Hahn et al. situations with very high structural model heterogeneity regardless of the measurement model heterogeneity and also in situations where the measurement model heterogeneity is low and the structural model heterogeneity is at moderate levels.
• Therefore, as the results in Figures 4a and 4b reveal, the parameter recovery ability of a segmentation method cannot be assessed independently for these two types of heterogeneity.
• It is worth noting that the interaction effect between method and data distribution is not substantial for either model 1 (reflective measures) or model 2 (formative measures).
• This might explain this initially surprising result.

Summary of Results

• Overall, the authors can conclude that the use of either PLS-POS or FIMIX-PLS is better for reducing biases in parameter estimates and avoiding inferential errors than ignoring unobserved heterogeneity in PLS path models.
• A notable exception is when there is low structural model heterogeneity and high formative measurement model heterogeneity; in this condition, FIMIX-PLS produces results that are even more biased than those resulting from ignoring heterogeneity and estimating the model at the overall sample level.
• FIMIX-PLS becomes more effective when there is high multicollinearity in the formative measures, while PLSPOS consistently performs well.
• The strongly correlated formative measures become closer to a homogenous reflective measurement of the construct.
• Thus, the simulation experiments provide an empirical assessment of the segmentation criteria associated with PLS-POS and FIMIX-PLS (Table 9).

A Process for Unobserved Heterogeneity Discovery

• Given the availability of methods to uncover unobserved heterogeneity, as discussed in the two previous sections, researchers working with SEM face the following two major questions: when to investigate unobserved heterogeneity and 684 MIS Quarterly Vol. 37 No. 3/September 2013.
• Becker et al./Discovering Unobserved Heterogeneity in SEM how to apply methods for uncovering unobserved heterogeneity and defining segments.
• The authors address these questions by proposing a UHD process and also by identifying how this process can be applied given the research objective (i.e., purely testing a model or testing and elaborating a model; Colquitt and Zapata-Phelan 2007).

How to Apply the UHD Process

• When selecting an appropriate UHD method, researchers have to determine whether they are interested in evaluating unobserved heterogeneity associated with latent segments or individual-level estimates (e.g., hierarchical Bayesian approach, fixed effects, and random effects).
• In contrast, if the objective is to examine unobserved heterogeneity for individual-level estimates, the described UHD process does not apply because the methods have different assumptions and objectives and require different data (i.e., several observations per individual).
• The UHD process for the discovery of latent segments consists of the following three stages:.

Selecting an Appropriate UHD Method (Stage 1 of the UHD Process)

• As discussed earlier, the methodological options for analyzing unobserved heterogeneity involving CB-SEM cover two conceptually different approaches (i.e., latent segment analysis and individual-level estimate correction).
• For latent segment analysis, the appropriate UHD choice is the finite mixture model as no model-based clustering alternative is available.
• For analyses involving PLS path modeling, there are no methods available that address unobserved heterogeneity associated with individual-level estimates.
• Latent segments in PLS path modeling can be uncovered using one of the two methods the authors present in this paper (i.e., FIMIX-PLS and PLSPOS).
• Therefore, researchers should choose FIMIX-PLS if their models include only reflective measures and heterogeneity is expected to affect only the structural model and not the measurement model.

Applying the UHD Method to Define Segments (Stage 2 of the UHD Process)

• After choosing the appropriate method for uncovering unobserved heterogeneity, the researcher has to apply the method to evaluate whether significant unobserved heterogeneity is present in the model and to define the number of segments to retain from the data.
• Determining the correct number of segments is important as under- or over-segmentation leads to biased results and misinterpretations.
• The second stage of the UHD process focuses on (1) defining with heuristics a range of statistically well-fitting segments and (2) evaluating the segments based on theoretical considerations.
• The steps in this stage emphasize that researchers (1) evaluate the plausibility of segments by connecting the segmentation solution to theory and (2) avoid capitalizing on data idiosyncrasies to improve the explained variance or significance of parameters.
• In mixture models, these heuristics include model-selection criteria that are well known from the model-selection literature (e.g., AIC, BIC, and CAIC) and can also be used to approximate the best fitting number of segments (Andrews and Currim 2003a; Sarstedt et al. 2011a).

In contrast, model-based clustering methods, such as PLSPOS, are not based on the mixture model concept and do not

• Becker et al./Discovering Unobserved Heterogeneity in SEM provide model-selection criteria.
• Researchers should not rely purely on heuristics (e.g., model-selection criteria in finite-mixture modeling or the explained variance per segment in PLS-POS) to retain the best fitting number of segments because past studies have shown heuristics to have a low probability of finding the true number of segments.

Validating the Segmentation Results (Stage 3 of the UHD Process)

• In the final stage of the UHD process, researchers should validate the segmentation results, including the number of segments, with external data not used in the estimation process.
• Becker et al./Discovering Unobserved Heterogeneity in SEM random splits to compare the stability of segmentation results (Jedidi et al. 1997), or (3) collect additional data (e.g., in a follow-up study) to evaluate the results and find new explanatory variables that match segments better to explain heterogeneity (i.e., make them accessible).
• Furthermore, repeating the segmentation study on a different population (i.e., sample) and testing the proposed explanatory variables (i.e., moderators or grouping variables) in follow-up studies increases the generalizability of the results.

When to Apply Methods to Uncover Unobserved Heterogeneity

• Given a model that is grounded in substantive theory, the complexity of the social and behavioral phenomena examined in IS research makes it plausible there will be heterogeneity in any sample that is used to test and refine the model.
• Hence, theory testers apply the UHD process to evaluate validity threats due to unobserved heterogeneity.
• If the research objective is theory testing and elaboration (i.e., expanders; Colquitt and Zapata-Phelan 2007), uncovering unobserved heterogeneity not only serves as a validity check but can also guide researchers to identify variables explaining the uncovered segments and to integrate these variables to expand the model/theory.
• Becker et al./Discovering Unobserved Heterogeneity in SEM contrast, the other segment shows a strong positive relationship between PEOU and IU and a weak, or nonsignificant, relationship between PU and IU .
• The researcher concludes that these two identified segments (i.e., users emphasizing PU or PEOU) are theoretically plausible (i.e., within TAM, it is reasonable that there are different users who emphasize different system characteristics) and conceptually important for the theory.

Limitations and Future Research

• While their study makes contributions, it has its limitations and opens up avenues for future research.
• First, the validity and generalizability of simulation studies are limited by the choice of design factors and factor levels.
• The analysis of all factorlevel combinations of the two PLS path models entailed 126,720 simulated segmentation runs for assessing the performance of PLS-POS and FIMIX-PLS.

In addition, researchers may want to assess the effect of unobserved heterogeneity in models that do not comply with the recursive nature of models imposed by PLS path models.

• If heterogeneity affects non-recursive relationships, it might have a strong impact on the ability of both PLS segmentation methods (FIMIX-PLS and PLS-POS) to uncover unobserved heterogeneity.
• Third, this research does not focus on the parameter settings of the methods or the time needed to arrive at the final segmentation solution.
• The authors simulations suggest that PLS-POS is more time consuming than FIMIX-PLS.
• In absolute terms, PLS-POS works within acceptable timeframes.

Conclusion

• The authors differentiated between observed and unobserved heterogeneity and showed why unobserved heterogeneity biases structural equation model estimates, leads to Type I and Type II errors, and is a threat to different types of validity (i.e., internal, instrumental, statistical conclusion, and external).
• The authors findings also reveal that unobserved heterogeneity in formative measures and in the structural model should be evaluated collectively.
• As FIMIX-PLS does not uncover heterogeneity in measurement models, PLS-POS should be applied for discovering unobserved heterogeneity if PLS path models include formative measures.
• Moreover, in situations in which the researcher discovers anomalies that must be resolved through theoretical elaboration, theory development is significantly enhanced by abduction.
• Using the presented methods in PLS path modeling and CB-SEM within the UHD process is a possible way to achieve this goal.

Figures

Table 3. Overview of CB-SEM Methods to Uncover Unobserved Heterogeneity in SEM

Figure 2. The Models

Table 1. Conclusions from the Simulation Study on Heterogeneity Effects

Table 6. MAB in Model 1 (Reflective Measures) for Each Method

Figure 3. MAB of Both Segmentation Methods for Model 1 (Reflective Measures)

Figure 5. Unobserved Heterogeneity Discovery (UHD) Process

Figure 1. Examples for Unobserved Heterogeneity in TAM

Table 7. Model 2 (Formative Measures) ANOVA Explaining MAB by Method (PLS-POS/FIMIX-PLS) and Design Factors

Table 5. Model 1 (Reflective Measures) ANOVA Explaining MAB by Method (PLS-POS/FIMIX-PLS) and Design Factors

Table 2. Implications of Unobserved Heterogeneity for Model Validity

Figure 4. MAB of Both Methods for Different Structural and Measurement Model Heterogeneity

Table 8. MAB in Model 2 (Formative Measures) for Each Method

Full text

Georgia State University Georgia State University
ScholarWorks @ Georgia State University ScholarWorks @ Georgia State University
Computer Information Systems Faculty
Publications
Department of Computer Information Systems
9-2013
Discovering Unobserved Heterogeneity in Structural Equation Discovering Unobserved Heterogeneity in Structural Equation
Models to Avert Validity Threats Models to Avert Validity Threats
Jan-Michael Becker
University of Cologne
, j.becker@wiso.uni-koeln.de
Arun Rai
Georgia State University
, arunrai@gsu.edu
Christian M. Ringle
University of Newcastle - Australia
, christian.ringle@newcastle.edu.au
Franziska Völckner
University of Cologne
, voelckner@wiso.uni-koeln.de
Follow this and additional works at: https://scholarworks.gsu.edu/cis_facpub
Part of the Management Information Systems Commons
Recommended Citation Recommended Citation
Becker, J-M., Rai, A., Voleckner, F., and Ringle, C., Discovering Unobserved Heterogeneity in Structural
Equation Models, MIS Quarterly, September 2013, 37(3), 665-694. http://misq.org/discovering-
unobserved-heterogeneity-in-structural-equation-models-to-avert-validity-threats.html.
This Article is brought to you for free and open access by the Department of Computer Information Systems at
ScholarWorks @ Georgia State University. It has been accepted for inclusion in Computer Information Systems
Faculty Publications by an authorized administrator of ScholarWorks @ Georgia State University. For more
information, please contact scholarworks@gsu.edu.

RESEARCH ESSAY
DISCOVERING UNOBSERVED HETEROGENEITY IN
STRUCTURAL EQUATION MODELS TO
AVERT VALIDITY THREATS
1
Jan-Michael Becker
Department of Marketing and Brand Management, University of Cologne,
Cologne, 50923, GERMANY {j.becker@wiso.uni-koeln.de}
Arun Rai
Center for Process Innovation and Department of Computer Information Systems, Robinson College of Business,
Georgia State University, Atlanta, GA 30303 U.S.A. {arunrai@gsu.edu}
Christian M. Ringle
Institute for Human Resource Management and Organizations, Hamburg University of Technology (TUHH),
Hamburg, 21073, GERMANY {ringle@tuhh.de} and
Faculty of Business and Law, University of Newcastle, Callaghan, NSW 2308 AUSTRALIA {christian.ringle@newcastle.edu.au}
Franziska Völckner
Department of Marketing and Brand Management, University of Cologne,
Cologne, 50923, GERMANY {voelckner@wiso.uni-koeln.de}
1
A large proportion of information systems research is concerned with developing and testing models pertaining
to complex cognition, behaviors, and outcomes of individuals, teams, organizations, and other social systems
that are involved in the development, implementation, and utilization of information technology. Given the
complexity of these social and behavioral phenomena, heterogeneity is likely to exist in the samples used in IS
studies. While researchers now routinely address observed heterogeneity by introducing moderators, a priori
groupings, and contextual factors in their research models, they have not examined how unobserved hetero-
geneity may affect their findings. We describe why unobserved heterogeneity threatens different types of
validity and use simulations to demonstrate that unobserved heterogeneity biases parameter estimates, thereby
leading to Type I and Type II errors. We also review different methods that can be used to uncover unobserved
heterogeneity in structural equation models. While methods to uncover unobserved heterogeneity in
covariance-based structural equation models (CB-SEM) are relatively advanced, the methods for partial least
squares (PLS) path models are limited and have relied on an extension of mixture regression—finite mixture
partial least squares (FIMIX-PLS) and distance measure-based methods—that have mismatches with some
characteristics of PLS path modeling. We propose a new method—prediction-oriented segmentation (PLS-
POS)—to overcome the limitations of FIMIX-PLS and other distance measure-based methods and conduct
extensive simulations to evaluate the ability of PLS-POS and FIMIX-PLS to discover unobserved heterogeneity
in both structural and measurement models. Our results show that both PLS-POS and FIMIX-PLS perform
1
Ron Thompson was the accepting senior editor for this paper. Ron Cenfetelli served as the associate editor.
The appendices for this paper are located in the “Online Supplements” section of the MIS Quarterly’s website (http://www.misq.org).
MIS Quarterly Vol. 37 No. 3, pp. 665-694/September 2013 665

Becker et al./Discovering Unobserved Heterogeneity in SEM
well in discovering unobserved heterogeneity in structural paths when the measures are reflective and that
PLS-POS also performs well in discovering unobserved heterogeneity in formative measures. We propose an
unobserved heterogeneity discovery (UHD) process that researchers can apply to (1) avert validity threats by
uncovering unobserved heterogeneity and (2) elaborate on theory by turning unobserved heterogeneity into
observed heterogeneity, thereby expanding theory through the integration of new moderator or contextual
variables.
Keywords: Unobserved heterogeneity, validity, structural equation modeling, partial least squares, formative
measures, prediction-oriented segmentation
Introduction
Assuming that data in empirical studies are homogeneous and
represent a single population is often unrealistic in the social
and behavioral sciences, such as information systems, man-
agement, and marketing (Rust and Verhoef 2005; Wedel and
Kamakura 2000). There may be significant heterogeneity in
the data across unobserved groups, and it can bias parameter
estimates, lead to Type I and Type II errors, and result in
invalid conclusions (Jedidi et al. 1997). Consider the fol-
lowing technology acceptance model (TAM) example: A
researcher is interested in individuals’ intention to use an IT
system or service (Davis et al. 1989; Venkatesh 2000;
Venkatesh and Davis 2000; Venkatesh et al. 2003). Informed
by existing theory, the researcher proposes a model in which
perceived usefulness (PU) and perceived ease of use (PEOU)
of the IT system explain intention to use the system (IU)
(Figure 1). The empirical results reveal that PU and PEOU
are equally important in explaining IU. However, the theory
and model overlook the two underlying groups: experienced
IT users (Figure 1a, segment 1) and inexperienced IT users
(Figure 1a, segment 2). Experienced users show a strong
positive relationship between PU and IU and a weak, or non-
significant, relationship between PEOU and IU. In contrast,
inexperienced users show a strong positive relationship
between PEOU and IU and a weak, or nonsignificant, rela-
tionship between PU and IU (Figure 1a). In this scenario,
drawing inferences based on results from the overall sample
would lead to Type I errors as we would be overgeneralizing
the significant findings from the overall sample to the
underlying user groups, one with a nonsignificant estimate for
PEOUIU and the other with a nonsignificant estimate for
PUIU. If the model is not refined to accommodate this
unobserved heterogeneity, a system that is unsuitable for
either user group (i.e., one with average usefulness and
average ease of use) may be provided to all users.
In addition, a study may not find PEOU to be a significant
predictor of IU because of unobserved heterogeneity across
two groups of users (i.e., experienced versus inexperienced).
If experienced users (Figure 1b, segment 1) perceive an easy-
to-use system (i.e., high PEOU) as being too simple to fulfill
their needs, they may show a strong negative relationship
between PEOU and IU. In contrast, if inexperienced users
(Figure 1b, segment 2) show a strong positive relationship
between PEOU and IU, as in the first example, a sign reversal
occurs between the two groups with regard to the effect of
PEOU on IU, thereby leading to an overall nonsignificant
effect of PEOU on IU and a Type II error.
Recent TAM models acknowledge existing heterogeneity by
incorporating experience as a moderator of PEOU’s effect on
IU. However, before its inclusion in the theory, experienced
versus inexperienced users represented unobserved hetero-
geneity that could lead to biased findings on the effects of PU
and PEOU on IU. This illustration shows how not accounting
for unobserved heterogeneity can lead to misinterpretations
and invalid conclusions in IS research—a point we emphasize
later in the paper based on a review of 12 meta-analysis
studies on key IS phenomena (see Table A1 in Appendix A).
Despite the threats to validity from unobserved heterogeneity,
there are important gaps in the IS literature about the specific
threats to validity and how to safeguard against them.
(1) While IS studies now routinely address observed hetero-
geneity by introducing moderators, a priori groupings,
contextual factors, and control variables in their research
models, they have not considered unobserved hetero-
geneity in their data. In fact, none of the papers ap-
pearing in the field’s two most widely recognized jour-
nals (MIS Quarterly and Information Systems Research)
over the last 20 years that have developed and tested
structural equation models have examined unobserved
heterogeneity. Our first research objective is to introduce
the concept of unobserved heterogeneity in the IS litera-
ture and to show how IS researchers can safeguard
against biases and facilitate theory development.
(2) While research in some fields notes that unobserved
heterogeneity threatens empirical results and their inter-
pretation, a systematic analysis of the threats to specific
666 MIS Quarterly Vol. 37 No. 3/September 2013

Becker et al./Discovering Unobserved Heterogeneity in SEM
(a) TAM Example 1 (b) TAM Example 2
Figure 1. Examples for Unobserved Heterogeneity in TAM
types of validity is missing in the literature. Our second
research objective is to evaluate the implications of
unobserved heterogeneity for four types of validity (i.e.,
instrument, internal, statistical conclusion, and external
validity; Cook and Campbell 1976, 1979; Straub 1989),
thereby broadening our understanding of the specific
validity threats that arise from unobserved heterogeneity.
(3) In structural equation modeling (SEM), unobserved
heterogeneity is not only a validity threat for the struc-
tural model but also for the measurement model regard-
less of whether the measures are reflective or formative.
While heterogeneity in reflective measures has been
discussed in terms of measurement equivalence or invari-
ance (ME/I) (e.g., Steenkamp and Baumgartner 1998;
Vandenberg and Lance 2000), the implications of unob-
served heterogeneity for formative measures have not
been examined. Our third research objective is to evalu-
ate the implications of unobserved heterogeneity for
formative measures.
(4) In contrast to covariance-based SEM (CB-SEM; e.g.,
Jöreskog 1978, 1982), research on partial least squares
(PLS) path modeling (e.g., Chin 1998; Lohmöller 1989;
Wold 1982) has paid limited attention to unobserved
heterogeneity. Only recently has a method been pro-
posed to detect unobserved heterogeneity in PLS path
models: finite mixture partial least squares (FIMIX-PLS;
Hahn et al. 2002; Sarstedt and Ringle 2010). However,
FIMIX-PLS does not account for heterogeneity in the
measurement model and assumes multivariate normal
distributions for latent variables. Furthermore, there is
limited evidence of this method’s performance in dis-
covering unobserved heterogeneity. Our fourth research
objective is to propose and evaluate a new method, PLS
prediction-oriented segmentation (PLS-POS), which does
not follow distributional assumptions and uncovers
unobserved heterogeneity not only in the structural model
but also in the measurement model.
(5) Researchers facing the problem of unobserved hetero-
geneity in their empirical work lack guidelines on how to
apply methods systematically to uncover unobserved
heterogeneity. Therefore, our fifth research objective is
to develop an unobserved heterogeneity discovery
(UHD) process to guide researchers in applying methods
to ensure the validity of findings and to elaborate theory
by turning unobserved heterogeneity into observed
heterogeneity.
By addressing the above research objectives, we make six
contributions. First, we provide evidence and reasoning for
why unobserved heterogeneity is an important issue in IS
research. Second, we demonstrate that unobserved hetero-
geneity in SEM has implications not only for the structural
model but also for measurement models. Third, we identify
the implications of unobserved heterogeneity for different
types of validity and surface the importance of uncovering
unobserved heterogeneity to avoid validity threats. Fourth,
we introduce the new PLS-POS method for detecting unob-
served heterogeneity. This method is specifically developed
to fit PLS path modeling, as it employs a prediction-oriented
and nonparametric approach and uncovers heterogeneity in
both the structural model and the (formative) measurement
MIS Quarterly Vol. 37 No. 3/September 2013 667

Becker et al./Discovering Unobserved Heterogeneity in SEM
models and thereby overcomes the limitations of FIMIX-PLS
and other distance measure-based methods. Fifth, we evaluate
FIMIX-PLS and PLS-POS using an extensive simulation
study and generate important insights into the performance of
the two methods in uncovering unobserved heterogeneity in
PLS path models. Sixth, we provide a UHD process to guide
researchers in discovering and addressing unobserved
heterogeneity in structural equation models.
Concept of Heterogeneity and its
Treatment in IS Research
Researchers can obtain different parameter estimates when
they consider differences among observations relative to when
they overlook them. However, heterogeneity among observa-
tions is not necessarily captured by variables that are precon-
ceived by the researcher and specified by existing theory, as
it can exist beyond these previously identified variables
(Jedidi et al. 1997). As a consequence, it is necessary to
differentiate between the following two types of hetero-
geneity: (1) observed heterogeneity when subpopulations are
defined a priori based on known variables and (2) unobserved
heterogeneity when the subpopulations in the data are
unknown (Lubke and Muthén 2005).
Observed Heterogeneity
Observed heterogeneity occurs when differences in parameter
estimates between groups are expected a priori for the phen-
omenon—that is, when group differences are explained by
existing theory that incorporates moderators or contextual
factors. Examples of such moderators or contextual factors
considered in IS research include individual cultural differ-
ences (e.g., individualism versus collectivism; Srite and Kara-
hanna 2006), individual demographic differences (e.g., gen-
der, income levels, and education; Hsieh et al. 2008; Venka-
tesh et al. 2003), and organizational demographic differences
(e.g., large versus small firms; Rai et al. 2006). In our TAM
example from earlier, existing theory expects gender-based
heterogeneity in structural paths (i.e., men are expected to
have a stronger relationship between PU and IU, and women
are expected to have a stronger relationship between PEOU
and IU) (e.g., Venkatesh and Morris 2000). Moreover,
existing theory expects contextual variables, such as volun-
tariness or task type (e.g., Venkatesh and Davis 2000), or
psychographic variables, such as personal innovativeness and
computer attitude, to cause heterogeneity in the relationships
among the TAM constructs (e.g., Venkatesh and Bala 2008).
Unobserved Heterogeneity
When theory does not assume heterogeneity even though it
exists or when theory indicates heterogeneity but the specified
group variables do not sufficiently capture it in the popula-
tion, unobserved heterogeneity occurs. In such situations,
researchers need to uncover unobserved heterogeneity by seg-
menting data to form homogenous groups. If the differences
uncovered by segmentation can be explained post hoc using
contextual or demographic variables (e.g., culture, gender,
experience, etc.) making the groups accessible, theory can be
expanded accordingly, and unobserved heterogeneity is
turned into observed heterogeneity for future studies. If the
differences cannot be explained by well-known contextual
variables, the researcher has to consider complementary
theoretical explanations for the phenomenon.
Treatment of Heterogeneity in IS Research
Given the complexity of the social and behavioral phenomena
tackled in IS research, heterogeneity is likely to exist in
samples that are used to develop, test, and refine models. If
this heterogeneity is not uncovered and controlled, the (unob-
served) heterogeneity can bias results and conclusions (e.g.,
Ansari et al. 2000; Johns 2006). Consequently, unobserved
heterogeneity is receiving increasing attention in related disci-
plines (e.g., marketing, where scholars study similar complex
phenomena pertaining to consumer choices and preferences,
the alignment of firm-level marketing strategies, interorgani-
zational relationships, and the business value of tangible and
intangible resources) to safeguard against biases and probe the
underlying reasons for unobserved heterogeneity (e.g.,
Rigdon et al. 2010). This enhances the likelihood of
obtaining valid results as well as of generating greater theo-
retical contributions. Methodologists in marketing, econo-
metrics, and psychology have proposed advances to uncover
unobserved heterogeneity in various approaches—for
instance, regression analysis (DeSarbo and Cron 1988; Späth
1979; Wedel and DeSarbo 1994), CB-SEM (e.g., Ansari et al.
2000; Jedidi et al. 1997; Muthén 1989), panel data models
(e.g., Allenby and Rossi 1998; Popkowski Leszczyc and Bass
1998), and conjoint analysis (e.g., DeSarbo et al. 1995;
Gilbride et al. 2006; Lenk et al. 1996).
While IS studies now routinely address observed hetero-
geneity by introducing moderators, a priori groupings, con-
textual factors, and control variables in their research models,
they have not examined threats to validity due to unobserved
heterogeneity. Our review of 12 meta-analysis studies that
synthesize the findings of empirical research across various IS
phenomena (e.g., technology acceptance, IT investment pay-
668 MIS Quarterly Vol. 37 No. 3/September 2013

Frequently asked questions

Q1. How many factor combinations are required to ensure stability of the results?

To ensure stability of the results, all factor combinations include 30 data-generation and segmentation runs for each segmentation method, so in total, (384 +Simulation studies in PLS path modeling require that data generated for the indicators (manifest variables) match the true values of the model. 

Q2. What is the advantage of applying the parametric finite mixture regression concept to PLS path models?

The advantage of applying the parametric finite mixture regression concept to PLS path models is that it offers segment retention criteria (e.g., AIC, BIC, and CAIC; Hahn et al. 

Q3. What are the main reasons why finite mixture models are being used in empirical CB-S?

An increasing number of applications, especially in the marketing, econometrics, and sociology literatures, have utilized finite mixture models to uncover unobserved heterogeneity, thereby improving theoretical and practical implications (e.g., Bart et al. 

Q4. What methods are used to uncover unobserved heterogeneity in the data?

finite mixture models address unobserved heterogeneity in the data by grouping observations and estimating group-specific 3There are several methods to uncover both observed and unobserved heterogeneity in other methodological contexts—for example, regression analysis (DeSarbo and Cron 1988; Späth 1979; Wedel and DeSarbo1994), panel data models (Allenby and Rossi 1998; Popkowski Leszczyc and Bass 1998), and conjoint analysis (DeSarbo et al. 

Q5. What is the effect of the measurement model heterogeneity on MAB?

In contrast, parameter recovery for FIMIX-PLS decreases with decreasing structural model heterogeneity or increasing measurement model heterogeneity. 

Q6. What is the effect of unobserved heterogeneity on the parameter estimates?

The results show that unobserved heterogeneity biases the parameter estimates, decreases the R², and increases the risk of Type The authorand Type II errors. 

Q7. What are the examples of moderators or contextual factors considered in IS research?

Examples of such moderators or contextual factors considered in IS research include individual cultural differences (e.g., individualism versus collectivism; Srite and Karahanna 2006), individual demographic differences (e.g., gender, income levels, and education; Hsieh et al. 

Q8. What is the reason why the results of one study are not generalized to other studies?

The observation of inconsistent, conflicting, or mixed findings in the 12 meta-analyses in Table A1 (Appendix A) also show that the results of one study often cannot be generalized to other studies (indicating low external validity) with unobserved heterogeneity being one of the plausible reasons. 

Q9. What is the main difference between formative and reflective measures?

While recent research has discussed ME/I in formative measures (Diamantopoulos and Papadopoulos 2010), it is important to uncover formative indicator weight differences due to unobserved heterogeneity in order to avoid ambiguous interpretations.