Showing papers in "Organizational Research Methods in 2022"

An Updated Guideline for Assessing Discriminant Validity

Abstract: Discriminant validity was originally presented as a set of empirical criteria that can be assessed from multitrait-multimethod (MTMM) matrices. Because datasets used by applied researchers rarely l...

Text Preprocessing for Text Mining in Organizational Research: Review and Recommendations:

Abstract: Recent advances in text mining have provided new methods for capitalizing on the voluminous natural language text data created by organizations, their employees, and their customers. Although often...

A Marginal Effects Approach to Interpreting Main Effects and Moderation

Abstract: This Short Methodological Report builds on research about moderation practices by focusing on a marginal effects approach to interpreting how a main effect is informed by the presence of a moderati...

Eight Simple Guidelines for Improved Understanding of Transformations and Nonlinear Effects

Abstract: Transforming variables before analysis or applying a transformation as a part of a generalized linear model are common practices in organizational research. Several methodological articles addressi...

Attitude Toward the Color Blue: An Ideal Marker Variable

Abstract: Researchers often turn to post hoc statistical techniques to identify common method variance (CMV) in same source data and one viable option is to use a marker variable. The choice of marker variable is important, yet these variables are difficult to find, primarily because they must be theoretically unrelated to study variables but measured in the same way (e.g., perceptual; on a Likert scale). This manuscript uses scale development best practices to create a marker variable—attitude toward the color blue—that can be applied in a wide variety of social science research. Scale reliability and validity are addressed, discriminant validity with other measures that detect CMV is tested, and the Confirmatory Factor Analysis Marker Technique is applied with this scale. An experiment designed to analyze the effect of the placement of the scale in surveys is reported. Recommendations to researchers for use of this new scale to detect CMV are provided.

Review Research as Scientific Inquiry

Abstract: This article and the related Feature Topic at Organizational Research Methods upcoming were motivated by the concern that despite the bourgeoning number and diversity of review articles, there was a lack of guidance on how to produce rigorous and impactful literature reviews. In this article, we introduce review research as a class of research inquiries that uses prior research as data sources to develop knowledge contributions for academia, practice and policy. We first trace the evolution of review research both outside of and within management including the articles published in this Feature Topic, and provide a holistic definition of review research. Then, we argue that in the plurality of forms of review research, the alignment of purpose and methods is crucial for high-quality review research. To accomplish this, we discuss several review purposes and criteria for assessing review research's rigor and impact, and discuss how these and the review methods need to be aligned with its purpose. Our paper provides guidance for conducting or evaluating review research and helps establish review research as a credible and legitimate scientific endeavor.

Anticipating and Addressing the Politicization of Research

Abstract: We examine an underaddressed issue in organizational research, the nature of the politicization of knowledge and its consequences for conducting research. Drawing on an illustrative case from a PhD...

Construct Development and Validation in Three Practical Steps: Recommendations for Reviewers, Editors, and Authors*

Abstract: We review contemporary best practice for developing and validating measures of constructs in the organizational sciences. The three basic steps in scale development are: (a) construct definition, (b) choosing operationalizations that match the construct definition, and (c) obtaining empirical evidence to confirm construct validity. While summarizing this 3-step process [i.e., Define-Operationalize-Confirm], we address many issues in establishing construct validity and provide a checklist for journal reviewers and authors when evaluating the validity of measures used in organizational research. Among other points, we pay special attention to construct conceptualization, acknowledging existing constructs, improving existing measures, multidimensional constructs, macro-level constructs, and the need for independent samples to confirm construct validity and measurement equivalence across subpopulations.

On the Use of Balanced Item Parceling to Counter Acquiescence Bias in Structural Equation Models

Abstract: We propose the use of balanced item parcels to account for method effects caused by acquiescent responding. The use of balanced parcels avoids the need to model method effects explicitly and result...

Conducting and Evaluating Multilevel Studies: Recommendations, Resources, and a Checklist

Abstract: Multilevel methods allow researchers to investigate relationships that expand across levels (e.g., individuals, teams, and organizations). The popularity of these methods for studying organizational phenomena has increased in recent decades. Methodologists have examined how these methods work under different conditions, providing an empirical base for making sound decisions when using these methods. In this article, we provide recommendations, tools, resources, and a checklist that can be useful for scholars involved in conducting or assessing multilevel studies. The focus of our article is on two-level designs, in which Level-1 entities are neatly nested within Level-2 entities, and top-down effects are estimated. However, some of our recommendations are also applicable to more complex multilevel designs.

From Ties to Events in the Analysis of Interorganizational Exchange Relations

Abstract: Relational event models expand the analytical possibilities of existing statistical models for interorganizational networks by: (i) making efficient use of information contained in the sequential ordering of observed events connecting sending and receiving units; (ii) accounting for the intensity of the relation between exchange partners, and (iii) distinguishing between short- and long-term network effects. We introduce a recently developed relational event model (REM) for the analysis of continuously observed interorganizational exchange relations. The combination of efficient sampling algorithms and sender-based stratification makes the models that we present particularly useful for the analysis of very large samples of relational event data generated by interaction among heterogeneous actors. We demonstrate the empirical value of event-oriented network models in two different settings for interorganizational exchange relations—that is, high-frequency overnight transactions among European banks and patient-sharing relations within a community of Italian hospitals. We focus on patterns of direct and generalized reciprocity while accounting for more complex forms of dependence present in the data. Empirical results suggest that distinguishing between degree- and intensity-based network effects, and between short- and long-term effects is crucial to our understanding of the dynamics of interorganizational dependence and exchange relations. We discuss the general implications of these results for the analysis of social interaction data routinely collected in organizational research to examine the evolutionary dynamics of social networks within and between organizations.

Immersion in Organizational Ethnography: Four Methodological Requirements to Immerse Oneself in the Field

Abstract: This article addresses the question of how to achieve immersion in organizational ethnography. Working through a broad set of ethnographies in organization studies, sociology, and anthropology, I develop a multi-faceted conceptualization of immersion and offer a framework that integrates four methodological principles—involvement, engagement, duration, and sites—to help organizational ethnographers achieve immersion. In closing, I discuss how this framework advances ongoing debates about involvement, multi-sitedness, and fieldwork design, resulting in a more systematic and reflexive approach to immersion in organizational ethnography.

Entwined Positionality and Interpretive Frames of Reference: An Autoethnographic Account

Abstract: This paper investigates the ethnographic researcher's positionality and its role in sensemaking within the research process. Using autoethnographic data of the first author - a black female West African (Yoruba) scholar in a Western organizational context, we adopt a critical sensemaking approach to make sense of the researcher's field experience. We propose a conceptualization of the researcher's positionality as one that is entwined in the field, being an active interaction of the researcher's formative context with her sensory capabilities. We demonstrate how openness to the researcher's entwined positionality generates interpretive frames of reference and uncovers nuances in the sensemaking process, which widens the scope for reflexivity. We offer a methodological roadmap for engaging entwined positionality in reflexive practice and contribute to the body of research which challenges the idea of the detached researcher; thus, we respond to the growing calls for integrating the elements of a researcher's positionality into research in a way that enhances reflexivity.

Reviewer Resources: Confirmatory Factor Analysis

Abstract: Confirmatory factor analyses (CFA) are widely used in the organizational literature. As a result, understanding how to properly conduct these analyses, report the results, and interpret their implications is critically important for advancing organizational research. The goal of this paper is to summarize the complexities of CFA models and, therefore, to provide a resource for journal reviewers and researchers who are using CFA in their research. The topics covered in this paper include the estimation process, power analyses, model fit, and model modifications, among other things. In addition, this paper concludes with a checklist that summarizes the key points that are discussed and can be used to evaluate future studies that incorporate CFA.

Endogenous Moderator Models: What They are, What They Aren’t, and Why it Matters

Abstract: Models that combine moderation and mediation are increasingly common. One such model is that in which one variable causes another variable that, in turn, moderates the relationship between two other variables. There are many recent examples of these Endogenous Moderator Models (EMMs). They bear little superficial resemblance to second-stage moderation models, and they are almost never conceptualized and tested as such. We use path analytic equations to show that this is precisely what EMMs are. Specifically, we use these path analytic equations and a review of recent EMMs in order to show that these models are seldom conceptualized or tested properly and to understand the best ways to handle such models. We then use Monte Carlo simulation to show the consequences of testing these models as they are typically tested rather than as second-stage moderation models. We end with recommendations and provide example datasets and code for SPSS and R.

Using CATA and Machine Learning to Operationalize Old Constructs in New Ways: An Illustration Using U.S. Governors’ COVID-19 Press Briefings

Abstract: Increased computing power and greater access to online data have led to rapid growth in the use of computer-aided text analysis (CATA) and machine learning methods. Using “big data”, researchers have not only advanced new streams of research, but also new research methodologies. Noting this trend and simultaneously recognizing the value of traditional research methods, we lay out a methodology that bridges the gap between old and new approaches to operationalize old constructs in new ways. With a combination of web scraping, CATA, and supervised machine learning, using labeled ground truth data (i.e., data with known inputs and outputs), we train a model to predict CIP (Charismatic-Ideological-Pragmatic) leadership styles from running text. To illustrate this method, we apply the model to classify U.S. state governors’ COVID-19 press briefings according to their CIP leadership style. In addition, we demonstrate content and convergent validity of the method.

How Rare Is Rare? How Common Is Common? Empirical Issues Associated With Binary Dependent Variables With Rare Or Common Event Rates

Abstract: The use of logit and probit models when examining binary dependent variables including those in the form 0/1 (i.e., dummy variables), yes/no, and true/false (hereafter binary DVs) is commonplace. Yet, the appropriateness and effectiveness of such models are challenged when the event rate of a binary DV is rare or common. To better understand the impact on the field of strategy, we undertook a literature review and assessed recently published research in the Strategic Management Journal. We then utilized Monte Carlo simulations with results showing that as event rates become rarer or more common, issues including biased coefficients, standard error inflation, low statistical power to detect significant effects, and model convergence failure increasingly arise. In addition, small sample sizes amplified these empirical issues. Using a strategy example study, we also show how various analytic tools can lead to different findings when empirical models face an extreme event rate with small sample sizes. Based on our findings, we provide step-by-step guidance for strategy researchers going forward.

Evaluating Interdependence in Workgroups: A Network-Based Method

Abstract: Interdependence is a defining characteristic of groups and teams. However, a vast range of constructs and conceptualizations for interdependence has left researchers with a dizzying array of frameworks, metrics, and perspectives with which to evaluate interdependence. This situation leaves researchers with little guidance on how to theorize about or measure interdependence. As a solution, we propose a network-based perspective of interdependence. This network-based framework moves beyond network approaches to understanding interdependence that have been proposed in the past in three ways. First, this framework is applied generally to interdependence and not to an isolated form of interdependence. Second, building on previous network-based perspectives of interdependence, we present a procedure to conceptualize a team's interdependent relationships in terms of networks. Third, we utilize the network perspective to present a standardized index of interdependence. Using illustrative examples, we demonstrate the utility of this network-based approach and present various recommendations discussing how these approaches advance the study of interdependence.

Should Moderated Regressions Include or Exclude Quadratic Terms? Present Both! Then Apply Our Linear Algebraic Analysis to Identify the Preferable Specification

Abstract: Organizational research increasingly tests moderated relationships using multiple regression with interaction terms. Most research does so with little concern regarding curvilinear relationships. But methodologists have established that omitting quadratic terms of correlated primary variables may create false interaction positives (type 1 errors). If dependent variables are generated by the canonical process where fully specified regressions satisfy the Gauss-Markov assumptions, including quadratics solves the problem. But our empirical analysis of published organizational research suggests that dependent variables are often generated by processes where, even with quadratics included, regression analyses will remain Gauss-Markov non-compliant. In such cases, our linear algebraic analysis demonstrates that including quadratics—even those motivated by compelling theory—may exacerbate rather than mitigate the incidence of false interaction positives. The interaction coefficient may substantially change its magnitude and even flip sign once quadratics are included, and not necessarily for the better. We encourage researchers to present two full sets of results when testing moderating hypotheses—one with, and one without, quadratic terms. Researchers should then answer five questions developed here in order to determine the preferable set of results.

Measuring What Matters: Assessing how Executives Reference Firm Performance in Corporate Filings

Abstract: We know very little about the performance measures executives use to make decisions. To fill this void, we investigate the performance variables that executives reference in corporate filings with the SEC. Our analyses suggest that executives refer to monetary variables (i.e., revenue, profit, and cash flow) in over 98% of filings. In contrast, executives refer to the unitless performance measures scaled by size (i.e., return on assets, return on equity), which are favored by organizational scholars, in less than 15% of filings. We find that this preference for unscaled measures remains across market capitalization and actual firm performance. In other words, even observations with the highest levels of ROA and ROE are more likely to include monetary measures as opposed to ratios. In fact, we find that almost every observation that references ratios also includes monetary measures of firm performance. Stated differently, our findings suggest executives use ratios in addition to—and not instead of—monetary measures. We discuss research opportunities for scholars to further align with the practitioner perspective and to revisit conceptualizations of firm performance.

Corrigendum to Using CATA and Machine Learning to Operationalize Old Constructs in New Ways: An Illustration Using U.S. Governors’ COVID-19 Press Briefings

Abstract: In the above-referenced article, Appendices A to D were inadvertently uploaded as a supplemental material, but now we have included the Appendices A to D back in the main published paper, as appendices included a sample code and how-to tutorial guidance and shall provide a great value to the readers. The referenced article has been corrected accordingly. © The Author(s) 2022.

SRM_R: A Web-Based Shiny App for Social Relations Analyses

Abstract: Many topics in organizational research involve examining the interpersonal perceptions and behaviors of group members. The resulting data can be analyzed using the social relations model (SRM). This model enables researchers to address several important questions regarding relational phenomena. In the model, variance can be partitioned into group, actor, partner, and relationship; reciprocity can be assessed in terms of individuals and dyads; and predictors at each of these levels can be analyzed. However, analyzing data using the currently available SRM software can be challenging and can deter organizational researchers from using the model. In this article, we provide a “go-to” introduction to SRM analyses and propose SRM_R ( https://davidakenny.shinyapps.io/SRM_R/ ), an accessible and user-friendly, web-based application for SRM analyses. The basic steps of conducting SRM analyses in the app are illustrated with a sample dataset of 47 teams, 228 members, and 884 dyadic observations, using the participants’ ratings of the advice-seeking behavior of their fellow employees.

Advancing Reproducibility and Accountability of Unsupervised Machine Learning in Text Mining: Importance of Transparency in Reporting Preprocessing and Algorithm Selection

Abstract: Machine learning (ML) enables the analysis of large datasets for pattern discovery. ML methods and the standards for their use have recently attracted increasing attention in organizational research; recent accounts have raised awareness of the importance of transparent ML reporting practices, especially considering the influence of preprocessing and algorithm choice on analytical results. However, efforts made thus far to advance the quality of ML research have failed to consider the special methodological requirements of unsupervised machine learning (UML) separate from the more common supervised machine learning (SML). We confronted these issues by studying a common organizational research dataset of unstructured text and discovered interpretability and representativeness trade-offs between combinations of preprocessing and UML algorithm choices that jeopardize research reproducibility, accountability, and transparency. We highlight the need for contextual justifications to address such issues and offer principles for assessing the contextual suitability of UML choices in research settings.

Sensitizing Social Interaction with a Mode-Enhanced Transcribing Process

Abstract: Qualitative researchers often work with texts transcribed from social interactions such as interviews, meetings, and presentations. However, how we make sense of such data to generate promising cues for further analysis is rarely discussed. This article proposes mode-enhanced transcription as a tool for sensitizing social interaction data, defined as a process in which researchers attune their attention to the dynamic interplay of verbal and nonverbal features, expressions, and acts when transcribing and proofreading professional transcripts. Two scenarios for using mode-enhanced transcription are introduced: sensitizing previously collected data and engaging with modes purposefully. Their implications for research focus, data collection, and data analysis are discussed based on a demonstration of the process with a previously collected dataset and an illustrative review of published articles that display mode-enhanced excerpts. The article outlines the benefits and further considerations of using mode-enhanced transcription as a sensitizing tool.

Assessment of Path Model Fit: Evidence of Effectiveness and Recommendations for use of the RMSEA-P

Abstract: We review the development of path model fit measures for latent variable models and highlight how they are different from global fit measures. Next, we consider findings from two published simulation articles that reach different conclusions about the effectiveness of one path model fit measure (RMSEA-P). We then report the results of a new simulation study aimed at resolving the questions of whether and how the RMSEA-P should be used by organizational researchers. These results show that the RMSEA-P and its confidence interval is very effective with multiple indicator models at identifying misspecifications across large and small sample sizes and is effective at identifying true models at moderate to large sample sizes. We conclude with recommendations for how the RMSEA-P can be incorporated along with other information into model evaluation.