The case for thoroughly testing complex system dynamic models
01 Jan 2005-
TL;DR: The study finds that rather than focusing primarily on sensitivity testing, modelers should consider other types of model tests such as extreme condition tests and family member tests, fully supporting the recommendations of the experts.
Abstract: The Case for Thoroughly Testing Complex System Dynamics Models Wayne Wakeland and Megan Hoarfrost 1 Systems Science Ph.D. Program, Portland State University 2 Stanford University 1 P.O. Box 751, Portland, OR 97207 (503) 725-4975(v) (503) 725-8489 (f) wakeland@pdx.edu Abstract In order to determine whether model testing is as useful as suggested by modeling experts, the full battery of model tests recommended by Forrester, Senge, Sterman, and others was applied retrospectively to a complex previously-published system dynamics model. The time required to carry out each type of test was captured, and the benefits that resulted from applying each test was determined subjectively. The resulting benefit to cost ratios are reported. These ratios suggest that rather than focusing primarily on sensitivity testing, modelers should consider other types of model tests such as extreme condition tests and family member tests. The study also finds that all of the different kinds of tests were either moderately useful or very useful--fully supporting the recommendations of the experts.
Citations
More filters
17 Jan 2008
TL;DR: The EcoPOST framework is introduced, which utilizes evaluation models to describe the interplay of technological, organizational, and projectspecific evaluation factors, and simulation concepts to unfold the dynamic behavior of PAIS engineering projects.
Abstract: Providing effective IT support for business processes has become crucial for enterprises to stay competitive in their market. Business processes must be defined, implemented, enacted, monitored, and continuously adapted to changing situations. Process life cycle support and continuous process improvement become critical success factors in contemporary and future enterprise computing.
In this context, process-aware information systems (PAIS) adopt a key role. Thereby, organization-specific and generic process support systems are distinguished. In the former case, the PAIS is build "from scratch" and incorporates organization-specific information about the structure and processes to be supported. In the latter case, the PAIS does not
contain any information about the structure and processes of a particular organization. Instead, an organization needs to configure the PAIS by specifying processes, organizational entities, and business objects.
To enable the realization of PAIS, numerous process support paradigms, process modeling standards, and business process management tools have been introduced. The application of these approaches in PAIS engineering projects is not only influenced by technological, but also by organizational and project-specific factors. Between these factors there exist numerous causal dependencies, which, in turn, often lead to complex and unexpected
effects in PAIS engineering projects. In particular, the costs of PAIS engineering projects are significantly influenced by these causal dependencies.
What is therefore needed is a comprehensive approach enabling PAIS engineers to systematically investigate these causal dependencies as well as their impact on the costs of PAIS engineering projects. Existing economic-driven IT evaluation and software cost estimation approaches, however, are unable to take into account causal dependencies and resulting effects.
In response, this thesis introduces the EcoPOST framework. This framework utilizes evaluation models to describe the interplay of technological, organizational, and projectspecific evaluation factors, and simulation concepts to unfold the dynamic behavior of PAIS engineering projects. In this context, the EcoPOST framework also supports the reuse of evaluation models based on a library of generic, predefined evaluation patterns and also
provides governing guidelines (e.g., model design guidelines) which enhance the transfer of the EcoPOST framework into practice. Tool support is available as well.
Finally, we present the results of two online surveys, three case studies, and one controlled software experiment. Based on these empirical and experimental research activities, we are able to validate evaluation concepts underlying the EcoPOST framework and additionally demonstrate its practical applicability.
79 citations
Cites background or methods from "The case for thoroughly testing com..."
...In [212], Wakeland and Hoarfrost apply model tests as described by Sterman [189] to a complex SD model, measure the time required to carry out each type of test, and finally relate these costs to the gained (and subjectively measured) benefits resulting from each test....
[...]
..., on its useful domain of applicability [212]....
[...]
TL;DR: Kenya as a country should consider adoption of secondary /catch up vaccination as an immediate measure to curb cervical cancer followed by primary vaccination of pre-adolescent girls after a simulation model predicted that catch up vaccination had the greatest impact in reducing the prevalence of cervical cancer.
Abstract: This paper presents a simulation model for evaluating the possible effects of a screening and vaccination campaign against Human Papillomavirus [HPV] in Kenya. A System Dynamics model was developed using the iThink™ computer simulation package. The model was based on data extracted from epidemiological, demographic and published research and where data was not available, expert opinion was sought. The deterministic model stratified the population by vaccination status, screening status and HPV infection status. The model was simulated to estimate outputs for the next 50 years from 2011. Cost Utility indicators of Disability Adjusted Life Years (DALYs) and cost per averted DALY were used for economic evaluation. The model predicted that catch up vaccination had the greatest impact in reducing the prevalence of cervical cancer. This was followed by Primary vaccination, with early detection through Screening having the lowest impact of the three choices of interventions in respect of averted cases of cervical cancer and DALY estimates. Kenya as a country should consider adoption of secondary /catch up vaccination as an immediate measure to curb cervical cancer followed by primary vaccination of pre-adolescent girls. Screening should be a complementary measure(s). This model provides a policy decision support vehicle that can allow for choice between different interventions based on their expected outcomes. It also allows modification to accommodate new research results and information to assess the clinical impact of different policies and interventions in cervical cancer management in Kenya.
32 citations
Cites background or methods from "The case for thoroughly testing com..."
...The female population was grouped into given age groups; The females were divided into seven age groups: [0–5], [4-9], [10-14], [15-24], [25-44], [45–64] and over 65 years based on age groups of published data on cervical cancer [3]....
[...]
...As the girls progress through the aging chain, they are eligible for catch up or secondary vaccination between the ages of [10-44] years....
[...]
...Validation and verification of the SD Model The model was validated through animation, face validity, predictive validation and extreme condition tests [36]....
[...]
TL;DR: Verification and validation of the DEVS models in DEVS-Suite environment are discussed and particular attention is paid to reliability and maintainability in view of the state-of-the-art network simulator ns-2.
11 citations
TL;DR: In this paper, the authors combine system dynamics (SD) and agent-based modeling (ABM) to build an integrated model, which can gain better understanding of MLs' behavioral diversities, reveal the associated impacts and improve project management.
Abstract: Employing multi-type laborers (MLs) is common in multinational and cross-culture projects (MPCs). Different attributes of MLs can lead to uncertain and dynamic laborer behaviors (i.e. behavioral diversities), which may cause project deviations. Previous studies do not consider the uncertainties or dynamics of behaviors adequately or they only provide general suggestions. The purpose of this paper is to combine system dynamics (SD) and agent-based modeling (ABM) to build an integrated model. The proposed ABM-SD can gain better understanding of MLs’ behavioral diversities, reveal the associated impacts and improve project management.,Based on extensively review in construction labor management and computer simulation, architecture is built to depict the relationships between the affecting factors of MLs’ behaviors, MLs’ behavioral diversities and project performance. Second, conceptual structures of the ABM-SD model are developed. Third, methods to implement the model in practice are introduced, focusing on data collection and model structure adjustment. Finally, the model is tested in a case study.,Different ML groups have distinctive behaviors which constantly change through interactions between MLs, engineers and external environment. Inadequate consideration of the diversities can result in inaccurate estimation of productivity, work quality and absenteeism, causing severe project deviations such as schedule delay, cost overrun and high absenteeism. On the other hand, using the ABM-SD model, the root causes of project deviations are analyzed from the perspective of MLs’ behavioral diversities and the optimization of labor management can significantly improve project performance.,This paper supplements previous studies because the ABM-SD model takes fully use of the strength of simulation of solving uncertain and dynamic problems and combines both qualitative and quantitative findings in existing studies of labor management. Besides, the ABM-SD model is also a practical management tool to better monitor laborer behaviors and forecast the impacts. The limitation is mainly about the small scale of the case study. However, the ABM-SD model already demonstrates the mechanism about how MLs’ different behaviors affect a project, which fulfill the aim of the study.,The ABM-SD model can simulate MLs’ behavioral diversities and produce reliable estimations of project performance. It also allows to optimize management plans. Furthermore, The ABM-SD model is adjustable based on specific project conditions, which make it applicable for different tasks, different laborer compositions and even different projects. Thus, the ABM-SD model can be a practical tool for engineers in MCPs.,SD and ABM are applied to study behaviors with well-known benefits in both separated and integrated manner. However, few studies use the approach to investigate MLs’ behaviors in MCPs. Hence, the proposed ABM-SD model is an original attempt to improve the laborer management level in MCPs.
9 citations
01 Jan 2012
TL;DR: It is found that validation is hardly described as an integrated part of any participatory modeling approach, and the process of validation and the end-result is little reported on in any case study.
Abstract: In this paper we report if and how validation is integrated in the different participatory approaches in System Dynamics. After this we present an analysis of 86 case studies to create a picture of how validation is conducted and reported in the field. We found that validation is hardly described as an integrated part of any participatory modeling approach. Furthermore, the process of validation and the end-result is little reported on in any case study. We believe that there is a significant difference between validation in non-participatory and participatory approaches due to the social aspect. In this context we derive three questions as a requirement for validation. The aim is balancing individual mental models, group mental models and logic and data. Based on the tension between these three factors we developed a preliminary approach to participatory structure validation. This procedure is focused on generating a productive amount of cognitive conflict in order to confront the structure of the model with all available information. The outcome is increased structure validity, documentation of limitations and increased understanding of the reasoning underlying the model's structure. Word count: 5292 (without references, appendices and abstract)
7 citations
References
More filters
Book•
01 Jan 2000
TL;DR: Requirements for the effective use of system dynamics are discussed and a successful application to a difficult business issue is illustrated.
Abstract: Introduction Part I. Perspective and Process 1. Learning In and About Complex Systems 2. System Dynamics In Action 3. The Modeling Process 4. Structure and Behavior of Dynamic Systems Part II. Tools for Systems Thinking 5. Causal Loop Diagrams 6. Stocks and Flows 7. Dynamics of Stocks and Flows 8. Closing the Loop: Dynamics of Simple Structures Part III. The Dynamics of Growth 9. S-Shaped Growth: Epidemics, Innovation Diffusion, and the Growth of New Products 10. Path Dependence and Positive Feedback Part IV. Tools for Modeling Dynamic Systems 11. Delays 12. Coflows and Aging Chains 13. Modeling Decision Making 14. Forming Non-linear Relationships Part V. Instability and Oscillation 15. Modeling Human Behavior: Bounded Rationality or Rational Expectations? 16. Forecasts and Fudge Factors: Modeling Expectation Formation 17. Supply Chains and the Origin of Oscillations 18. Managing Supply Chains in Manufacturing 19. The Labor Supply Chain and the Origin of Business Cycles 20. The Invisible Hand Sometimes Shakes: Commodity Cycles Part VI. Validation and Model Testing 21. Truth and Beauty Part VII. Commencement 22. Challenges for the Future Appendix A: Numerical Integration Appendix B: Noise References Index
6,808 citations
01 Jan 2000
5,457 citations
TL;DR: This book is most obviously relevant to practitioners who already have some experience of multiagency facilitation, but might also serve as an introduction to working in this arena, if carefully supplemented with further reading and exploration of the topics it covers.
Abstract: (2002). Business Dynamics—Systems Thinking and Modeling for a Complex World. Journal of the Operational Research Society: Vol. 53, No. 4, pp. 472-473.
2,977 citations
01 May 2002
TL;DR: The field of system dynamics, created at MIT in the 1950s by Jay Forrester, is designed to help us learn about the structure and dynamics of complex systems in which we are embedded, design high-leverage policies for sustained improvement, and catalyze successful implementation and change as discussed by the authors.
Abstract: Todays problems often arise as unintended consequences of yesterdays solutions. Social systems often suffer from policy resistance, the tendency for well-intentioned interventions to be defeated by the response of the system to the intervention itself. The field of system dynamics, created at MIT in the 1950s by Jay Forrester, is designed to help us learn about the structure and dynamics of the complex systems in which we are embedded, design high-leverage policies for sustained improvement, and catalyze successful implementation and change. Drawing on engineering control theory and the modern theory of nonlinear dynamical systems, system dynamics often involves the development of formal models andmanagement flight simulators� to capture complex dynamics, and to create an environment for learning and policy design. Unlikepureengineering problemsif any existhuman systems present unique challenges, including long time horizons, issues that cross disciplinary boundaries, the need to develop reliable models of human behavior, and the great difficulty of experimental testing. Successful change in social systems also requires the active participation of a wide range of people in the modeling and policy design process, people who often lack technical training. In this paper I discuss requirements for the effective use of system dynamics and illustrate with a successful application to a difficult business issue.
1,555 citations
TL;DR: This paper focuses on the formal aspects of validation and presents a taxonomy of various aspects and steps of formal model validation, including structure-oriented behavior tests, which seem to be the most promising direction for research on model validation.
Abstract: Model validation constitutes a very important step in system dynamics methodology. Yet, both published and informal evidence indicates that there has been little effort in system dynamics community explicitly devoted to model validity and validation. Validation is a prolonged and complicated process, involving both formal/quantitative tools and informal/qualitative ones. This paper focuses on the formal aspects of validation and presents a taxonomy of various aspects and steps of formal model validation. First, there is a very brief discussion of the philosophical issues involved in model validation, followed by a flowchart that describes the logical sequence in which various validation activities must be carried out. The crucial nature of structure validity in system dynamics (causal-descriptive) models is emphasized. Then examples are given of specific validity tests used in each of the three major stages of model validation: Structural tests, structure-oriented behavior tests and behavior pattern tests. Also discussed is if and to what extent statistical significance tests can be used in model validation. Among the three validation stages, the special importance of structure-oriented behavior tests is emphasized. These are strong behavior tests that can provide information on potential structure flaws. Since structure-oriented behavior tests combine the strength of structural orientation with the advantage of being quantifiable, they seem to be the most promising direction for research on model validation.
1,406 citations