https://espace.library.uq.edu.au/view/UQ:381881/UQ381881_OA.pdf

Recent evidence on the effectiveness of group model building

Systems science methods such as dynamic simulation modelling are well suited to address questions about public health policy as they consider the complexity, context and dynamic nature of system-wide behaviours. Advances in technology have led to increased accessibility and interest in systems methods to address complex health policy issues. However, the involvement of policy decision makers in health-related simulation model development has been lacking. Where end-users have been included, there has been limited examination of their experience of the participatory modelling process and their views about the utility of the findings. This paper reports the experience of end-user decision makers, including senior public health policy makers and health service providers, who participated in three participatory simulation modelling for health policy case studies (alcohol related harm, childhood obesity prevention, diabetes in pregnancy), and their perceptions of the value and efficacy of this method in an applied health sector context. Semi-structured interviews were conducted with end-user participants from three participatory simulation modelling case studies in Australian real-world policy settings. Interviewees were employees of government agencies with jurisdiction over policy and program decisions and were purposively selected to include perspectives at different stages of model development. The ‘co-production’ aspect of the participatory approach was highly valued. It was reported as an essential component of building understanding of the modelling process, and thus trust in the model and its outputs as a decision-support tool. The unique benefits of simulation modelling included its capacity to explore interactions of risk factors and combined interventions, and the impact of scaling up interventions. Participants also valued simulating new interventions prior to implementation in the real world, and the comprehensive mapping of evidence and its gaps to prioritise future research. The participatory aspect of simulation modelling was time and resource intensive and therefore most suited to high priority complex topics with contested options for intervening. These findings highlight the value of a participatory approach to dynamic simulation modelling to support its utility in applied health policy settings.

/pdf/decision-makers-experience-of-participatory-dynamic-3jhc9y6zxx.pdf

Decision makers' experience of participatory dynamic simulation modelling: methods for public health policy.

Evidence-based decision-making is an important foundation for health policy and service planning decisions, yet there remain challenges in ensuring that the many forms of available evidence are considered when decisions are being made. Mobilising knowledge for policy and practice is an emergent process, and one that is highly relational, often messy and profoundly context dependent. Systems approaches, such as dynamic simulation modelling can be used to examine both complex health issues and the context in which they are embedded, and to develop decision support tools. This paper reports on the novel use of participatory simulation modelling as a knowledge mobilisation tool in Australian real-world policy settings. We describe how this approach combined systems science methodology and some of the core elements of knowledge mobilisation best practice. We describe the strategies adopted in three case studies to address both technical and socio-political issues, and compile the experiential lessons derived. Finally, we consider the implications of these knowledge mobilisation case studies and provide evidence for the feasibility of this approach in policy development settings. Participatory dynamic simulation modelling builds on contemporary knowledge mobilisation approaches for health stakeholders to collaborate and explore policy and health service scenarios for priority public health topics. The participatory methods place the decision-maker at the centre of the process and embed deliberative methods and co-production of knowledge. The simulation models function as health policy and programme dynamic decision support tools that integrate diverse forms of evidence, including research evidence, expert knowledge and localised contextual information. Further research is underway to determine the impact of these methods on health service decision-making.

https://researchonline.nd.edu.au/cgi/viewcontent.cgi?article=1862&context=med_article

Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling.

Evidence-informed decision-making is essential to ensure that health programs and services are effective and offer value for money; however, barriers to the use of evidence persist. Emerging systems science approaches and advances in technology are providing new methods and tools to facilitate evidence-based decision-making. Simulation modelling offers a unique tool for synthesising and leveraging existing evidence, data and expert local knowledge to examine, in a robust, low risk and low cost way, the likely impact of alternative policy and service provision scenarios. This case study will evaluate participatory simulation modelling to inform the prevention and management of gestational diabetes mellitus (GDM). The risks associated with GDM are well recognised; however, debate remains regarding diagnostic thresholds and whether screening and treatment to reduce maternal glucose levels reduce the associated risks. A diagnosis of GDM may provide a leverage point for multidisciplinary lifestyle modification interventions. This research will apply and evaluate a simulation modelling approach to understand the complex interrelation of factors that drive GDM rates, test options for screening and interventions, and optimise the use of evidence to inform policy and program decision-making. The study design will use mixed methods to achieve the objectives. Policy, clinical practice and research experts will work collaboratively to develop, test and validate a simulation model of GDM in the Australian Capital Territory (ACT). The model will be applied to support evidence-informed policy dialogues with diverse stakeholders for the management of GDM in the ACT. Qualitative methods will be used to evaluate simulation modelling as an evidence synthesis tool to support evidence-based decision-making. Interviews and analysis of workshop recordings will focus on the participants’ engagement in the modelling process; perceived value of the participatory process, perceived commitment, influence and confidence of stakeholders in implementing policy and program decisions identified in the modelling process; and the impact of the process in terms of policy and program change. The study will generate empirical evidence on the feasibility and potential value of simulation modelling to support knowledge mobilisation and consensus building in health settings.

/pdf/simulation-modelling-as-a-tool-for-knowledge-mobilisation-in-410hqghd3i.pdf

Simulation modelling as a tool for knowledge mobilisation in health policy settings: a case study protocol.

Background System science approaches are increasingly used to explore complex public health problems. Quantitative methods, such as participatory dynamic simulation modelling, can mobilise knowledge to inform health policy decisions. However, the analytic and practical steps required to turn collaboratively developed, qualitative system maps into rigorous and policy-relevant quantified dynamic simulation models are not well described. This paper reports on the processes, interactions and decisions that occurred at the interface between modellers and end-user participants in an applied health sector case study focusing on diabetes in pregnancy. Methods An analysis was conducted using qualitative data from a participatory dynamic simulation modelling case study in an Australian health policy setting. Recordings of participatory model development workshops and subsequent meetings were analysed and triangulated with field notes and other written records of discussions and decisions. Case study vignettes were collated to illustrate the deliberations and decisions made throughout the model development process. Results The key analytic objectives and decision-making processes included: defining the model scope; analysing and refining the model structure to maximise local relevance and utility; reviewing and incorporating evidence to inform model parameters and assumptions; focusing the model on priority policy questions; communicating results and applying the models to policy processes. These stages did not occur sequentially; the model development was cyclical and iterative with decisions being re-visited and refined throughout the process. Storytelling was an effective strategy to both communicate and resolve concerns about the model logic and structure, and to communicate the outputs of the model to a broader audience. Conclusion The in-depth analysis reported here examined the application of participatory modelling methods to move beyond qualitative conceptual mapping to the development of a rigorously quantified and policy relevant, complex dynamic simulation model. The analytic objectives and decision-making themes identified provide guidance for interpreting, understanding and reporting future participatory modelling projects and methods.

/pdf/turning-conceptual-systems-maps-into-dynamic-simulation-52u9sn3e3w.pdf

Turning conceptual systems maps into dynamic simulation models: An Australian case study for diabetes in pregnancy.

It is assumed that more education leads to better understanding of complex systems. Some researchers, however, find indications that simple mechanisms like stocks and flows are not well understood even by people who have passed higher education. In this paper, we test people’s understanding of complex systems with the widely studied stock-and-flow (SF) tasks. SF tasks assess people’s understanding of the interplay between stocks and flows. We investigate SF failure of domain experts and novices in different knowledge domains. In particular, we compare performance on the original study’s bathtub task with the square wave pattern with two alternative cover stories from the engineering and business domains on different groups of business and engineering students from different semesters. Further, we show that, while engineering students perform better than business students, with progressing in higher education, students may lose the capability of dealing with simple SF tasks. We thus find hints on deformation professionelle in higher education.

Bathtub Dynamics Revisited: An Examination of Déformation Professionelle in Higher Education

In this paper we report if and how validation is integrated in the different participatory approaches in System Dynamics. After this we present an analysis of 86 case studies to create a picture of how validation is conducted and reported in the field. We found that validation is hardly described as an integrated part of any participatory modeling approach. Furthermore, the process of validation and the end-result is little reported on in any case study. We believe that there is a significant difference between validation in non-participatory and participatory approaches due to the social aspect. In this context we derive three questions as a requirement for validation. The aim is balancing individual mental models, group mental models and logic and data. Based on the tension between these three factors we developed a preliminary approach to participatory structure validation. This procedure is focused on generating a productive amount of cognitive conflict in order to confront the structure of the model with all available information. The outcome is increased structure validity, documentation of limitations and increased understanding of the reasoning underlying the model's structure. Word count: 5292 (without references, appendices and abstract)

Group Model Validation

Organizations are driven by social interactions such as decision processes, negotiations or operations. Those interactions are composed of multiple simultaneous, dynamically evolving processes with several different agents. However, researchers in the field of management science traditionally focus on aggregated characteristics and assume equilibria, thus correctly neglecting the level of individual agents. Thus, most research methods in management science are based on cross-sectional data as well as stable and predictable events.

Simulation as Research Method: Modeling Social Interactions in Management Science

SF-failure, the inability of people to correctly determine the behavior of simple stock and flow structures is subject of a long research stream. Reasons for SF-failure can be attributed to different reasons, one of them being lacking domain specific experience, thus familiarity with the problem context. In this article we present a continuation of an experiment to examine the role of educational background in SFperformance. We base the question set on the Bathtub Dynamics tasks introduced by Booth Sweeney and Sterman (2000) and vary the cover stories. In this paper we describe how we developed and tested a new cover story for the engineering domain and implemented the recommendations from a prior study. We test three sets of questions with engineering students which enables us to compare the results to a previous study in which we tested the questions with business students. Results mainly support our hypothesis that context familiarity increases SF-performance. With our findings we further develop the methodology of the research on SF-failure.

Bathtub Dynamics Revisited: Exploring the engineering domain

In prior studies, people’s poor performance in dealing with basic systems concepts has been ascribed to different causes. While results indicate that, among other things, domain specific experience and familiarity with the problem context play a role in this stock-flow-(SF-)performance, this has not yet been fully clarified. In this article, we present an experiment that examines the role of educational background in SF-performance. We hypothesize that SF-performance increases when the problem context is embedded in the problem solver’s knowledge domain, indicated by educational background. Using the square wave pattern and the sawtooth pattern tasks from the initial study by Booth Sweeney and Sterman (2000), we design two additional cover stories for the former, the Vehicle story from the engineering domain and the Application story from the business domain, next to the original Bathtub story. We then test the three sets of questions on business students. Results mainly support our hypothesis. Interestingly, participants even do better on a more complex behavioral pattern from their knowledge domain than on a simpler pattern from more distant domains. Although these findings have to be confirmed by further studies, they contribute both to the methodology of future surveys and the context familiarity discussion.

Roland Maximilian Happach

Papers

Bathtub Dynamics Revisited: An Examination of Déformation Professionelle in Higher Education

Group Model Validation

Simulation as Research Method: Modeling Social Interactions in Management Science

Bathtub Dynamics Revisited: Exploring the engineering domain

Bathtub Dynamics Revisited: Does Educational Background Matter?