A load-balancing algorithm for a parallel electromagnetic particle-in-cell code

This paper proposes a new method for Particle-in-Cell (PIC) simulations which aims at achieving both good load balancing and scalability so as to be efficiently executed on distributed memory systems. This method, named OhHelp, simply and equally partitions the space domain where charged particles reside and assigns each partitioned subdomain to each computation node for scalable simulation with respect to the size of the domain. Load balancing and thus the scalability in terms of the number of particles are accomplished by making each node help another heavily loaded node which deputes a part of particles in its subdomain and replicated subdomain field data to its helpers. The OhHelp load balancer monitors particle movements through subdomain boundaries to check if the helpand-helpers configuration keeps good load balancing and, when unacceptable imbalance is found, dynamically reconfigures it to regain perfect balancing. The efficiency and scalability of OhHelp are confirmed through our experiment with a production-level full-3D plasma simulator and with uniform and non-uniform particle distributions. As a result, we found 256-core parallel simulations, including an extremely imbalanced setting to pack all the particles in a small region, exert 159-190 speedup compared to sequential execution.

/pdf/ohhelp-a-scalable-domain-decomposing-dynamic-load-balancing-3oo7hdh4w8.pdf

OhHelp: a scalable domain-decomposing dynamic load balancing for particle-in-cell simulations

https://ir.nctu.edu.tw:443/bitstream/11536/4429/1/000248161700038.pdf

Development of a Parallelized 3D Electrostatic PIC-FEM Code and Its Applications

Summary form given only, as follows. The Z accelerator at Sandia National Laboratories generates current pulses rising up to ∼20 MA in ∼100 ns. Inside the insulator stack at r∼2 m, power flows inward along four conical, vacuum, magnetically-insulated transmission lines (MITL's) to the post-hole convolute at r∼10 cm. The convolute adds the current in the four feed MITL's into the inner MITL, which delivers it to the load. The convolute has magnetic field nulls, which could potentially result in large electron losses. To improve our understanding of the electron flow, we are using the 3-D, electromagnetic, PIC code QUICKSILVER. The simulations accurately model the Z vacuum section out to r∼25 cm. Even with fairly coarse zoning, we still need ∼750,000 cells for a 12-post convolute. The complex cathode structure has ∼20,000 particle-emitting cells. We handle electron motion in the wide-ranging magnetic field strength (0-200 T) with a subcycling model. The field solver uses a fixed time step, but each particle is pushed s times per field solver step, with the integer s dynamically chosen so that the particle makes at least 6 steps per orbit. We have done several series of idealized simulations using a fixed load impedance and 20 ns rise-time pulse. Locations of major electron losses are in qualitative agreement with damage to the Z hardware. We are now doing these simulations on a massively-parallel platform, typically 100 processors on Janus (ASCI Red), making more realistic, time-accurate (400,000 time steps) simulations feasible. A z-pinch load model is under development. In preliminary simulations using only electron emission from the cathode, convolute electron losses are much smaller than experiment. A likely source of the discrepancy is the effect of plasmas created at hot spots on the anode from electron deposition.

3-D PIC simulations of electron flow in the vacuum power flow section of the Z accelerator

Summary form only given In the past, there have been very few studies concentrating on developing dynamic load-balancing technique for particle-based PIC/MCC code The only exception to the authors' best knowledge is the work by Seidel et al, in which the method dynamically repartitioned the computational domain and intended to balance the workload among processors under the framework of structured mesh However, there are several disadvantages in using structured mesh For example, it becomes rather cumbersome to treat the boundary conditions if complicated geometry is involved In the current paper, method of dynamic domain decomposition (DDD), based on multi-level graph-partitioning technique, for a parallel 3D electrostatic PIC/MCC code that utilized unstructured tetrahedral mesh is proposed and verified This parallel code is designed to run on memory-distributed machine with standard MPI protocol In a typical particle simulation, for example, DSMC, the computational weight (~actual computational time) is generally in proportion to the number of particles In PIC/MCC method, weight for each graph vertex (or cell center) is instead a combination of particle computational weight and cell computational weight The computational weight of the former results from time required for the particle push and Monte Carlo collisions, while the latter results from the time required for solving the Poisson's equation A preliminary simulation can determine the computational weights of particle and cell, respectively A quasi-1D argon gas discharge with application of 1000 V difference at 42 mtorr is used as the test case for studying the performance of DDD Strategy of deciding when to repartition the domain and resulting parallel performance is discussed in the meeting In addition, the contribution to the total computational time from actual computation, communication and repartition is also discussed by emphasizing their impact to the resulting parallel performance

Dynamic Domain Decomposition for a Parallel 3-D Electrostatic PIC/MCC Code using Unstructured Mesh

Summary form only given. QUICKSILVER is a 3-D electromagnetic particle-in-cell simulation code developed and used at Sandia to model relativistic charged particle transport. It was originally written for shared-memory, multi-processor supercomputers such as the Cray X/MP. A new parallel version of QUICKSILVER has been developed to enable large-scale simulations to be efficiently run on massively-parallel distributed memory supercomputers with thousands of processors, such as the Intel Tflops and Cplant machines at Sandia. The new parallel code implements all the features of the original QUICKSILVER and can be run on any platform that supports the message-passing interface (MPI) standard as well as on single-processor workstations. The original QUICKSILVER code was based on a multiple-block grid, which provided a natural strategy for extending the code to partition a simulation among multiple processors. By adding the automated capability to divide QUICKSILVER's existing blocks into sub-blocks and then distribute those sub-blocks among processors, a simulation's spatial domain can be easily and efficiently partitioned. Based upon this partitioning scheme as well as QUICKSILVER's existing particle-handling infrastructure, an algorithm has been developed for dynamically rebalancing the particle workload on a timestep-by-timestep basis that has proven to be very efficient. This paper will elaborate on the strategies used and describe the algorithms developed to parallelize and dynamically load-balance the code. Results of several benchmark simulations will be presented that illustrate the code's performance and parallel efficiency for a wide variety of simulation conditions. These calculations have as many as 10/sup 8/ grid cells and 10/sup 9/ particles and were run on thousands of processors.

G.R. Montry

Papers

Dynamic load-balancing for a parallel electromagnetic particle-in-cell code