A Parallel P^3M Code for Very Large Scale Cosmological Simulations

We have developed a parallel Particle-Particle, Particle-Mesh (P^3M) simulation code for the T3E well suited to studying the time evolution of systems of particles interacting via gravity and gas forces in cosmological contexts. The parallel code is based upon the public-domain serial Adaptive P^3M code of Couchman et al(1). The algorithm resolves gravitational forces into a long range component computed by discretizing the mass distribution and solving Poisson's equation on a grid using an FFT convolution method, and a short range component computed by direct force summation for sufficiently close particle pairs. The code consists primarily of a particle-particle computation parallelized by domain decomposition over blocks of neighbor-cells, a more regular mesh calculation distributed in planes along one dimension, and several transformations between the two distributions. Great care was taken throughout to make optimal use of the available memory, so that the current implementation is capable of simulating systems approaching 10^9 particles using a 1024^3 mesh for the long range force computation. These are thus among the largest N-body simulations ever carried out. We discuss these memory optimizations as well as those motivated by computational performance. Results from production runs have been very encouraging, and even prior to the implimentation of the full adaptive scheme the code has been used effectively for simulations in which the particle distribution becomes highly clustered as well as for other non-uniform systems of astrophysical interest.