Know what a rank owns

MPI starts separate processes called ranks. Each rank owns part of a global grid. A local finite-difference stencil needs values just across that part’s boundary; halo exchange supplies those neighbour values before the stencil reads them. A higher-order stencil generally needs a wider halo.

A distributed Fourier transform has another communication pattern. It rearranges data between ranks to transform different directions. An FFT method’s arithmetic accuracy does not remove the cost of those exchanges. Read the halo explanation and spectral stack for OpenPFC’s implementation contracts.

Establish the small answer first

Start with a checked one-rank model. Save its global diagnostics. Then compare with two ranks on the same global grid:

openpfc run model.cpp input.json --ranks=2 --launcher=mpirun

The CLI runs once; the launcher starts the model ranks. Do not put this command inside a second mpirun. For the starter, check conservation and decay against the serial calculation. Parallel reductions can change the last digits, so bitwise identity is not the general acceptance criterion.

Changing rank count at fixed global problem size is a strong-scaling comparison. Increasing the global problem with rank count is a weak-scaling comparison. State which you mean, and include communication and output time when they are part of the work you need to perform.

Put the work inside an allocation

Use the site’s permitted environment to compile before launching a large job. Keep the compiler, MPI and SDK consistent between compilation and execution. A source-model run can compile if its build is missing, so ensure its cache and source are visible to the job.

The following is the shape of a small Slurm CPU job. Add the account, partition and module setup required by your site, and adjust the resource request to the actual model. Submit the file with sbatch; run it from the model directory on a filesystem visible to all ranks.

#!/bin/bash
#SBATCH --job-name=openpfc-first-model
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:10:00

# Load the site's matching compiler, MPI and OpenPFC modules here.

openpfc run model.cpp input.json --ranks=2 --launcher=srun

An allocation is not a launcher: it reserves resources. srun starts job steps inside those resources. Open MPI and Slurm need matching PMI/PMIx support for direct srun launch. The Slurm tutorial and site guides own the concrete launch settings.

Choose the site and device path

Path Read before launching
Tohtori CPU or NVIDIA Site installation and modules
LUMI CPU or AMD GPU LUMI installation and storage
Choosing a GPU application CPU/GPU decision guide and GPU app tutorial
Packaging a compiled CPU model Run and bundle

The native source-model CLI currently qualifies CPU execution. GPU applications use their supported CUDA/HIP builds and matching HeFFTe and MPI stack. A CPU doctor result or a working CPU image does not establish GPU or multi-node fabric qualification.

Make output part of the plan

Use shared project or scratch storage for large fields. A rank-local path may not be visible to another rank or to a later analysis job. Preserve the input, code revision and environment together with the diagnostics needed to interpret a result. Prefer compact summaries while learning; write large fields when they answer a question.

See the MPI-I/O checklist, output format and operator guide for production details. Use the profiling guide to find the cost that matters before tuning a launcher or FFT plan.

Explore measured performance Return to the numerical methods