Keep the global problem fixed
Goal: explain why halos are needed and verify that changing rank count preserves the global solution. Begin with the checked one-rank model. Keep its source and input unchanged. Invoke the CLI once, letting it launch model processes:
openpfc run model.cpp input.json --sdk /path/to/openpfc-sdk --ranks=2 --launcher=mpirunUse your SDK prefix and the same compiler/MPI environment. On a cluster, request resources first; the HPC guide explains Slurm launchers and storage. Do not nest this compilation workflow under mpirun -n 2 openpfc …: that would start two competing CLI workflows.
This is a correctness comparison with the global cells, dt and steps unchanged. It is not a performance benchmark. Rank count changes ownership and communication, not the physical wavelength or final time.
A stencil crosses an ownership boundary
Consider a one-dimensional illustration with 16 periodic cells split into two contiguous pieces. This picture explains neighbour data; the starter itself decomposes a 3D cube and need not choose this split direction.
At each update: exchange current neighbours → evaluate owned stencils → advance owned values.
Rank 0 can evaluate the stencil at cell 3 from local owned data. At cell 7, it needs cell 8 from rank 1. At cell 0, periodicity requires cell 15. Ghosts are copies of another rank’s owned values; they are not extra global cells. They become stale after an update and must be refreshed before the next stencil evaluation.
A width-one halo suffices for this nearest-neighbour Laplacian. A wider stencil can require a wider halo. The requirement comes from the operator, not simply from the number of MPI ranks. A missing or late exchange can produce an ownership-boundary error while most of a plot still looks plausible.
Count owned cells once
The starter computes local sums over owned values and calls MPI_Allreduce to obtain global sums. Its diagnostics divide by the total global cell count:
\[\overline u=\frac{\sum_{\text{ranks}}\sum_{\text{owned}}u}{N^3},\qquad \mathrm{l2}=\sqrt{\frac{\sum_{\text{ranks}}\sum_{\text{owned}}u^2}{N^3}}.\]
Including ghosts would count some cells twice. Writing only rank zero’s field would omit the other owners. For this starter, each rank writes results/field_rank_R.csv; global diagnostics and the summary are written by rank zero. The checker reads all rank files, keys values by global coordinate, and requires exactly \(N^3\) unique owned coordinates.
Compare coordinates, not CSV row positions
Download check-results.py if you have not already. Give it the two results directories printed by the one-rank and two-rank attempts:
python3 ../check-results.py input.json \
/path/to/one-rank-attempt/results \
/path/to/two-rank-attempt/resultsFile order and local row order can change with decomposition. The checker compares the same global coordinate in each run, checks both against the independent discrete-mode formula, and reports the maximum difference.
Our recorded comparison found 4,096 unique owned cells in each run and a maximum cross-rank field difference of 0.0 for this build and input. The actual diagnostics may differ in their last digits because floating-point reduction order can change. Exact bitwise equality is an observation here, not a portability promise. The checker uses an absolute tolerance of \(2\times10^{-11}\).
If the comparison fails, ask a narrower question
| Symptom | Check first |
|---|---|
| Missing or duplicate coordinates | Owned extents, global coordinates, and whether ghosts were written |
| Error concentrated near rank boundaries | Halo width, neighbours, periodic wrap and exchange before stencil use |
| Correct field but slightly different global norm | Floating-point reduction order and tolerance |
| Both runs agree but both miss the PDE answer | Units, spatial/time discretization and the independent reference |
| Same appearance but wrong amplitude | Compare numerical values, not just colour limits or a full-field RMS |
Agreement across rank counts tests decomposition and communication for this case. It does not establish strong scaling, GPU-aware MPI, a different operator, or correctness for every decomposition. Add a new independent reference and targeted checks when changing the problem.
Checkpoint: why does the mean alone miss a halo bug?
A two-rank run preserves the mean but disagrees with the serial field near a boundary. Is it correct?
No. Positive and negative local errors can cancel in the global sum. Mean conservation is one scalar constraint on thousands of values. Inspect the coordinate-wise difference, verify that each cell has one owner and that each stencil sees current neighbour values. Then compare the amplitude and full field with the discrete-mode formula. Fixing the mean by adjusting values would conceal the error rather than repair communication.
Explain the complete calculation
Before marking the course complete, try to explain these without opening the answers:
- Why can modes 3 and 13 have identical samples on 16 points?
- Why can a Fourier derivative be accurate while forward Euler is unstable?
- Which quantities must stay fixed in a controlled refinement study?
- Why are the discrete and continuous decay amplitudes different?
- Why must ghost cells be excluded from global counts and norms?
- What does a one-rank/two-rank agreement establish, and what remains untested?
You now have a repeatable way to connect an equation, a discretization, a working program and evidence about its output. To continue, study nonlinear spectral stacks, Krylov solvers and multigrid, or run an application example. Those topics need their own assumptions and validation, just as diffusion did.