Launch the model, not several compilers

Goal: keep one global spectral problem while changing its ownership. Use the same source, SDK and default input as the serial run:

openpfc run allen_cahn.cpp input.json --sdk /path/to/openpfc-sdk --ranks=2 --launcher=mpirun

Invoke openpfc once outside an MPI launch. It compiles once and launches the requested model processes. On Slurm, request the required resources and follow the HPC guide. Keep the global N, K, physical domain, time step and final time fixed for this comparison.

Real inboxes and complex outboxes are different spaces

A finite-difference rank needs neighbouring physical values in a halo. A distributed FFT instead redistributes data while transforming along global axes. Its communication is not a one-cell neighbour exchange. The exact transposes and process grids belong to the FFT plan.

In this example:

Quantity Global shape Count at N=32
Real input/output grid N × 4 × 4 512 real values
x-compressed r2c grid (N/2+1) × 4 × 4 272 complex values
Retained physical representation (k,0,0), −K ≤ k ≤ K 15 signed modes at K=7

The retained representation, full allocated spectrum and sampled physical grid therefore have three different sizes. Do not size a Fourier vector from the real field length, and do not infer a coefficient’s global mode from its rank-local array index alone.

The program asks the FFT for both bounds and sizes:

fft.size_inbox();
fft.get_inbox_bounds();
fft.size_outbox();
fft.get_outbox_bounds();

SpectralCPUStack constructs its field from the FFT inbox. Spectral iteration uses the actual outbox and the global grid geometry. Bounds are inclusive; the local buffer is x-fastest under the CPU iterator’s contract. Code that assumes identical inbox and outbox boxes can appear to work on one rank and fail after redistribution.

Inspect the actual recorded boxes

The following table is extracted from layout_rank_R.json in the recorded one-rank and two-rank attempts. It reports plan output, not a universal decomposition direction or a promise of another backend’s layout.

Run Rank Real inbox: low → high Complex outbox: low → high
1 rank(s) 0 [0, 0, 0] → [31, 3, 3] [0, 0, 0] → [16, 3, 3]
2 rank(s) 0 [0, 0, 0] → [15, 3, 3] [0, 0, 0] → [16, 1, 3]
2 rank(s) 1 [16, 0, 0] → [31, 3, 3] [0, 2, 0] → [16, 3, 3]

Download the complete layout records and compare them with your own run. N or rank count can change a plan, and another transform/backend may choose a different distribution.

Compare complete global objects

Use the downloaded verifier with both attempt directories:

python3 verify.py input.json \
  /path/to/serial-attempt/results \
  /path/to/two-rank-attempt/results

The verifier reads every rank’s field and mode file, maps entries by global coordinate, and checks complete, nonoverlapping ownership. CSV row order, local vector position and rank-zero output alone cannot establish equality of the global objects.

The recorded comparison reports the maximum field difference across rank counts and the separate \(N=32\) versus \(N=64\), fixed-K coefficient difference. Small global-diagnostic differences can arise from floating-point reduction order. The native comparison uses an absolute \(2\times10^{-11}\) rank-agreement threshold; exact bitwise identity is not required across machines or plans.

Keep normalization global

For raw forward output \(F\), normalized coefficients are \(c=F/V\) with \(V=N\times4\times4\). This is the same V on every rank. A coefficient’s amplitude must not depend on how many cells its owner happens to hold. Global norms likewise sum each physical cell once; this spectral field has no halo cells to include or exclude.

The r2c Parseval check weights interior x modes by 2 and the DC/Nyquist planes by 1, summing all owned complex cells. It provides another check on normalization and completeness without comparing only two FFT runs.

Checkpoint: rank count changes a reported amplitude

The reconstructed field is unchanged, but dividing the raw DC coefficient by each rank’s local inbox size doubles the reported mean on two ranks. Is the solver wrong?

The diagnostic normalization is wrong. Raw FFT output uses a global sum; the inverse divides by the global V. Normalize the owned DC coefficient by that same V, independently of its owner’s local inbox size. The coefficient can belong to only one rank, so first identify its actual outbox owner rather than assuming it is stored on rank zero.

Explain the complete method

Before marking the course complete, explain these without opening the answers:

  1. Which coefficient convention does the mathematics use, and which does the native forward FFT return?
  2. Why can an initially resolved mode create an unresolved cubic product?
  3. Why does the cubic require 4K<N rather than the quadratic condition 3K<N?
  4. Which part of ETD2 is exact, and which part needs time refinement?
  5. What does the independent fixed-K reference check, and what does it leave unresolved?
  6. Why can the real and Fourier vectors have different owners and lengths?

You have followed a nonlinear equation through a representation, an alias-free product, a time method and an actual MPI program. For larger models, keep those same boundaries visible. Continue with the application gallery, OpenPFC spectral stack and spectral examples sequence.

Course map Previous: independent verification Run on a cluster