Geometry / Ownership

One box, many owners

A distributed array begins with an agreement about who owns each value.

Includes the model, inputs, required headers and a short run guide.

A temperature field may contain millions of values, but a rank needs to store only its own part. The decomposition describes that ownership. It does not itself move field values, calculate a derivative, or fill a halo. Keeping those responsibilities distinct makes parallel code easier to reason about.

This refactors example 02. Its original two-subdomain demonstration is now driven by the actual communicator size, so changing the launch changes the decomposition being demonstrated.

What must a partition preserve?

The global index set is

\[ I=\{0,\ldots,15\}\times\{0,\ldots,7\}\times\{0,\ldots,3\}. \tag{1}\]

Let \(I_r\) be the owned box of rank \(r\). A correct partition satisfies \(\bigcup_r I_r=I\) and \(I_r\cap I_s=\varnothing\) for \(r\ne s\). In particular,

\[ \sum_r |I_r|=16\cdot8\cdot4=512. \tag{2}\]

The count is necessary but insufficient: an overlap could conceal a missing region. The numerical check therefore enumerates all the small boxes and verifies the union and pairwise uniqueness as well.

Describe ownership in C++

const auto domain = pfc::domain::create({16, 8, 4});
const auto partition = pfc::decomposition::create(domain, comm.size());
const auto box = pfc::decomposition::local_box(partition, comm.rank());
const long long local_cells =
    static_cast<long long>(box.size[0]) * box.size[1] * box.size[2];
long long global_cells = 0;
MPI_Allreduce(&local_cells, &global_cells, 1, MPI_LONG_LONG, MPI_SUM, comm);

Every rank knows the same global domain and constructs the same partition. local_box selects its own index box. The reduction checks Equation 2 without collecting a global field.

The rest of model.cpp gathers only six integers per rank to write results/decomposition.csv. Each row contains a rank, the three global lower indices, and the three local sizes. An upper index is lower + size - 1: OpenPFC index boxes have inclusive bounds.

Change the number of owners

From this directory with a provisioned SDK:

openpfc run model.cpp input.json --ranks=2 --launcher=mpirun

The input is an empty JSON object; the small domain is deliberately stated in the source. Compare one, two, and four ranks. The shapes and number of rows change, while the union remains the same 512 cells. Do not hard-code the assumption that a particular rank always owns the left half.

Next, change the global shape from {16,8,4} to {8,16,4} and update the source’s independent count. Predict which local shapes will change. The partitioning policy is a numerical/runtime choice; the physical problem should not depend on it.

Where this agreement ends

The example check launches one and two ranks and validates every owned global index. It does not claim optimal communication volume, load balance for arbitrary physics, or support for more ranks than meaningful cells. FFT algorithms may redistribute values between a real-space inbox and a Fourier-space outbox; the FFT layout lesson explains why those are separate ownership descriptions.

For exact API contracts see the decomposition header. Continue with indices and memory offsets. The complete C++ model and starting input are available with this lesson.