OpenPFC / Make the result portable between ranks

Many local bricks, one binary field

Each process owns a piece of the field. The file describes the global array.

Includes the model, inputs, required headers and a short run guide.

The old write_results demonstration assigned four hard-coded values to each of exactly two ranks. Here the decomposition supplies every rank’s brick, so the same program works on one or multiple processes. An affine field makes the expected file values unambiguous:

\[ u(x,y,z)=1+x+2y+3z. \tag{1}\]

The default grid has size \(8\times4\times2\), origin \((-1,2,3)\) and spacing \((.25,.5,.75)\). Nonzero geometry catches a common mistake: writing local array indices as though they were physical coordinates.

Tell MPI-IO where the local brick belongs

pfc::BinaryWriter writer("results/field_%04d.bin", comm);
writer.set_domain(pfc::domain::get_size(domain), field.box().size,
                  field.box().low);
(void)writer.write(5, field.vec());
pfc::BinaryReader reader(comm);
reader.set_domain(pfc::domain::get_size(domain), field.box().size,
                  field.box().low);
std::vector<double> local_copy(field.size());
(void)reader.read("results/field_0005.bin", local_copy);
if (local_copy != field.vec())
  throw std::runtime_error("Local binary read changed a value");

All processes in the communicator call write and read. The global dimensions, local dimensions and global offsets define the MPI file view. The owned buffer is compact, so its length is exactly the local brick product. Ghost padding is not part of this raw file.

The filename’s increment is 5, hence results/field_0005.bin. Writing a file does not store the field’s geometry or a complete simulation checkpoint. Keep that metadata separately, or use the snapshot series or checkpoint service as appropriate.

Inspect the global layout independently

The example additionally reads the complete file on rank zero using an ordinary C++ binary stream:

std::ifstream file("results/field_0005.bin", std::ios::binary);
const auto size = pfc::domain::get_size(domain);
std::vector<double> restored(size[0] * size[1] * size[2]);
file.read(reinterpret_cast<char *>(restored.data()),
          static_cast<std::streamsize>(restored.size() * sizeof(double)));
if (!file || file.peek() != std::char_traits<char>::eof())
  throw std::runtime_error("Unexpected binary file length");
double error = 0;
for (int k = 0; k < size[2]; ++k)
  for (int j = 0; j < size[1]; ++j)
    for (int i = 0; i < size[0]; ++i) {
      const auto x = pfc::domain::to_coords(domain, {i, j, k});
      error =
          std::max(error, std::abs(restored[i + size[0] * (j + size[1] * k)] -
                                   output_lesson::value(x[0], x[1], x[2])));
    }
if (error != 0) throw std::runtime_error("Binary round trip changed a value");

The flat offset is \(i+N_x(j+N_yk)\): x varies fastest. This independent read checks both values and ordering. The file contains exactly \(N_xN_yN_z\) native double values. It has no header, endianness tag or geometry block; do not infer cross-platform portability from exact round trips on the same platform.

Run on a different partition

openpfc run model.cpp input.json --source-root .. --launcher=none
openpfc run model.cpp input.json --source-root .. --ranks=2 --launcher=mpirun

Both runs should print that all values were restored exactly. The tests also decode the raw file independently in Python and compare every global sample with Equation 1. Change n and predict the file size before running. The writer and reader are destroyed before the MPI environment is finalized.

Keep the model, input and geometry helper together.