Binary field I/O specification (MPI-IO)

This document describes the raw binary files produced by pfc::BinaryWriter (including the JSON-driven path in add_result_writers_from_json). Those headerless MPI-IO bricks remain the scheduled ResultsWriter / BinaryWriter format for periodic field dumps and post-processing when you are not using VTK or PNG.

Durable versioned restart bundles use pfc::checkpoint::publish_checkpoint_directory instead — a directory with metadata.json plus accepted field bricks, published atomically. See docs/development/checkpoint_publish.md. Do not treat a lone headerless BinaryWriter file as the full checkpoint / restart contract.

Implementation references: include/openpfc/frontend/io/binary_writer.hpp, include/openpfc/kernel/simulation/binary_reader.hpp, include/openpfc/frontend/utils/utils.hpp (format_with_number).

File contents

Property

Value

Layout

A single global 3D array in Fortran (column-major) order (MPI_ORDER_FORTRAN in MPI_Type_create_subarray).

Element type

Real fields: double (MPI_DOUBLE). Complex fields: std::complex<double> (MPI_DOUBLE_COMPLEX). Each complex element occupies sizeof(std::complex<double>) bytes (two native doubles) in the raw payload.

MPI filetype / etype

BinaryWriter and BinaryReader build the MPI filetype subarray from the same element type used as the MPI_File_set_view etype. Real paths use MPI_DOUBLE; complex paths use MPI_DOUBLE_COMPLEX. The filetype is rebuilt if the same writer/reader instance later switches element type after one set_domain().

Byte order

Native ("native" in MPI_File_set_view) — endianness matches the machine that wrote the file.

Header / magic

None. The file is only the raw payload for the MPI file view (no metadata block, no version tag).

Per-rank data

Each MPI rank writes its local brick; together the ranks cover the global grid without overlap. The view is built from (global_size, local_size, local_offset) passed to set_domain().

Domain geometry

BinaryWriter::set_domain and BinaryReader::set_domain fail closed before marking the domain valid: global and local dimensions must be positive, offsets non-negative, and offset[i] + local[i] <= global[i] on every axis. Invalid triples throw std::invalid_argument via pfc::mpi::validate_subarray_domain (domain_geometry.hpp).

Local buffer contract

Each rank’s buffer element count must equal the local brick product local_x * local_y * local_z from set_domain. BinaryWriter::write and BinaryReader::read compute that product with overflow-safe multiplication (pfc::mpi::checked_local_extent_product) and, on disagreement, throw std::runtime_error after communicator-wide agreement (MPI_Allreduce) and before MPI_File_open / MPI_File_write_all / MPI_File_read_all.

Local count limit

Classic MPI-IO count parameters are int. Each rank’s local element count (data.size()) must be ≤ INT_MAX. BinaryWriter and BinaryReader fail closed via pfc::mpi::expect_mpi_io_count (include/openpfc/kernel/mpi/mpi_io_helpers.hpp), throwing std::overflow_error before MPI_File_open / MPI_File_write_all / MPI_File_read_all — never a silent static_cast<int> truncation.

Each write(increment, field) call:

  1. Builds the output filename from the template and increment (see below).

  2. Opens the file (truncate), sets the MPI-IO file view, performs MPI_File_write_all, closes the file.

  3. Is collective over the writer’s communicator (default MPI_COMM_WORLD): every rank in that communicator must participate in write() with a consistent domain layout, or the job can deadlock.

Filename template and increment

The JSON fields[].data string is passed to BinaryWriter as the filename template.

  • If the string contains %, it is passed to printf-style formatting with the integer increment supplied by the simulator (see simulation_wiring_writers.hpp and BinaryWriter::write_mpi_binary).
    Examples: ./psi_%d.bin, ./data/u_%04d.bin.

  • If there is no %, the same path is used on every write (overwrites each time).

The increment value is advanced by the simulator according to configuration (see app_pipeline.md and the simulator section in JSON).

Reading back (BinaryReader)

BinaryReader uses the same Fortran-ordered 3D subarray view and MPI_File_read_all at displacement 0.

API

Element type

BinaryReader::read(filename, Field&) / RealField

MPI_DOUBLE

BinaryReader::read(filename, ComplexField&)

MPI_DOUBLE_COMPLEX

Implications:

  • Files written as pure double real fields with the same (global, local, offset) decomposition can be read back for restart or FileReader initial conditions.

  • Files written as std::complex<double> round-trip through BinaryReader::read(..., ComplexField&) with the same decomposition.

  • Match communicator, decomposition, and datatype when reading: a file written on N ranks is not automatically loadable on a different N without re-decomposition logic outside this low-level format. Do not read a complex file with the real read overload (or vice versa).

JSON configuration surface

When saveat > 0 and fields is present, add_result_writers_from_json registers one BinaryWriter per fields[] entry:

  • name — field label used by the simulator when dispatching the writer.

  • data — filename template as above.

Rank 0 may create parent directories for data. See io_results.md and configuration.md.

Post-processing without OpenPFC

Because there is no file header, external tools need out-of-band metadata: global Lx, Ly, Lz, dtype (float64 or complex-of-two-float64), Fortran ordering, and how ranks map to subdomains if you concatenate manually. In practice, teams either:

  • Read with OpenPFC (BinaryReader or existing tooling), or

  • Record metadata alongside runs (JSON/YAML sidecar), or

  • Use VTK export for visualization (VTKWriter; see tutorials/vtk_paraview_workflow.md).

Longer offline sketch (metadata, NumPy layout): postprocess_binary_fields.md.

See also