Library basics: domain, decomposition, and FFT

This tutorial introduces the lowest-level path through OpenPFC:

Domain → Decomposition → FFT → physics step → simulation driver

You will first build a minimal downstream consumer, then connect its concepts to the repository’s runnable MPI and FFT examples.

Complete Start here before this tutorial. It verifies that OpenPFC itself builds and that the MPI and HeFFTe stack works.

What each object owns

Object

Responsibility

Domain

Global grid size, spacing, origin, and periodicity

Decomposition

Partition of the global domain across MPI ranks

FFT

Distributed real-to-complex and complex-to-real transforms

Physics object

Application-specific fields and callable step(t)

pfc::sim::SimulationDriver

Clock, time loop, modifiers, and result writers

The full package boundaries are described in Architecture. Exact declarations belong in the generated API reference.

1. Create a downstream project

Create a new directory outside the OpenPFC source tree:

openpfc-domain-check/
├── CMakeLists.txt
└── main.cpp

Use the installed package target exercised by the packaging smoke test:

cmake_minimum_required(VERSION 3.21)
project(openpfc_domain_check LANGUAGES C CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(OpenPFC REQUIRED)

add_executable(domain_check main.cpp)
target_link_libraries(domain_check PRIVATE OpenPFC::openpfc)

The project enables C as well as C++ because the installed package resolves MPI components that include an MPI C target.

2. Construct a domain

// main.cpp
#include <array>
#include <iostream>

#include <openpfc/kernel/data/domain.hpp>

int main() {
  const auto domain = pfc::domain::create({32, 16, 8});
  const auto size = pfc::domain::get_size(domain);

  std::cout << "grid = " << size[0] << " x " << size[1] << " x "
            << size[2] << '\n';
  return 0;
}

domain::create validates that every grid dimension and spacing component is positive. Prefer the factory over aggregate initialization so invalid input fails at the construction boundary.

Configure and run:

cmake -S . -B build \
  -DCMAKE_PREFIX_PATH=/path/to/openpfc/install
cmake --build build -j"$(nproc)"
./build/domain_check

Expected output:

grid = 32 x 16 x 8

When CMake cannot find the package, set CMAKE_PREFIX_PATH to the installation prefix or set OpenPFC_DIR to the directory containing OpenPFCConfig.cmake. See Troubleshooting.

3. Add MPI decomposition

A Domain describes the global grid. Decomposition describes how that grid is split across ranks for distributed transforms and communication.

The maintained example is:

examples/geometry/decomposition/model.cpp

It constructs the objects with the current factories:

const auto domain = pfc::domain::create({32, 4, 4});
const auto decomposition = pfc::decomposition::create(domain, comm_size);

Build the examples in the OpenPFC tree and run:

./scripts/build.sh --cpu
mpirun -n 2 ./builds/release/examples/02_domain_decomposition

On Tohtori, load openmpi/5.0.10 in the launch shell after the build; the script loads modules only in its own process. On LUMI use the scheduled build and launch route instead of running MPI examples on the login node.

The decomposition is determined by the global grid and communicator size. Code that owns MPI must initialize it before creating communication-dependent objects and finalize it only after those objects are no longer used.

4. Perform a distributed FFT

The next maintained example is:

examples/fourier/round-trip/model.cpp

Its essential construction sequence is:

const auto domain = pfc::domain::create({8, 1, 1});
const auto decomposition = pfc::decomposition::create(domain, num_procs);
auto fft = pfc::fft::create(decomposition);

std::vector<double> input(fft.size_inbox());
std::vector<std::complex<double>> output(fft.size_outbox());
fft.forward(input, output);

Build and run it first with one rank:

./scripts/build.sh --cpu
mpirun -n 1 ./builds/release/examples/03_parallel_fft

Then run with a rank count supported by the example dimensions and your launcher allocation:

mpirun -n 2 ./builds/release/examples/03_parallel_fft

The real-space inbox and Fourier-space outbox can have different local extents. Always allocate buffers from fft.size_inbox() and fft.size_outbox() rather than deriving local sizes from the global domain manually.

5. Move from transforms to a model

A physics object owns or borrows the fields and supplies a callable step(t). The current framework uses pfc::sim::run with a clock and driver; it has no Model or Simulator base class. A typical spectral step contains:

  1. a forward transform from real space;

  2. multiplication by linear operators in Fourier space;

  3. optional inverse transforms for nonlinear terms;

  4. an update of the accepted field state.

Do not build that logic from this page alone. Continue with the maintained example sequence:

  1. examples/heat/implicit-euler/model.cpp;

  2. examples/heat/driver-hooks/model.cpp;

  3. examples/phase-fields/cahn-hilliard/model.cpp.

The guided reading is in Spectral examples sequence.

Maintenance rule

The runnable files under examples/ are the source of truth for code-level examples. This tutorial explains their sequence and contracts; it should not copy complete implementations that can drift independently from the build.