Installing OpenPFC

This document is the supported source build guide. On Tohtori, the current site Open MPI 5.0.10 module uses GCC 15.2.0. Other compilers may work, but OpenPFC, HeFFTe, and MPI must use one matching compiler/MPI stack.

Choose an installation route

If your site already provides an OpenPFC installation, load its module and matching compiler/MPI environment. The Tohtori and LUMI guides list the site modules. The first-model walkthrough then checks the SDK with openpfc doctor, creates a CPU diffusion model and shows how to inspect its output. This requires an installation containing the native CLI commands described here; older release modules may not include them.

For a source checkout, use scripts/build.sh for routine builds and tests. On Tohtori, run ./scripts/build.sh --machine=tohtori --cpu. On an ordinary workstation with the dependencies below installed, use ./scripts/build.sh --machine=portable --cpu; this preserves your environment and package discovery. On LUMI, follow the site’s scheduled build instructions and keep build trees on flash outside the clone. Run ./scripts/build.sh --help for options. The remaining sections provide manual dependency and toolchain details.

Reference stack (at a glance)

Piece

Documented / exercised baseline

CMake

3.15+

GCC

15.2.x with the current Tohtori Open MPI 5.0.10 module

Open MPI

5.0.10 (Slurm srun needs PMI/PMIx); OPENMPI_ROOT overrides IDE toolchain paths if your site differs

HeFFTe

2.4.1 (separate install per CPU/CUDA/ROCm variant)

Native CLI cryptography

OpenSSL Crypto 1.1 or newer, with development headers

The native openpfc CLI requires OpenSSL development files (libssl-dev on Debian/Ubuntu, openssl-devel on RPM-based systems). It uses Crypto for SHA-256 and does not link the simulation runtime. Python remains optional tooling for scientific reference checks, documentation and build scripts; the installed CLI and generated model projects do not require it.

One-page matrix (optional / QA tooling, doc check scripts): docs/dependency_matrix.md.

For common failures (mpi.h, wrong compiler in cache, HeFFTe not found, link errors involving libheffte.so and GLIBCXX_*), see docs/troubleshooting.md.

After a successful install, use Start here for your first model and result files. For source examples, applications and CMake integration, continue with Quick start. The learning paths and documentation index help you choose the next step.

Contributing (builds, tests, documentation, changelog notes): CONTRIBUTING.md.

MPI: OpenMPI from modules — do not improvise with another stack

This guide’s source of truth for MPI is Open MPI from environment modules (module load openmpi after module load gcc/…), as in §1 below. That matches how OpenPFC and HeFFTe are meant to be built together on clusters and many dev machines.

Recommended on Slurm clusters: Open MPI 5.0.10 from modules (e.g. module load openmpi/5.0.10), built or packaged with Slurm PMI/PMIx support, so srun direct launch works. If you see OMPI was not built with SLURM's PMI support, switch to this stack and rebuild HeFFTe and OpenPFC against it.

IDE / preset pins: cmake/toolchains/tohtori-gcc11-openmpi.cmake defaults to /share/apps/OpenMPI/5.0.10 when OPENMPI_ROOT is unset (matching module show openmpi/5.0.10 on typical nodes). For another install or module layout, set OPENMPI_ROOT to that prefix before CMake.

Keep one compiler and MPI stack per build: use the compiler loaded by the Open MPI module for HeFFTe and OpenPFC, then use the same Open MPI for mpirun/srun at run time.

  • Run with the same MPI you built with: after loading modules, which mpirun and which mpicxx must come from the same Open MPI installation (e.g. both under /share/apps/OpenMPI/5.0.10/bin when using that module). Do not launch binaries with a system MPICH mpirun (e.g. /usr/lib64/mpich/bin/mpirun) if OpenPFC was linked against Open MPI — you will see link warnings about conflicting libmpi.so and runtime errors around MPI_Finalize.

  • Do not point CMake at an arbitrary system MPI just because mpicc exists (for example MPICH under /usr/lib64/mpich on RHEL-style systems). Open MPI and MPICH are different implementations; HeFFTe and OpenPFC must be configured, built, and run against the same MPI. Mixing “HeFFTe built with MPICH” and “OpenPFC expecting Open MPI” (or the reverse) produces confusing link or runtime failures.

  • Do run module load openmpi (or your site’s equivalent name — use module avail openmpi), then confirm you are using that toolchain before building HeFFTe and OpenPFC:

    which mpicc
mpicc --version   # should identify Open MPI / openmpi, not MPICH

  • Bare OS without modules: install Open MPI development packages (not MPICH) if you want to follow this document literally. If your site standardizes on MPICH only, you may still build everything consistently with MPICH — but that is not the combination this file describes step-by-step; keep HeFFTe and OpenPFC on the same mpicc/mpicxx throughout.

2. Other dependencies

  • CMake 3.15+

  • FFTW (development packages) — required for HeFFTe’s CPU backend

  • nlohmann/json — optional system install; otherwise CMake may fetch it during configuration

  • MPI — Required for all supported builds (HeFFTe and the distributed FFT stack depend on it). The documented setup is OpenMPI from modules (see §1 and the MPI callout above). CMake must find MPI_CXX for that MPI — not a random MPICH install earlier on PATH. There is no supported serial-only configuration yet — -DOpenPFC_ENABLE_MPI=OFF is rejected at configure time with a pointer to this list. If mpi.h is missing during configure, see §5.1 (modules + CC/CXX + stale cache).

  • toml++ and Catch2 — if not found on the system, CMake may download them (e.g. when building tests). Ensure network access on first configure, or install/provide packages your site supports.

  • Doxygen — optional; if present, documentation generation is enabled by default (disable with -DOpenPFC_BUILD_DOCUMENTATION=OFF if you do not need it)

Installed consumers of the public settings loader also need toml++. OpenPFC exports that dependency through its CMake target. When the build fetches toml++ 3.4, installation delivers its headers, license and a separate relocatable tomlplusplus CMake package beside OpenPFC. A system-provided toml++ remains an external package discoverable through the consumer’s normal CMake search paths.

3. HeFFTe (required)

Build and install HeFFTe yourself, then point OpenPFC at it with CMAKE_PREFIX_PATH or Heffte_DIR.

If you omit those, CMake still runs find_package(Heffte) first, then probes common install prefixes automatically (see cmake/OpenPFCHeffteHints.cmake: e.g. **$HOME/opt/heffte/*, /opt/heffte/..., /share/apps/heffte/..., Spack EBROOTHEFFTE). The first prefix that contains lib64/cmake/Heffte or lib/cmake/Heffte wins. Add your site’s root to that file if needed.

Install layout: $HOME/opt/heffte/<variant>

Use one install prefix per backend, all under $HOME/opt/heffte/, so you can keep CPU, CUDA, and ROCm builds side by side and wire them in with modules:

Variant

Typical CMAKE_INSTALL_PREFIX

Notes

CPU (FFTW only)

$HOME/opt/heffte/2.4.1-cpu

No GPU backend in HeFFTe

CUDA

$HOME/opt/heffte/2.4.1-cuda

-DHeffte_ENABLE_CUDA=ON, nvcc on PATH when building

ROCm / HIP

$HOME/opt/heffte/2.4.1-rocm

-DHeffte_ENABLE_ROCM=ON (see §8.1)

OpenPFC CPU builds should use CMAKE_PREFIX_PATH (or Heffte_DIR) pointing at 2.4.1-cpu. GPU builds must use the matching -cuda or -rocm install.

Do not unpack or build HeFFTe inside the OpenPFC repository

Keep the OpenPFC clone free of HeFFTe source trees, long-lived build trees, and release tarballs (nothing like heffte-2.4.1/ or v2.4.1.tar.gz next to OpenPFC’s top-level CMakeLists.txt).

  • Download tarballs to $HOME/src, /tmp, or any directory outside the OpenPFC tree.

  • Point -S at the extracted source outside the repo (e.g. $HOME/src/heffte-2.4.1).

  • Point -B at a build directory outside the repo (e.g. $HOME/opt/heffte/build-cpu-2.4.1 or /tmp/heffte-build-cpu-2.4.1).

  • cmake --install into $HOME/opt/heffte/2.4.1-cpu (or -cuda / -rocm).

OpenPFC’s own build-cpu/ / build-gpu/ directories are only for OpenPFC (usually gitignored); do not use the repository root as a dump for HeFFTe sources or tarballs.

GitHub Actions in this repository follow the same policy: workflows download HeFFTe to /tmp, build in a temporary directory there, install to $HOME/opt/heffte/2.4.1-cpu on the runner, and delete the extract — the checked-out OpenPFC tree is not used as a HeFFTe working directory.

Optional: OpenPFC_FETCH_HEFFTE=ON

If no HeFFTe is found, CMake can fetch v2.4.1 via FetchContent into build/_deps/ inside an OpenPFC build directory (FFTW + MPI still required). This is a fallback only: for day-to-day work, prefer installs under $HOME/opt/heffte/<variant> as above. FetchContent must not be used as an excuse to extract HeFFTe into the OpenPFC source tree.

The fallback pins the v2.4.1 source commit and archive checksum and applies OpenPFC’s FFTW real-plan correction (Git is required). Stock HeFFTe 2.4.1 plans with two null pointers, which FFTW treats as an in-place layout, then executes with distinct arrays. On macOS this can corrupt a 512-point transform or crash with a reduced singleton axis. The correction plans with separate temporary arrays sized for the complete strided batch. It uses FFTW_UNALIGNED so later execution need not match the temporary arrays’ alignment; FFTW may consequently use fewer SIMD optimizations. Performance has not been qualified.

An already installed HeFFTe takes precedence and is never modified by OpenPFC. Rebuild it from corrected sources using the manual instructions below. For an explicit fallback build, combine -DOpenPFC_FETCH_HEFFTE=ON with -DCMAKE_DISABLE_FIND_PACKAGE_Heffte=ON.

Optional: Lmod modulefiles ($HOME/privatemodules)

After installing to e.g. $HOME/opt/heffte/2.4.1-cpu, you can add a personal module that prepends CMAKE_PREFIX_PATH** (and anything else your site needs):

#%Module1.0
set root $env(HOME)/opt/heffte/2.4.1-cpu
prepend-path CMAKE_PREFIX_PATH $root

Place the file where Lmod looks (e.g. $HOME/privatemodules/heffte/2.4.1-cpu.lua or .modulepath layout your site uses), then module load heffte/2.4.1-cpu before configuring OpenPFC. Repeat with different paths for 2.4.1-cuda and 2.4.1-rocm.

Build and install HeFFTe manually (outside OpenPFC)

Load Open MPI (which selects its matching GCC) and, for CUDA, cuda first (§1). Use the same CC/CXX as for OpenPFC.

Working copy outside the repo (adjust SRC / BUILD):

export VER=2.4.1
export SRC=$HOME/src/heffte-${VER}
export BUILD=$HOME/opt/heffte/build-cuda-${VER}
mkdir -p "$HOME/src" "$HOME/opt/heffte"
wget -q -O "$HOME/src/v${VER}.tar.gz" \
  https://github.com/icl-utk-edu/heffte/archive/refs/tags/v${VER}.tar.gz
tar xf "$HOME/src/v${VER}.tar.gz" -C "$HOME/src"
export CC=$(which gcc)
export CXX=$(which g++)

From the OpenPFC checkout, apply the bounded correction before configuring HeFFTe (including CPU, CUDA and ROCm variants that enable FFTW):

cmake -DSOURCE_DIR="$SRC" -P cmake/PatchHeffteFftw.cmake

This accepts the original v2.4.1 header or an already corrected copy. It refuses unexpected header contents without overwriting them. Rebuild and reinstall HeFFTe afterward; changing source alone does not update an installed provider.

CUDA + FFTW → install to $HOME/opt/heffte/2.4.1-cuda:

cmake -S "$SRC" -B "$BUILD" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="$CC" \
  -DCMAKE_CXX_COMPILER="$CXX" \
  -DCMAKE_INSTALL_PREFIX=$HOME/opt/heffte/2.4.1-cuda \
  -DHeffte_ENABLE_FFTW=ON \
  -DHeffte_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=80
cmake --build "$BUILD" -j"$(nproc)"
cmake --install "$BUILD"

Replace 80 with your GPU’s compute capability, use native with CMake 3.24+, or pass several architectures (e.g. 75;80;86).

CPU only → install to $HOME/opt/heffte/2.4.1-cpu (use a separate BUILD path, e.g. $HOME/opt/heffte/build-cpu-${VER}):

cmake -S "$SRC" -B "$BUILD" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="$CC" \
  -DCMAKE_CXX_COMPILER="$CXX" \
  -DCMAKE_INSTALL_PREFIX=$HOME/opt/heffte/2.4.1-cpu \
  -DHeffte_ENABLE_FFTW=ON \
  -DHeffte_ENABLE_CUDA=OFF
cmake --build "$BUILD" -j"$(nproc)"
cmake --install "$BUILD"

Point OpenPFC at the install (see lib vs lib64 on your system):

export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cpu:$CMAKE_PREFIX_PATH
# or, explicitly:
# cmake ... -DHeffte_DIR=$HOME/opt/heffte/2.4.1-cpu/lib64/cmake/Heffte

Use the directory that contains HeffteConfig.cmake.

4. Get OpenPFC sources

git clone https://github.com/VTT-ProperTune/OpenPFC.git
cd OpenPFC

5. Configure and build OpenPFC (CPU)

5.1 If cmake fails immediately — what is actually blocking?

OpenPFC’s CMake is not broken on a correctly prepared machine. Configuration almost always fails for one of these environment reasons (all fixable without changing the repo):

Symptom

Cause

Fix (see earlier sections)

HeFFTe was not found / fatal error mentioning Heffte

No install on CMAKE_PREFIX_PATH, and OpenPFC_FETCH_HEFFTE=OFF (default).

Build and install HeFFTe to $HOME/opt/heffte/2.4.1-cpu (§3), then export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cpu:$CMAKE_PREFIX_PATH or pass -DHeffte_DIR=$HOME/opt/heffte/2.4.1-cpu/lib64/cmake/Heffte (use the directory that contains HeffteConfig.cmake — sometimes lib/cmake/Heffte). Alternatively, -DOpenPFC_FETCH_HEFFTE=ON (still requires FFTW + MPI; artifacts only under build/_deps/, not the source tree).

mpi.h: No such file or directory during MPI probe, or MPI from the wrong vendor

Shell never ran module load openmpi (§1), or CMake used /usr/bin/cc without OpenMPI include/lib flags.

Load GCC + OpenMPI modules first, then export CC=$(which gcc) and export CXX=$(which g++), then configure — or pass -DCMAKE_C_COMPILER=... -DCMAKE_CXX_COMPILER=... explicitly. Confirm mpicc --version reports Open MPI, not MPICH (see top callout).

Wrong compiler / MPI after you “fixed” modules

Stale CMakeCache.txt in build/ still points at old paths.

Remove the build directory (rm -rf build-cpu) and re-run cmake from a shell after modules are loaded.

Agents and IDEs: Do not point CMake at a hard-coded MPI under /share/apps/... unless that matches this machine’s documented modules. Follow §1 and §3 every time; the supported recipe is module-loaded OpenMPI + HeFFTe under $HOME/opt/heffte/<variant>.

With modules loaded (§1), CMAKE_PREFIX_PATH including $HOME/opt/heffte/2.4.1-cpu (or a module load that sets it — §3), pass the same compilers as for HeFFTe:

export CC=$(which gcc)
export CXX=$(which g++)
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cpu:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -S . -B build-cpu
cmake --build build-cpu -j"$(nproc)"

If you omit -DCMAKE_BUILD_TYPE, single-config generators default to RelWithDebInfo (cmake/ProjectSetup.cmake), not Debug. Multi-config generators ignore CMAKE_BUILD_TYPE; CMake presets may set the type explicitly. The examples below still pass -DCMAKE_BUILD_TYPE=Release when a fully optimized, non-debug-info build is intended.

Minimal configure (optional): By default, OpenPFC may enable code coverage and (if Doxygen is installed) documentation. For binaries only:

export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cpu:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -DOpenPFC_ENABLE_CODE_COVERAGE=OFF \
      -DOpenPFC_BUILD_DOCUMENTATION=OFF \
      -S . -B build-cpu

5.2 CMake cache variables (quick reference)

These switches are defined under cmake/ (see BuildOptions.cmake, Dependencies.cmake, CUDASupport.cmake, HIPSupport.cmake, LibraryConfiguration.cmake, CodeCoverage.cmake, ProjectSetup.cmake, CompilerSettings.cmake). Use -DNAME=ON|OFF (or STRING where noted) on the cmake command line.

Cache variable

Typical use

CMAKE_BUILD_TYPE

Single-config default is RelWithDebInfo if unset (cmake/ProjectSetup.cmake). Pass Release or Debug to override.

OpenPFC_ENABLE_MPI

Must stay ON for supported builds (see §2).

OpenPFC_ENABLE_HEFFTE

HeFFTe FFT stack (ON by default). OFF is an FD-only / kernel-only build: HeFFTe is not required; spectral sources, apps, and examples are skipped. Catch2 TUs that include fft_fftw.hpp are skipped; FD/halo tests still build.

OpenPFC_FETCH_HEFFTE

Fetch HeFFTe into build/_deps/ if no install is found.

OpenPFC_ENABLE_HDF5

Optional HDF5 for profiling dumps and JSON "writer": "hdf5" (/field + XDMF; parallel HDF5 when the library was built with MPI, otherwise multi-rank gather to rank 0). OFF by default.

OpenPFC_ENABLE_CUDA / OpenPFC_ENABLE_HIP

GPU toolkit backends. Finite-difference / kernel-only GPU apps can build with the toolkit plus CPU HeFFTe; spectral GPU FFT paths additionally require matching CUDA/ROCm HeFFTe (§3, §6–§8).

OpenPFC_ENABLE_CUDA_SPECTRAL / OpenPFC_ENABLE_HIP_SPECTRAL

Derived from the found HeFFTe package (CUDA/ROCm backend present). Gates spectral GPU FFT sources and spectral GPU apps; finite-difference GPU targets can still build with CPU-only HeFFTe.

OpenPFC_MPI_CUDA_AWARE / OpenPFC_MPI_HIP_AWARE

GPU-aware MPI when supported (defaults ON when CUDA/HIP is enabled).

OpenPFC_BUILD_APPS, OpenPFC_BUILD_EXAMPLES, OpenPFC_BUILD_TESTS, OpenPFC_BUILD_BENCHMARKS

Scope of targets built alongside openpfc (benchmarks are slow).

OpenPFC_BUILD_DOCUMENTATION

Doxygen docs (ON if doxygen is found).

OpenPFC_ENABLE_CODE_COVERAGE

gcov coverage (ON by default; disable on clusters without lcov).

OpenPFC_DEVELOPMENT

Development extras such as compile_commands.json (OFF by default).

OpenPFC_PROFILING_LEVEL

0, 1, or 2 compile-time profiling macros (cmake/LibraryConfiguration.cmake).

OpenPFC_ENABLE_NAN_CHECK

Optional NaN guards (OFF by default). Debug builds enable NaN checks automatically; set ON to force them in non-Debug builds (cmake/CompilerSettings.cmake).

BUILD_SHARED_LIBS

Build openpfc as shared (OFF = static, default).

5.2.1 GPU-aware MPI (configure vs runtime)

When CUDA is enabled and OpenPFC_MPI_CUDA_AWARE=ON, CMake builds and runs a tiny probe (cmake/openpfc_mpix_cuda_probe.c, linked against MPI::MPI_C from find_package(MPI REQUIRED COMPONENTS C CXX)) using try_run, which exercises MPIX_Query_cuda_support() under MPI_Init / MPI_Finalize. If the probe exits non-zero, configuration prints a warning—use a CUDA-aware Open MPI build, or set -DOpenPFC_MPI_CUDA_AWARE=OFF and reconfigure.

Other MPI stacks (for example Cray MPICH on LUMI-G) are not probed this way. OpenPFC_MPI_HIP_AWARE is also not configure-probed; follow docs/INSTALL.LUMI.md for MPICH_GPU_SUPPORT_ENABLED, and use verify_gpu_aware_mpi (installed under bin/ with HIP builds) for a multi-rank device-buffer smoke test.

Cross-compiling: the runtime part of the CUDA probe is skipped; validate on the target machine instead.

For toolchain-specific examples (cluster CMAKE_TOOLCHAIN_FILE), see CMakePresets.json and cmake/toolchains/.

6. Configure and build OpenPFC (CUDA)

HeFFTe requirement: OpenPFC always needs a HeFFTe package for CPU FFT/decomposition. With OpenPFC_ENABLE_CUDA=ON, CMake splits CUDA support into two levels:

  • CUDA runtime / kernel targets: require the CUDA toolkit. Finite-difference CUDA apps that do not use FFTs (for example wave2d_cuda) can still build when the detected HeFFTe package is CPU-only.

  • CUDA spectral FFT targets: require HeFFTe built with -DHeffte_ENABLE_CUDA=ON, installed under e.g. $HOME/opt/heffte/2.4.1-cuda (§3). If this is missing, CMake prints OpenPFC_ENABLE_CUDA_SPECTRAL = OFF, skips CUDA spectral FFT runtime and spectral CUDA apps/tests, and continues configuring finite-difference CUDA targets.

For a full CUDA spectral build, set CMAKE_PREFIX_PATH to include the CUDA HeFFTe prefix (or load a module that does).

Load the CUDA module so nvcc is on PATH, then configure with explicit host compilers:

export CC=$(which gcc)
export CXX=$(which g++)
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cuda:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -DOpenPFC_ENABLE_CUDA=ON \
      -DCMAKE_CUDA_ARCHITECTURES=80 \
      -S . -B build-gpu
cmake --build build-gpu -j"$(nproc)"

Match CMAKE_CUDA_ARCHITECTURES to your GPU (or native on CMake 3.24+). To find your GPU’s compute capability, run:

nvidia-smi --query-gpu=compute_cap --format=csv,noheader   # e.g. 8.6 → use 86

Minimal configure (optional): As with the CPU build (§5), you can disable optional features:

export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-cuda:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -DOpenPFC_ENABLE_CUDA=ON \
      -DCMAKE_CUDA_ARCHITECTURES=80 \
      -DOpenPFC_ENABLE_CODE_COVERAGE=OFF \
      -DOpenPFC_BUILD_DOCUMENTATION=OFF \
      -S . -B build-gpu

If CUDA is missing: If you pass -DOpenPFC_ENABLE_CUDA=ON but CMake cannot find the CUDA toolkit, configuration still succeeds with a warning and CUDA support is turned off. Always check the configuration summary: OpenPFC_ENABLE_CUDA should be ON for a true GPU build. Fix PATH, CMAKE_CUDA_COMPILER, or CUDAToolkit_ROOT, then reconfigure from a clean build directory.

7. Install prefix (optional)

export CC=$(which gcc)
export CXX=$(which g++)
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -DCMAKE_INSTALL_PREFIX=$HOME/opt/openpfc \
      -S . -B build
cmake --build build -j"$(nproc)"
cmake --install build

(Keep CMAKE_PREFIX_PATH including HeFFTe when running this cmake.)

8. AMD GPU (HIP)

For ROCm / HIP builds, load a recent GCC, OpenMPI, and ROCm before configuring anything (see §1 for compiler notes). Many clusters provide a ROCm module:

module purge
module load openmpi/5.0.10   # also selects its matching GCC on Tohtori
module load rocm/6.4.0       # for GPU — run `module avail rocm` and pick a version

If ROCm is not in a module, ensure its bin directory is on PATH (e.g. export PATH=/opt/rocm-6.4.0/bin:$PATH).

Verify:

g++ --version
mpicc --version
hipcc --version   # after loading ROCm or setting PATH
rocm-smi          # optional: list AMD GPUs

CMAKE_PREFIX_PATH for ROCm: CMake finds HIP via find_package(HIP). If HIP is not found, set CMAKE_PREFIX_PATH to your ROCm installation (e.g. -DCMAKE_PREFIX_PATH=/opt/rocm or /opt/rocm-6.4.0) so that HIPConfig.cmake is found.

HeFFTe requirement (same split as CUDA in §6): OpenPFC always links a HeFFTe package for CPU FFT/decomposition. With OpenPFC_ENABLE_HIP=ON, CMake separates:

  • HIP runtime / kernel targets: need ROCm/HIP. Finite-difference HIP apps (for example kobayashi_fd_hip, wave2d_hip, allen_cahn_hip) can build when the detected HeFFTe install is CPU-only.

  • HIP spectral FFT targets: need HeFFTe built with -DHeffte_ENABLE_ROCM=ON (§8.1). If ROCm HeFFTe is missing, CMake sets OpenPFC_ENABLE_HIP_SPECTRAL = OFF, skips HIP spectral FFT sources and spectral HIP apps/tests, and keeps finite-difference HIP targets enabled.

8.1. Build and install HeFFTe with ROCm

For HIP spectral FFT (HeFFTe with rocFFT), install HeFFTe with -DHeffte_ENABLE_ROCM=ON under $HOME/opt/heffte/2.4.1-rocm (§3). Pure finite-difference HIP builds do not need this variant. Use the same host compilers as for OpenPFC (§1). Do not build inside the OpenPFC repo; use $HOME/src (or similar) for the tarball and source, and a build directory outside the repo.

export VER=2.4.1
export SRC=$HOME/src/heffte-${VER}
export BUILD=$HOME/opt/heffte/build-rocm-${VER}
mkdir -p "$HOME/src" "$HOME/opt/heffte"
wget -q -O "$HOME/src/v${VER}.tar.gz" \
  https://github.com/icl-utk-edu/heffte/archive/refs/tags/v${VER}.tar.gz
tar xf "$HOME/src/v${VER}.tar.gz" -C "$HOME/src"
export CC=$(which gcc)
export CXX=$(which g++)
# Ensure ROCm is on PATH; optionally:
export CMAKE_PREFIX_PATH=/opt/rocm-6.4.0:$CMAKE_PREFIX_PATH
cmake -S "$SRC" -B "$BUILD" -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER="$CC" \
  -DCMAKE_CXX_COMPILER="$CXX" \
  -DCMAKE_INSTALL_PREFIX=$HOME/opt/heffte/2.4.1-rocm \
  -DHeffte_ENABLE_FFTW=ON \
  -DHeffte_ENABLE_ROCM=ON
cmake --build "$BUILD" -j"$(nproc)"
cmake --install "$BUILD"

Optionally set -DCMAKE_HIP_ARCHITECTURES=<arch> to match your GPU (e.g. gfx90a for MI210, gfx1100 for some RDNA3). Use rocm-smi or your vendor docs to get the architecture code.

Point CMake at this installation when building OpenPFC (see §3 for lib vs lib64):

export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-rocm:$CMAKE_PREFIX_PATH

8.2. Configure and build OpenPFC (HIP)

Load the ROCm module (or set PATH) so hipcc and HIP are available. Set CMAKE_PREFIX_PATH to include both the HeFFTe ROCm install and your ROCm installation, so OpenPFC can find HeFFTe and find_package(HIP) succeeds:

export CC=$(which gcc)
export CXX=$(which g++)
export CMAKE_PREFIX_PATH=$HOME/opt/heffte/2.4.1-rocm:/opt/rocm-6.4.0:$CMAKE_PREFIX_PATH
cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" \
      -DCMAKE_CXX_COMPILER="$CXX" \
      -DOpenPFC_ENABLE_HIP=ON \
      -S . -B build-hip
cmake --build build-hip -j"$(nproc)"

Adjust the ROCm path in CMAKE_PREFIX_PATH if your install is elsewhere (e.g. /opt/rocm). Optionally add -DCMAKE_HIP_ARCHITECTURES=<arch> to match your AMD GPU.

GPU-aware MPI (HIP): OpenPFC_MPI_HIP_AWARE defaults to ON when HIP is found. It enables device-pointer halo exchange in OpenPFC and should match a HeFFTe build with GPU-aware MPI (see HeFFTe’s Heffte_ENABLE_GPU_AWARE_MPI). The install includes verify_gpu_aware_mpi in bin/ to probe device-buffer MPI at runtime. On LUMI-G, see docs/INSTALL.LUMI.md for MPICH_GPU_SUPPORT_ENABLED and job layout.

Minimal configure (optional): To disable code coverage and documentation (avoids gcov link issues with the HIP toolchain):

cmake -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_C_COMPILER="$CC" -DCMAKE_CXX_COMPILER="$CXX" \
      -DOpenPFC_ENABLE_HIP=ON \
      -DOpenPFC_ENABLE_CODE_COVERAGE=OFF \
      -DOpenPFC_BUILD_DOCUMENTATION=OFF \
      -S . -B build-hip

If HIP is not found: -DOpenPFC_ENABLE_HIP=ON is fail-closed — configure stops with a fatal error instead of silently building a CPU tree. Load the ROCm module so hipcc is on PATH, or set CMAKE_PREFIX_PATH / ROCM_PATH to the ROCm prefix that contains lib/cmake/hip/hip-config.cmake (ROCm 6/7) or HIPConfig.cmake. scripts/build.sh --with-rocm locates that prefix (Tohtori rocm/7.2.1 does not set CMAKE_PREFIX_PATH; hipcc may live in /usr/bin, so the script does not use dirname hipcc/.. as ROCM_PATH). CMake 3.21–3.24 must use ROCm Clang as CMAKE_HIP_COMPILER, not the hipcc wrapper. Then reconfigure from a clean build directory.

CMake will warn if HIP is enabled but HeFFTe lacks ROCm support; that only disables HIP spectral paths (OpenPFC_ENABLE_HIP_SPECTRAL = OFF). Add the ROCm HeFFTe install from §8.1 when you need spectral HIP FFT or Tungsten HIP.

Code coverage: If the HIP build fails at link with undefined __gcov_* symbols, disable code coverage (e.g. -DOpenPFC_ENABLE_CODE_COVERAGE=OFF); coverage is not always compatible with the HIP/Clang toolchain.

toml++ and ROCm headers: If you see a preprocessor error in toml++ about __has_attribute requiring an identifier, it is due to ROCm headers defining the __noinline__ macro. As a workaround, ensure translation units that use both OpenPFC (with HIP) and toml++ include the toml-based headers before any OpenPFC or HIP includes, or try a different ROCm version.

Compiler notes

  • The current Tohtori reference stack is GCC 15.2.0 with Open MPI 5.0.10.

  • Older GCC (e.g. 8.x) may need extra link flags for std::filesystem; OpenPFC’s CMake links libstdc++fs automatically for GCC versions older than 9 when using GNU.