Building OpenPFC on tohtori (GCC + Open MPI + optional CUDA)¶

This note complements the generic instructions in INSTALL.md. The current Tohtori openmpi/5.0.10 site module loads GCC 15.2.0 and provides PMI/PMIx support for Slurm srun. The historically named cmake/toolchains/tohtori-gcc11-openmpi.cmake defaults to those current site paths. For a custom stack, set both OPENPFC_GCC_ROOT and OPENMPI_ROOT.

0. Released modules (no build)¶

Members of the propertune group can load a released OpenPFC instead of building it:

: "${OPENPFC_MODULE_ROOT:?Set OPENPFC_MODULE_ROOT to the shared module directory}"
module use "$OPENPFC_MODULE_ROOT"
module avail openpfc
module load openpfc/0.6.0-cpu
srun --mpi=pmix -p all6 -n 4 tungsten case.json

Every release stays installed, one module per version and variant: openpfc/<version>-cpu and openpfc/<version>-cuda. Without a version, openpfc/cpu and openpfc/cuda load the newest release of that variant, and plain openpfc loads the newest CPU release. Loading another openpfc module replaces the loaded one, so switching is a single module load:

module load openpfc/0.6.0-cuda   # one version and variant
module load openpfc/0.5.0-cpu    # replaces it
module load openpfc/cuda         # newest CUDA release

Pass --mpi=pmix: Tohtori’s Slurm has no MPI default, and without it each task starts as its own one-rank job. The module loads its MPI stack, puts the shipped applications on PATH, and adds the install to CMAKE_PREFIX_PATH, so find_package(OpenPFC) works for code built against it. The package asks MPI for its C and CXX components, so a consumer declares project(<name> C CXX).

openpfc/<version>-cuda adds the *_cuda applications and GPU-aware MPI (UCX 1.20 and Open MPI 5.0.10 built with CUDA 13.1, HeFFTe with cuFFT). It is compiled for A100, A40 and H100 (sm_80, sm_86, sm_90), so it runs on the cuda partition nodes with those GPUs and on nvidia_h100. The P100 node has no CUDA 13. Request one GPU per rank:

module load openpfc/0.6.0-cuda
srun --mpi=pmix -p cuda -n 2 --gres=gpu:2 -c 4 --mem=16G -t 1:00:00 \
  aluminum_etd_cuda case.json

$OPENPFC_ROOT/share/openpfc/install-provenance.txt records the tag, revision, HeFFTe and MPI of the install.

Maintainers install a new release with scripts/install_release_module.sh. It builds the tag with that tag’s own scripts/build.sh, installs under /task/$USER/apps/openpfc/<version>-<backend>, writes the modulefile and refreshes the openpfc/cpu, openpfc/cuda and default aliases in modulefiles/openpfc/.modulerc.lua. --update-aliases refreshes only those. Nothing it installs may resolve through /home, which is private to its owner. Run it inside an allocation:

git fetch --tags origin
srun -p all6 -c 32 --mem=64G -t 2:00:00 \
  scripts/install_release_module.sh --version=0.6.0 --backend=cpu

An existing install is never replaced. --help lists the HeFFTe and MPI prefixes it expects. The CPU build uses the site openmpi/5.0.10 and /task/$USER/apps/heffte/2.4.1-fftw-plans-cpu-ompi5.

Both HeFFTe prefixes are HeFFTe 2.4.1 with cmake/patches/heffte-2.4.1-fftw-plans.patch applied; scripts/build_tohtori.sh applies it to the 2.4.1 source it downloads. Without it, FFTW real transforms can crash or give wrong float and strided results (#781). The 0.4.0 and 0.5.0 modules keep the unpatched 2.4.1-cpu-ompi5 and 2.4.1-cuda-ompi5 they were built with. A corrected include/heffte_backend_fftw.h has SHA-256 c7b0d134809a3fa8105bb52cb8ee1476a386ac30a6bc1951ee24e81f11ae6788. The CUDA stack under /task/$USER/apps comes from scripts/build_tohtori.sh:

T=/task/$USER
srun -p cuda -c 16 --mem=64G -t 4:00:00 bash -lc "
  module purge; module load gcc/15.2.0 cuda/13.1
  export OPENPFC_GCC_MODULE=gcc/15.2.0 CUDA_ARCHITECTURES='80;86;90'
  export UCX_PREFIX=$T/apps/ucx/1.20.0-cuda UCX_BUILD_DIR=$T/build/ucx-1.20.0-cuda
  export OPENMPI_PREFIX=$T/apps/openmpi/5.0.10-cuda OPENMPI_BUILD_DIR=$T/build/openmpi-5.0.10-cuda
  export HEFFTE_PREFIX=$T/apps/heffte/2.4.1-fftw-plans-cuda-ompi5 HEFFTE_BUILD_DIR=$T/build/heffte-2.4.1-fftw-plans-cuda-ompi5
  scripts/build_tohtori.sh --build-ucx --build-openmpi --heffte-only --cuda"

Some nodes in the all partition lack libefa.so.1, which the site Open MPI needs through libfabric; ompi_info fails there. Use all6 or week until that is fixed.

1. CPU build (reference)¶

In an interactive shell:

module purge
module load openmpi/5.0.10   # also loads its matching GCC (currently 15.2.0)
export CC=$(which gcc) CXX=$(which g++)
which mpicc mpicxx   # both should resolve under the same Open MPI prefix

Install HeFFTe (FFTW CPU backend) under e.g. $HOME/opt/heffte/2.4.1-cpu as in INSTALL.md §3, then configure OpenPFC with that prefix on CMAKE_PREFIX_PATH or -DHeffte_DIR=.../lib64/cmake/Heffte.

For CMake without loading modules (IDE, batch scripts), use the toolchain file:

cmake -S . -B build-cpu \
  -DCMAKE_TOOLCHAIN_FILE=$PWD/cmake/toolchains/tohtori-gcc11-openmpi.cmake \
  ...

2. CUDA toolkit: load a `cuda` module¶

Before configuring a CUDA-enabled OpenPFC build, load a CUDA module so nvcc and the toolkit match your driver and CMake can detect CUDAToolkit:

module load cuda/13.1    # current Tohtori toolkit; verify with `nvcc --version`
which nvcc
nvcc --version

The site may default to a newer CUDA release; use module avail cuda and choose a version that installs a real toolkit (not only driver stubs). The obsolete cuda/12.9 modulefile can still be visible even though its toolkit directory is absent, so always confirm that nvcc --version succeeds.

Without this, CMake may still find /usr/local/cuda/bin/nvcc, but your PATH and library layout are easier to reason about if they come from one module stack.

3. CUDA-capable HeFFTe (required only for CUDA spectral FFT)¶

OpenPFC links HeFFTe for FFT/decomposition. A CPU-only HeFFTe install (e.g. 2.4.1-cpu) is sufficient for CPU FFTs and for CUDA finite-difference / kernel-only applications that do not use HeFFTe’s cuFFT backend (for example wave2d_cuda).

For CUDA spectral FFT paths (Tungsten CUDA spectral binaries, CUDA FFT runtime, CUDA FFT tests), HeFFTe must be configured with the CUDA backend (-DHeffte_ENABLE_CUDA=ON when building HeFFTe) and compiled for every GPU it will run on. HeFFTe launches its own pack, unpack and scaling kernels around cuFFT. On a GPU for which libheffte.so carries no kernel image, cuFFT still runs but those kernels never launch, and before #848 the transform returned unscaled data with no error; OpenPFC now throws HeFFTe device transform failed: no kernel image is available for execution on the device (209) from the transform instead.

The cuda partition mixes architectures, so the HeFFTe build must cover all of them:

Prefix	Kernel images	Runs on
`/task/$USER/apps/heffte/2.4.1-fftw-plans-cuda-ompi5` (also under `$HOME/opt/heffte/`)	`sm_80`, `sm_86`, `sm_90`	`cuda` partition A100 (g0003) and A40 (g0001, g0002), `nvidia_h100` H100
`$HOME/opt/heffte/2.4.1-cuda` and the private module `heffte/2.4.1-cuda-openmpi5`	`sm_90` only	H100 only; wrong results on A100 and A40

scripts/build.sh --machine=tohtori --with-cuda defaults to the first prefix that exists in that order and to CUDA_ARCHITECTURES=80;86;90. It refuses to configure when the chosen libheffte.so lacks an image for a requested architecture (HEFFTE_SKIP_ARCH_CHECK=1 overrides). Check any prefix yourself with the CUDA module loaded:

cuobjdump --list-elf $HEFFTE_PREFIX/lib64/libheffte.so | grep -o 'sm_[0-9]*' | sort -u

The multi-arch prefix is built by scripts/build_tohtori.sh as shown in §0 together with its CUDA-aware Open MPI in /task/$USER/apps/openmpi/5.0.10-cuda; configure OpenPFC against that MPI, not the site openmpi/5.0.10 module. Set the package location explicitly when configuring by hand:

-DHeffte_DIR=/task/$USER/apps/heffte/2.4.1-fftw-plans-cuda-ompi5/lib64/cmake/Heffte

Important: If both CPU and CUDA HeFFTe installs exist under the same parent directory, OpenPFC’s autodetection may pick the CPU tree first. For CUDA spectral builds, set Heffte_DIR explicitly on the first cmake invocation, as above.

(or set environment variable HEFFTE_DIR to that same directory before configuring).

4. Example: configure with CUDA¶

After module load (GCC, Open MPI, CUDA), a CUDA finite-difference build can use the CPU HeFFTe prefix and will report OpenPFC_ENABLE_CUDA_SPECTRAL = OFF if CUDA HeFFTe is not found. To enable CUDA spectral FFT targets as well, configure with CUDA HeFFTe installed:

export CC=$(which gcc) CXX=$(which g++)
H=/task/$USER/apps/heffte/2.4.1-fftw-plans-cuda-ompi5
cmake -S . -B build-gpu \
  -DOpenPFC_ENABLE_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES='80;86;90' \
  -DMPI_C_COMPILER=/task/$USER/apps/openmpi/5.0.10-cuda/bin/mpicc \
  -DMPI_CXX_COMPILER=/task/$USER/apps/openmpi/5.0.10-cuda/bin/mpicxx \
  -DHeffte_DIR=$H/lib64/cmake/Heffte \
  -DCMAKE_PREFIX_PATH=$H \
  ...

Use CMAKE_CUDA_ARCHITECTURES=native only inside an allocation on the node type you will run on, and never a single architecture for a binary that may land on another cuda node (§3). CMake 3.22 on this system is supported for CUDA: OpenPFC teaches older CMake releases the --std=c++20 nvcc flag so CUDA targets can include OpenPFC’s C++20 public headers without requiring native “CUDA20” dialect support from CMake itself.

5. wave2d: VTK output and CUDA binary¶

CPU binaries wave2d_fd_manual / wave2d_fd and the CUDA app wave2d_cuda accept optional VTK flags (see apps/wave2d/README.md):

mpirun -n 2 ./wave2d_fd_manual 96 96 200 0.02 neumann \
  --vtk $PWD/wave2d-run-out/cpu/u_%04d.vti --vtk-every 50
mpirun -n 2 ./wave2d_cuda 96 96 200 0.02 neumann \
  --vtk $PWD/wave2d-run-out/gpu/u_%04d.vti --vtk-every 50

Open the generated .pvti files in ParaView (multi-rank) or the single .vti if mpirun -n 1.

6. HIP on the `amdgpu` partition (MI210)¶

Tohtori has AMD GPUs on amdgpu (2× MI210 gfx90a). CUDA remains NVIDIA-only (nvidia_h100).

# From the repo root, after the HIP/ROCm HeFFTe prefix exists:
sbatch scripts/sbatch_openpfc_hip_amdgpu.sbatch

The job loads rocm/7.2.1 + openmpi/5.0.10, rebuilds HeFFTe-ROCm into $HOME/opt/heffte/2.4.1-rocm-ompi5 if that prefix is missing (the older 2.4.1-rocm tree was Open MPI 4.1.1 / ROCm 6.4.0 and does not run on current amdgpu nodes), then runs scripts/build.sh --machine=tohtori --with-rocm. With the site module GPU-aware MPI stays off (MPI_HIP_AWARE=0): openmpi/5.0.10 was configured --with-rocm but ships only the null accelerator component and a UCX without ROCm memory support, so MPIX_Query_rocm_support() returns 0 and device buffers in MPI_Send would be copied as host memory.

ROCm-aware MPI (device-buffer transport, SDK `HIP_device_buffers`)¶

The installed-SDK acceptance (tests/packaging/backend_acceptance) and resident multi-GPU stepping need MPI that accepts device pointers. Build the AMD twin of the CUDA stack once, on an amdgpu node (ROCm headers live only there), with the same script that builds the CUDA-aware stack:

# sbatch --partition=amdgpu --gres=gpu:mi210:1 --cpus-per-task=16 ... on g0004
module load rocm/7.2.1
UCX_PREFIX=/task/$USER/apps/ucx/1.20.0-rocm \
OPENMPI_PREFIX=/task/$USER/apps/openmpi/5.0.10-rocm \
  scripts/build_tohtori.sh --build-ucx --build-openmpi --ucx-only --rocm   # UCX
UCX_HOME=/task/$USER/apps/ucx/1.20.0-rocm \
OPENMPI_PREFIX=/task/$USER/apps/openmpi/5.0.10-rocm \
  scripts/build_tohtori.sh --build-openmpi --openmpi-only --rocm           # Open MPI

UCX then lists the rocm_copy and rocm_ipc transports, ompi_info the accelerator:rocm component, and MPIX_Query_rocm_support() returns 1. HeFFTe links libmpi, so build it again against that MPI into /task/$USER/apps/heffte/2.4.1-rocm-ompi5-rocmaware (-DHeffte_ENABLE_ROCM=ON -DHeffte_ENABLE_FFTW=ON -DMPI_CXX_COMPILER=.../5.0.10-rocm/bin/mpicxx). When /task/$USER/apps/openmpi/5.0.10-rocm/bin/mpicc exists, scripts/build.sh --machine=tohtori --with-rocm uses it instead of the module, picks that HeFFTe prefix and sets MPI_HIP_AWARE=1; the SDK manifest then records HIP_device_buffers: true. OPENMPI_ROOT_ROCM, HEFFTE_PREFIX and MPI_HIP_AWARE override each step. Two ranks need both MI210 cards (--gres=gpu:mi210:2); a 4-rank device run is not qualified on this node.

Slurm exports both ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES for an amdgpu allocation. The HIP runtime honours CUDA_VISIBLE_DEVICES as well, so with ROCR_VISIBLE_DEVICES=1 ROCr already exposes the assigned card as device 0 and CUDA_VISIBLE_DEVICES=1 then filters it away: rocminfo lists one gfx90a, hipGetDeviceCount returns no ROCm-capable device is detected, tests report HIP not available and rocFFT plan create failed with error code: 1. Jobs that land on the second card see this; unset CUDA_VISIBLE_DEVICES in the job script before running HIP binaries (the ROCr variable keeps the job on its assigned device).

module load rocm/7.2.1 does not set CMAKE_PREFIX_PATH. build.sh --with-rocm infers ROCM_PATH from hipcc. If you configure by hand, pass -DCMAKE_PREFIX_PATH=/opt/rocm-7.2.1:$HOME/opt/heffte/2.4.1-rocm-ompi5.

Request at least 4 Slurm tasks so CTest’s 2-/3-/4-rank MPI suites have Open MPI slots. One or two GPUs are enough; do not pair --ntasks=4 with --gpus-per-task=1 unless the node has four GPUs.