Method

examples/25_halo_bandwidth builds a periodic grid, exchanges halo faces, and prints a throughput table. It does not update a field. The program is on branch feat/halo-bandwidth of ahojukka5/OpenPFC, based on master 15775de1. It is not merged yet.

n is the global grid, with \(N_y = N_z = N_x\). The default directions are \(\pm X\). --axis y or --axis z selects one pair, and --faces uses all six. The field is stored with \(x\) fastest, so a \(\pm Z\) or \(\pm Y\) face is a run of consecutive values and a \(\pm X\) face is one value per row. The process grid has to cut the axis being timed. OPENPFC_FD_PROC_GRID sets that grid. On two ranks the default grid is \(1\times1\times2\), which cuts \(z\).

On the device, each remote face is packed into a contiguous buffer, sent with MPI on that device pointer, and unpacked. On the host, the face is a strided subarray of the field.

The clock is one barrier, then \(N\) exchanges, then one barrier, divided by \(N\). The reported time is the maximum across ranks. Uni_GBps is remote send bytes divided by that time and by \(10^9\). Duplex_GBps is twice that, so a pair of directions is counted in both columns. Faces that land on the same rank are left out of the send byte count. Off-node faces are counted from the gathered host names.

Rates below are decimal GB/s, the same unit as a 200 Gb/s endpoint quoted at 25 GB/s. Halo rates include the pack and the unpack. The NVLink section is the buffer transfer alone.

Tohtori

The Tohtori node has eight NVIDIA H100 80GB HBM3 GPUs on NVSwitch. nvidia-smi reports NV18 between pairs. These runs used GPUs 0–3. The MPI is CUDA-aware Open MPI 5.0.10 with UCX 1.20 and CUDA 13.3. MPIX_Query reported GPU support on, and the exchanger printed gpu_aware 1 and contiguous 1.

Two ranks, grid \(2\times1\times1\), devices 0 and 1, halo width 1, 30 iterations after 5 warm-up steps. Both faces stay on the node.

Table 1: Two H100s, \(\pm X\), CUDA-aware MPI.
\(n\) Send bytes Time (µs) One way (GB/s) Both ways (GB/s)
128 262144 39.2 6.68 13.36
256 1048576 43.4 24.15 48.29
512 4194304 89.6 46.79 93.57
1024 16777216 421.1 39.84 79.69
1536 37748736 667.5 56.55 113.10
2048 67108864 1173.9 57.17 114.33

At \(n = 2048\) each rank sends 64 MiB. The halo path moves 57.2 GB/s one way and 114 GB/s both ways.

An H100 SXM GPU attaches to NVLink 4.0 at 450 GB/s in one direction and 900 GB/s in both, across 18 links into the switch. UCX 1.20 advertises the cuda_ipc lane at 400 GB/s. This \(\pm X\) halo reaches 57 GB/s, about 14% of the direct copy measured below. UCX_PROTO_INFO on the same exchange selected a rendezvous zero-copy read on cuda_ipc between GPU 0 and GPU 1. That trace timed \(n = 2048\) at 57.8 GB/s one way. The protocol line shows that the bytes stayed on the GPU path. The rate is the strided pack of an \(x\)-face.

With OPENPFC_ASSUME_GPU_AWARE_MPI=0 the exchanger takes the host-staged path (gpu_aware 0, contiguous 0). At \(n = 2048\) that run is 4.31 GB/s one way and 8.63 GB/s both ways.

Four ranks on devices 0–3, still one node:

Table 2: Four H100s. The all-faces row has two further faces that stay on the same rank.
Layout \(n\) Remote faces Send bytes One way Both ways
\(4\times1\times1\), \(\pm X\) 2048 2 67108864 50.45 100.90
\(2\times2\times1\), all faces 1536 4 37748736 66.61 133.22

Which face

The same binary, still on the same H100 node, with devices 2 and 3. Both are NV18 to each other, as is every pair on this node. Two ranks, halo width 1, 30 iterations after 5 warm-up steps, \(n = 2048\). Each rank sends 64 MiB. Only the cut axis changes.

Table 3: Same 64 MiB send. \(\pm Y\) and \(\pm Z\) are contiguous in the field.
Grid Faces Time (µs) One way (GB/s) Both ways (GB/s)
\(2\times1\times1\) \(\pm X\) 1157.1 58.00 116.00
\(1\times2\times1\) \(\pm Y\) 323.0 207.75 415.49
\(1\times1\times2\) \(\pm Z\) 316.2 212.24 424.48

\(\pm Z\) is 212 GB/s one way and 424 GB/s both ways. \(\pm Y\) is 208 GB/s and 415 GB/s. \(\pm X\) stays at 58 GB/s. Cutting \(z\) or \(y\) is 3.7 times the \(x\)-face rate on this pair.

At \(n = 1024\) the same launch gives \(\pm Z\) 153.7 GB/s one way and \(\pm Y\) 151.1 GB/s. The \(x\)-face is 52.0 GB/s.

Four ranks on devices 0–3, \(n = 1536\), all six faces, 20 iterations after 5 warm-up steps. Both grids send 37,748,736 remote bytes. Two faces stay on the rank.

Table 4: Four H100s, all faces, same send size.
Grid One way (GB/s) Both ways (GB/s)
\(2\times2\times1\) (cuts \(x\) and \(y\)) 66.26 132.51
\(1\times2\times2\) (cuts \(y\) and \(z\)) 85.24 170.48

The default process grid already prefers the second layout. For a cube it picks \(1\times1\times2\) on two ranks and \(1\times2\times2\) on four.

A device MPI subarray of the face, without the pack, is not a faster path. On the \(\pm Y\) and \(\pm Z\) runs it fell to 0.28 GB/s.

NVLink copy

The limit for this node is a contiguous copy between two GPUs, with no pack and no MPI. nvidia-smi topo reports NV18 for every pair, so devices 2 and 3 are the same class of link as devices 0 and 1. Each GPU has 18 NVLink lanes, and nvidia-smi nvlink reports 26.562 GB/s on every lane. Eighteen lanes add to 478 GB/s in one direction.

The copy is cudaMemcpyPeerAsync of one buffer into a second buffer on the other GPU, after cudaDeviceEnablePeerAccess. Peer access is supported in both directions. The clock is a CUDA event around the copies, after 4 warm-up copies. One-way figures use a single stream. Both-ways figures issue the two directions together on two streams and take the slower stream. GB/s is bytes per second divided by \(10^9\).

The plateau is 398.7 GB/s one way and 788 GB/s with both directions in flight. A larger buffer does not go faster. At the halo’s own 64 MiB the raw copy is already 389 GB/s and 172 µs. The \(\pm Z\) halo moves that 64 MiB in 316 µs, 212 GB/s, which is 53% of this plateau. The \(\pm X\) halo, 58 GB/s, is 15% of it.

The H100 node has one ConnectX-6 HCA (MT4123) at 100 Gb/s, 12.5 GB/s one way before overhead. The partition has no second H100 node, so this benchmark did not cross that HCA. The tables above are intra-node.

LUMI-G

The HIP binary was built on dev-g with LUMI/25.09, cpeGNU, Cray MPICH and the ROCm HeFFTe module. GPU-aware MPI used MPICH_GPU_SUPPORT_ENABLED=1. Measurements used one node for the in-package pair and two nodes for the off-node pair.

In-package pair

Two ranks, GCDs 0 and 1, the same grid and iteration count as the H100 pair. rocm-smi on that node places both GCDs on NUMA node 3, one XGMI hop apart, link weight 15. That is the in-package pair of one MI250X. LUMI-G documents that link at up to 400 GB/s bidirectional, 200 GB/s each way. The exchanger printed gpu_aware 1 and contiguous 1.

Table 6: Two GCDs of one MI250X, \(\pm X\), HIP-aware MPI.
\(n\) Send bytes Time (µs) One way (GB/s) Both ways (GB/s)
256 1048576 119.4 8.78 17.56
512 4194304 266.6 15.73 31.47
1024 16777216 856.7 19.58 39.17
1536 37748736 1960.8 19.25 38.50
2048 67108864 3399.7 19.74 39.48

At \(n = 2048\) the halo path is 19.7 GB/s one way and 39.5 GB/s both ways, about 10% of the 200 GB/s one-way in-package peak.

With MPICH_GPU_SUPPORT_ENABLED=0 the exchanger stayed on the host-staged path. At \(n = 2048\) that run is 4.63 GB/s one way and 9.25 GB/s both ways.

LUMI’s documented binding, ROCR_VISIBLE_DEVICES set to the local rank, also completed in this job. At \(n = 2048\) it reaches 15.95 GB/s one way. Each rank then sees its own GCD as device 0.

Whole node

Eight ranks, devices 0–7, no face leaving the node. Cross-module Infinity Fabric is 100 GB/s bidirectional on a single link and 200 GB/s bidirectional on a double link. The \(\pm X\) column is limited by the slowest neighbour pair on the line.

Table 7: Eight GCDs, one node.
Layout \(n\) Iterations Remote faces Send bytes One way Both ways
\(8\times1\times1\), \(\pm X\) 2048 20 2 67108864 16.56 33.11
\(2\times2\times2\), all faces 1536 20 6 28311552 25.42 50.83

Two nodes, one endpoint

One GCD on each of two nodes, grid \(2\times1\times1\), 30 iterations after 5 warm-up steps. Both faces leave the node (OffNodeFaces = 2). Each MI250X has one Slingshot-11 endpoint at 25 GB/s each way (network, and the 25+25 GB/s figure on the LUMI-G page). This launch uses one endpoint per node.

Table 8: Two nodes, one GCD and one Slingshot endpoint each.
\(n\) Send bytes Time (µs) One way (GB/s) Both ways (GB/s)
256 1048576 195.1 5.38 10.75
512 4194304 404.7 10.37 20.73
1024 16777216 1478.7 11.35 22.69
1536 37748736 3413.7 11.06 22.12
2048 67108864 5968.8 11.24 22.49

At \(n = 2048\) the halo path is 11.2 GB/s one way and 22.5 GB/s both ways, about 45% of that endpoint. A full node has four endpoints. That aggregate is not in this table.

A first one-node launch passed --gpus-per-task=1. Slurm then put each GCD in its own cgroup, hsa_amd_ipc_memory_attach returned HSA_STATUS_ERROR_INVALID_ARGUMENT, and the step produced no table. That attempt was cancelled. The reported one-node run kept all eight GCDs in one cgroup and selected devices with ROCR_VISIBLE_DEVICES.

Scratch image

On the same H100 node, openpfc bundle built a scratch image of the CPU program (profile local-cpu, GCC 15.2.0, glibc 2.28). The image is 17,489,920 bytes. The generated helper launched it under mpirun -n 2 on the same \(2\times1\times1\) grid, 30 iterations after 5 warm-up steps.

Table 9: Host \(\pm X\) at \(n = 2048\).
Launch One way (GB/s) Both ways (GB/s)
Native host binary 0.204 0.407
Scratch image 0.351 0.701

Both columns are the strided host face, on a node that was already running other jobs. The image reported that it could not open the verbs device mlx5_0. The launch still wrote the table, which is the check that the bundle workflow runs.

openpfc bundle rejects profiles tohtori-cuda and lumi-g. The GPU tables above are from native binaries. There is no CUDA or HIP image in this round.

Scope

The device halo figures include the pack, the contiguous MPI, and the unpack. On this H100 node a direct copy reaches 398.7 GB/s one way. The \(\pm Z\) halo reaches 212 GB/s of that, and the \(\pm X\) halo 58 GB/s. The LUMI figures are still the \(\pm X\) face, set against Infinity Fabric and one Slingshot endpoint rather than against this NVLink copy. Tohtori’s 100 Gb/s HCA was not crossed. The host figures are not a shared-memory peak. The example is not on master yet.

Performance at scale keeps the earlier LUMI-G campaigns. Those runs are a different record.