How well does DNDSR scale when each CPU core has roughly the same amount of work? This study measures its fourth-order Euler3D solver on four CPU clusters. The largest accepted production run reaches 8,192 MPI ranks, but the clusters differ sharply in how they behave at scale.

Every run uses whole nodes and one MPI rank per physical core. The weak-scaling runs and workload-matched retests keep roughly 1,024 periodic hexahedral cells per rank; four diagnostic controls use heavier workloads. The measurement covers the steady internal update of the P3 VFV scheme (maxOrder=3, quadrature intOrder=5, limiter disabled), not mesh setup.

Each completed run records 500 solver updates. The first 100 serve as warmup; all 400 remaining updates enter the statistics without trimming. The mean includes slow updates and stalls, while the median better describes a typical update. Their difference matters most at high rank counts, where a few long updates can lower sustained throughput. A cell-iteration is one cell processed in one solver update; MCell-it/s means millions of these per second. Timing CV describes the relative variation among update times.

The audit contains 31 runs with all 400 measured updates:

• 22 production weak-scaling runs using serial CGNS input;
• five workload-matched distributed-HDF5/new-MPI retests;
• four distributed-HDF5 diagnostic controls with deliberately heavier per-rank workloads.

Runs that stopped during mesh setup produced no update timings, so they are not performance points.

Machines and MPI stacks

These are the normal production partitions: a8 used its 384 GB class rather than the dedicated 768 GB large-memory partition, and m9 used 768 GB rather than 1.5 TB. m8 used its M8-specific 768 GB partition; thtj1 used cp6.

Production weak scaling

Production weak scaling. Throughput continues to rise to m9’s 8,192-rank point, although its efficiency falls. thtj1 shows a different pattern: a sharp throughput drop beyond 16 nodes.

Download the vector PDF.

The horizontal axis counts physical CPU cores. The left panel shows stall-inclusive mean throughput; the right panel compares that throughput with each cluster’s one-node baseline.

m9 reaches the largest accepted production point: 8,192 ranks on 32 nodes. It delivers 30.20 million cell-iterations/s by the mean, versus 44.44 million by the median. Mean and median efficiencies are 39.6% and 49.5%. That gap, alongside a timing CV of 0.978, shows how strongly slow updates affect the sustained average.

m8 and a8 retain 71.8% and 74.7% mean efficiency at 16 nodes. thtj1 also scales smoothly to 16 nodes, then drops from 7.31 to 2.10 million cell-iterations/s at 32 nodes. Its 32-node timing CV is only 0.011: the slowdown is consistent, not the result of a few outlier updates. Throughput partly recovers at 64 nodes, but efficiency stays near 13%. The attempted 128-node runs never reached solver updates and therefore have no throughput measurement.

Per-core throughput and MPI retests

Throughput per core. Production and workload-matched HDF5/MPI retest series are distinguished in the legend. Completing a large run does not guarantee strong per-core performance, particularly on m8.

Download the vector PDF.

The five HDF5 retests keep the workload within 1.2% of the 1,024-cell-per-rank target. They use validated MPI stacks available on the same machines.

On m8 and a8, the newer Open MPI stacks complete workloads at scales where the earlier stacks failed during setup. The executable and UCX version were retained; HDF5 input alone did not make the old Open MPI 4.1.0 large-rank attempts complete. These checks support an MPI-stack explanation for the completion difference, but not a general speedup.

Performance remains uneven. Relative to the matching one-node retest, a8 retains 57.7% mean and 62.9% median efficiency at 32 nodes. m8 retains only 21.7% mean and 22.0% median efficiency, with CV 0.944. Its newer stack solves the completion problem, not the large-scale performance problem.

The m9 HDF5 run produced all 400 samples, and the application and MPI steps exited successfully. The outer allocation reached its shortened time limit only during cleanup. Its performance sample is therefore included, with that status noted in the table.

Heavier diagnostic controls

The remaining four completed runs have 1.33 to 15.92 times the target cells per rank. Their timings are valid for those heavier workloads, but they do not belong to the workload-matched weak-scaling curves.

Interpretation and scope

The benchmark reveals three distinct behaviors:

1. m9 runs the unchanged production path at 8,192 ranks, beyond 5,000. There is no universal rank-count ceiling near that value.
2. On m8 and a8, repeatable large-rank setup failures are associated with the production Open MPI 4.1.0 stacks. Newer validated stacks complete at the same scale, although m8’s sustained efficiency remains poor.
3. thtj1 has a different limit: its 32-node updates are consistently slow, and its 128-node attempts spend hours in distributed mesh setup without producing update timings.

This is a whole-node, pure-MPI application benchmark, not a controlled comparison of CPU architectures. Core counts, processors, MPI and UCX versions, and network fabrics differ among clusters. Read mean/median gaps and timing CVs alongside throughput rather than judging a cluster by a single high-rank result.

The solver source is DNDSR dev/harry commit f6ece7e91e97c6337ef0761c30017b2ddad6b24b. Exact configuration hashes, machine-readable timing rows, scheduler classifications, and failure evidence are in the public benchmark report.