During multi-node HPL burn-in, GPUs show uneven utilization.

Uneven GPU utilization in a multi-node cluster is a classic symptom of PCIe/NUMA imbalance. In a DGX H100, each set of GPUs is physically closer to a specific CPU socket and a specific set of high-speed NICs. If the workload (HPL) is launched without strict affinity, an MPI rank running on a CPU core attached to Socket 0 might attempt to control a GPU attached to Socket 1. This forces data to cross the inter-processor links (UPI/QPI), which have significantly higher latency and lower bandwidth than a direct PCIe path. This " bottlenecking " causes some GPUs to wait for data longer than others, leading to the uneven utilization observed. The verified solution is to use an orchestration script or flags that enforce Affinity. By setting --gpu-affinity and --cpu-affinity, the administrator ensures that each GPU is managed by a CPU core on its local NUMA node. This alignment minimizes latency and ensures that every GPU in the cluster receives data at the same rate, resulting in the flat, high-utilization profile required for a successful HPL burn-in record.