A systems engineer is updating firmware across a large DGX cluster using automation.

Updating firmware on an NVIDIA DGX cluster is a critical operation that involves multiple sensitive components, including the GPU baseboard, the BMC, the motherboard tray (SBC), and the InfiniBand HCAs. In a production environment, "Batching" is the industry standard to prevent a single corrupted firmware image or update failure from taking down the entire AI factory. The process must begin with "Draining" the nodes in the workload scheduler (like Slurm or Kubernetes) to ensure no active training jobs are interrupted. Running pre-update diagnostics—using tools like nvsm show health or dcgmi diag—is vital to establish a baseline and ensure the hardware is stable before applying changes. Once the firmware is applied in a controlled batch, post-update verification is required to confirm the system returns to a "Healthy" state and that all versions match the target manifest. This "Rolling Update" strategy allows the engineer to pause the automation if a specific node fails to return to service, protecting the overall availability of the cluster. Skipping diagnostics (Option D) or leaving nodes on mismatched versions (Option C) creates "configuration drift," which leads to unpredictable performance in collective communication libraries.