An administrator observes that inference requests to Nutanix Enterprise AI are experiencing increased latency.

Comprehensive and Detailed Explanation From Exact Extract of Nutanix Artificial Intelligence documents knowledge without any external URL or links:

When a specific endpoint or node is showing 100% GPU utilization while others are underutilized, it indicates that the inference workload is bottlenecked on a single instance of the model. To resolve this latency and uneven distribution, the administrator should rebuild (or update) the endpoint to include multiple instances (replicas).

By increasing the instance count for that specific endpoint, the Nutanix Enterprise AI platform (via the underlying KServe/Knative orchestration) will provision additional pods. These new pods will be scheduled on the other available nodes (which are currently showing moderate usage), thereby distributing the inference requests (load balancing) across multiple GPUs. This effectively lowers the utilization per node and reduces the latency for incoming requests.