You have noticed that users can access all GPUs on a node even when they...

Comprehensive and Detailed Explanation From Exact Extract:

To restrict users’ access strictly to the GPUs allocated to their jobs, Slurm usescgroups (control groups)for resource isolation. Enabling device cgroup enforcement by settingConstrainDevices=yesincgroup.confenforces device access restrictions, ensuring jobs cannot access GPUs beyond those assigned.

Increasing memory allocation or setting job priorities does not restrict device access.

Modifying job scripts to request additional CPU cores does not limit GPU access.

Hence, enablingcgroup enforcement with ConstrainDevices=yesis the correct method to prevent users from accessing unallocated GPUs.