NVIDIA DRA on a Mixed Cluster: Hard Memory Slices with MPS, Time-Slicing on a GB10

The classic Kubernetes way to request a GPU is an integer: nvidia.com/gpu: 1. You get the whole card, or you share it blindly with time-slicing and no memory limits. Dynamic Resource Allocation (DRA) replaces that with ResourceClaim objects, which let you describe how a device should be shared.

This is the runbook for a hybrid cluster, with the strategy chosen per node according to what the hardware can actually enforce.

Cluster: Kubernetes v1.35.8 (RKE2), containerd 2.2.7 with CDI enabled. Driver: kubernetes-sigs/dra-driver-nvidia-gpu. API: resource.k8s.io/v1.

My notes mark this as verified on 2026-10-27, but the status field in my docs still said “not started”. Note also that I reconstructed the YAML indentation below from a flattened copy, so run kubectl apply --dry-run=server on each manifest before trusting it.

Hardware and sharing strategy

The split is the point. MPS can pin a per-client memory limit, so on the discrete cards a workload that asks for more than its slice gets an immediate CUDA out-of-memory error instead of starving its neighbours. The GB10 has unified memory shared with the CPU, so there’s no hard memory partition to enforce; time-slicing just takes turns.

GPU nodes are tainted nvidia.com/gpu=true:NoSchedule and labelled with abstract tiers (gpu.tier/gpu-1 on the GB10, gpu.tier/gpu-2 and gpu-3 on the RTX node), so workloads pick a tier rather than a machine.

The DaemonSet runs only on the GPU nodes. Control-plane nodes get nothing, which is deliberate.

Deploying

1. Disable the legacy device plugin

The device plugin and the DRA driver can’t manage the same hardware at once:

kubectl patch clusterpolicy cluster-policy --type=merge \
  -p '{"spec": {"devicePlugin": {"enabled": false}}}'

kubectl get pods -n gpu-operator -l app=nvidia-device-plugin-daemonset
# expect: No resources found

2. Install the DRA driver

helm upgrade --install dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
  --namespace dra-driver-nvidia-gpu --create-namespace \
  --set gpuResourcesEnabledOverride=true \
  --set featureGates.MPSSupport=true \
  --set featureGates.TimeSlicingSettings=true \
  --set resources.computeDomains.enabled=false

The flags that matter:

• gpuResourcesEnabledOverride=true: acknowledges the legacy device plugin is off.
• featureGates.MPSSupport=true: enables MPS memory slicing (RTX node).
• featureGates.TimeSlicingSettings=true: enables time-slicing (GB10).
• resources.computeDomains.enabled=false: disables the multi-node NVLink/IMEX controller, which I don’t use, keeping the control plane clean.

3. Check what was published

kubectl get deviceclasses,resourceslices

Expect DeviceClasses gpu.nvidia.com, mig.nvidia.com and vfio.gpu.nvidia.com, plus one ResourceSlice per GPU node.

Workload 1: an 8 GiB-bounded slice on the RTX node (MPS)

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rtx-8gb-mps-claim
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: gpu-slice
          exactly:
            deviceClassName: gpu.nvidia.com
      config:
        - requests: ["gpu-slice"]
          opaque:
            driver: gpu.nvidia.com
            parameters:
              apiVersion: resource.nvidia.com/v1beta1
              kind: GpuConfig
              sharing:
                strategy: MPS
                mpsConfig:
                  defaultPinnedDeviceMemoryLimit: 8Gi
                  defaultActiveThreadPercentage: 50
---
apiVersion: v1
kind: Pod
metadata:
  name: mps-8gb-workload
  namespace: default
spec:
  restartPolicy: OnFailure
  nodeSelector:
    kubernetes.io/arch: amd64
    gpu.tier/gpu-2: "true"
  tolerations:
    - key: nvidia.com/gpu
      operator: Equal
      value: "true"
      effect: NoSchedule
  resourceClaims:
    - name: gpu-allocation
      resourceClaimTemplateName: rtx-8gb-mps-claim
  containers:
    - name: runner
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        claims:
          - name: gpu-allocation

Anything that tries to allocate more than 8 GiB fails immediately with a CUDA OOM. That’s the intended protection for neighbouring workloads.

Workload 2: time-sliced sharing on the GB10 (arm64)

Several pods share the GPU by taking turns:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gb10-timeslice-claim
  namespace: default
spec:
  spec:
    devices:
      requests:
        - name: gpu-timeslice
          exactly:
            deviceClassName: gpu.nvidia.com
      config:
        - requests: ["gpu-timeslice"]
          opaque:
            driver: gpu.nvidia.com
            parameters:
              apiVersion: resource.nvidia.com/v1beta1
              kind: GpuConfig
              sharing:
                strategy: TimeSlicing
                timeSlicingConfig:
                  interval: Default      # Default, Short, Medium, Long
---
apiVersion: v1
kind: Pod
metadata:
  name: gb10-timeslice-workload
  namespace: default
spec:
  nodeSelector:
    kubernetes.io/arch: arm64
    gpu.tier/gpu-1: "true"
  tolerations:
    - key: nvidia.com/gpu
      operator: Equal
      value: "true"
      effect: NoSchedule
  resourceClaims:
    - name: shared-gb10
      resourceClaimTemplateName: gb10-timeslice-claim
  containers:
    - name: worker
      image: nvcr.io/nvidia/pytorch:25.01-py3
      command: ["python3", "-c", "import torch; print('CUDA:', torch.cuda.is_available(), torch.cuda.get_device_name(0))"]
      resources:
        claims:
          - name: shared-gb10

Day-2 operations and gotchas

Check the driver sees the hardware:

kubectl get resourceslice -l resource.kubernetes.io/driver-name=gpu.nvidia.com -o yaml \
  | grep -E "productName|memory|uuid"

Confirm MPS is active on the RTX node while an MPS pod runs: nvidia-smi should list nvidia-cuda-mps-server under processes.

Pod Pending with an unbound ResourceClaim. Nine times in ten it’s the missing toleration for the nvidia.com/gpu taint.

App crashes with out-of-memory at exactly 8 GB. That’s the limit doing its job. If the workload genuinely needs more, raise defaultPinnedDeviceMemoryLimit in the claim template (for example to 16Gi).

Frameworks that pre-allocate based on total VRAM. Some tools see the full 96 GB and try to grab ~90% by default, which collides with a slice limit. Cap them at startup:

• PyTorch: torch.cuda.set_per_process_memory_fraction(0.1)
• vLLM: --gpu-memory-utilization 0.1

Rolling back to the device plugin

helm uninstall dra-driver-nvidia-gpu -n dra-driver-nvidia-gpu
kubectl delete namespace dra-driver-nvidia-gpu
kubectl patch clusterpolicy cluster-policy --type=merge \
  -p '{"spec": {"devicePlugin": {"enabled": true}}}'

kubectl get pods -n gpu-operator -l app=nvidia-device-plugin-daemonset
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable."nvidia\.com/gpu"

Takeaways

1. Choose the sharing strategy per hardware, not per cluster. MPS gives you real memory isolation where the GPU supports it; unified-memory parts can’t offer it, so don’t promise it.
2. DRA and the legacy device plugin are mutually exclusive on the same GPU. Plan the cutover and the rollback together.
3. With claims, “my pod is Pending” now has a new place to look: the ResourceClaim, not just node capacity.

Background on how I first got GPUs advertised at all: Making a GPU Visible to Kubernetes.