Running Containerized GPU Workloads

This page covers running GPU workloads in regular Kubernetes pods (CUDA, ML training, inference) on Cozystack management cluster nodes. It targets the typical Linux GPU node shape — apt-installed (or dnf-installed) NVIDIA driver plus nvidia-container-toolkit — and uses the container variant of the cozystack.gpu-operator package.

If instead you want to pass whole GPUs to KubeVirt VMs, see GPU Passthrough and GPU Sharing with HAMi (HAMi is for fractional sharing in tenant Kubernetes clusters and is orthogonal to this variant; you can stack HAMi on top once the container variant is up).

When to pick this variant

The cozystack.gpu-operator package exposes three architectural variants. Pick container when all of the following are true:

• The host already runs the NVIDIA driver, installed via the distro package manager (apt install nvidia-driver-* on Ubuntu/Debian or the equivalent on RHEL/Fedora/openSUSE). The operator must not load its own kernel module.
• The host already has nvidia-container-toolkit installed (apt install nvidia-container-toolkit). The operator must not deploy its own toolkit DaemonSet — that would overwrite /etc/containerd/config.toml and the CDI hooks the host package already wired up.
• You want GPUs exposed to containers as nvidia.com/gpu, not passed through to KubeVirt VMs.

The other two variants exist for the opposite host shape: default (passthrough) unbinds the host driver and binds vfio-pci for VM passthrough, and vgpu requires the proprietary NVIDIA vGPU host driver plus a license server. Neither path produces a working setup on a host that already ships the driver and container toolkit through apt — the operator and the host install fight each other.

Prerequisites

• A Cozystack management cluster with at least one GPU-enabled node.
• The GPU node runs a supported Linux distribution (Ubuntu, Debian, RHEL, Fedora, openSUSE) with the NVIDIA driver installed via the distro package manager. Verify with nvidia-smi over SSH or kubectl debug node/<node-name> — it must enumerate the physical GPUs and report a working driver version.
• nvidia-container-toolkit installed on the same node and registered with containerd (grep nvidia /etc/containerd/config.toml shows the runtime entry).
• kubectl configured against the management cluster.

The operator-validator auto-detects pre-installed host drivers by probing /host/usr/bin/nvidia-smi, so on standard Ubuntu/Debian/RHEL/Fedora layouts no hostPaths.driverInstallDir override is needed. On Talos this probe misses (the extension installs nvidia-smi at /usr/local/bin/), so Talos requires a different starting point — see packages/system/gpu-operator/examples/values-native-talos.yaml in the cozystack repo for a working reference with the compat DaemonSet and the matching driverInstallDir override.

1. Install the GPU Operator (container variant)

Do not add cozystack.gpu-operator to bundles.enabledPackages for this variant. The platform Helm chart’s optional-package template hardcodes spec.variant: default for every name in enabledPackages and reconciles the resulting Package CR under Helm ownership — any user Package CR with variant: container is overwritten on the next reconcile. Apply the Package CR directly instead; the cozystack platform controller installs it without the bundle entry.

Apply a Package CR with variant: container:

apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.gpu-operator
spec:
  variant: container

kubectl apply -f gpu-operator-container.yaml

The platform controller resolves the variant against the PackageSource (packages/core/platform/sources/gpu-operator.yaml), pulls values.yaml + values-container.yaml from the OCI repository, and installs the chart into cozy-gpu-operator.

2. Verify the operator is healthy

All pods in the cozy-gpu-operator namespace should reach Running:

kubectl get pods --namespace cozy-gpu-operator

Example output (pod names will vary):

NAME                                                          READY   STATUS    RESTARTS   AGE
gpu-feature-discovery-7jpzv                                   1/1     Running   0          2m
gpu-operator-7976b5b8fb-xqg2z                                 1/1     Running   0          3m
nvidia-cuda-validator-tjkfh                                   0/1     Completed 0          2m
nvidia-dcgm-exporter-rmpfg                                    1/1     Running   0          2m
nvidia-device-plugin-daemonset-cqj9w                          1/1     Running   0          2m
nvidia-operator-validator-q5n4k                               1/1     Running   0          3m

The container variant does not spawn nvidia-driver-daemonset, nvidia-container-toolkit-daemonset, or nvidia-vfio-manager — all three are pinned off by design.

The node should advertise nvidia.com/gpu as an allocatable resource:

kubectl describe node <node-name>

...
Capacity:
  ...
  nvidia.com/gpu:         2
  ...
Allocatable:
  ...
  nvidia.com/gpu:         2
...

3. Run a sample CUDA pod

Create a pod that requests one GPU and runs nvidia-smi:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-smoke
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda
    image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
    command: ["nvidia-smi"]
    resources:
      limits:
        nvidia.com/gpu: 1

kubectl apply -f cuda-smoke.yaml
kubectl logs cuda-smoke

The output should enumerate the GPU(s) visible to the pod and report the driver version that the host runs.

Fractional GPU sharing

The container variant exposes whole GPUs through the upstream NVIDIA device plugin. To slice one GPU across multiple pods (memory and compute quotas per pod), enable HAMi on top — HAMi reuses the same device plugin layer and is wired in via the cozystack.hami package, which already depends on cozystack.gpu-operator. See GPU Sharing with HAMi for the tenant Kubernetes flow; for management-cluster workloads the wiring is the same package set with HAMi enabled.

Variant comparison