Migrate Existing Workload

This guide moves a workload that requests nvidia.com/gpu (or another vendor's device-plugin resource) onto TensorFusion. The safe order is: migrate it as a whole GPU first, a few replicas at a time, then shrink it to the fraction it really uses.

There are two ways to migrate. Pick one per workload.

Prerequisites

• TensorFusion is installed, and the GPU nodes the workload runs on belong to a GPUPool. Check with kubectl get gpupool,gpunode,gpu.
• The workload requests GPUs through a device-plugin resource in resources.limits, for example nvidia.com/gpu: 1.
• You can restart the workload. Migration changes the Pod template, so Pods are recreated.

Migrate one workload

Step 1. Record what the workload uses today

kubectl get pod <pod-name> -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'
kubectl get gpu -o custom-columns='NAME:.metadata.name,MODEL:.status.gpuModel,CAPACITY:.status.capacity'

Note the number of GPUs per Pod, the GPU model, and the TFLOPS and VRAM capacity of that model. From your monitoring, also note the real peak GPU utilization and memory of the workload; you need them in step 5.

Some frameworks reserve a memory pool up front, so the memory the GPU reports can be higher than what the model needs. If so, estimate VRAM from the model size instead.

Step 2. Hand the workload to TensorFusion as a whole GPU

Add the label and the isolation: shared annotation to the Pod template. Keep the existing nvidia.com/gpu limit.

spec:
  template:
    metadata:
      labels:
        tensor-fusion.ai/enabled: "true"       # added
      annotations:
        tensor-fusion.ai/isolation: "shared"    # added
        tensor-fusion.ai/enabled-replicas: "1"  # added, see step 3
    spec:
      containers:
        - name: app
          resources:
            limits:
              nvidia.com/gpu: 1                 # unchanged

With isolation: shared, each requested GPU is allocated in full and no limiter is injected, so the workload behaves as it did before. The webhook reads the GPU count from the nvidia.com/gpu limit, removes that limit from the container, and lets the TensorFusion scheduler assign the cards.

If the Pod has several containers, only the ones with a GPU limit are handed to TensorFusion.

Step 3. Roll out to a few replicas first

tensor-fusion.ai/enabled-replicas caps how many Pods of the workload TensorFusion takes over. The remaining Pods are left untouched and keep using the device plugin.

kubectl apply -f <workload>.yaml
kubectl get pod -l <selector> -L tensor-fusion.ai/component

Pods that TensorFusion took over show a value in the COMPONENT column. The others show none.

Raise the number in stages, for example 1, then 25%, 50% and 100% of the replicas, and check the workload at each stage. Each change to the annotation edits the Pod template, so it triggers a rolling update of the whole Deployment.

enabled-replicas works for Pods that have a controller such as a Deployment or StatefulSet. It has no effect when the label tensor-fusion.ai/enabled is not "true".

Step 4. Verify the migrated Pods

POD=<a-migrated-pod>
kubectl get pod "$POD" -o jsonpath='{.metadata.annotations.tensor-fusion\.ai/gpu-ids}{"\n"}'
kubectl exec "$POD" -c app -- nvidia-smi
kubectl get gpu -o custom-columns='NAME:.metadata.name,AVAILABLE:.status.available'

• The Pod has a tensor-fusion.ai/gpu-ids annotation that lists its GPUs.
• nvidia-smi in the container shows the same number of GPUs as before.
• The assigned GPUs show zero available TFLOPS and VRAM.
• Your own health checks and performance tests pass.

When all replicas are migrated and healthy, remove the enabled-replicas annotation.

Step 5. Shrink to the fraction the workload needs

A whole GPU per Pod saves nothing yet. Replace the whole-GPU request with TFLOPS and VRAM values taken from the usage you recorded in step 1:

spec:
  template:
    metadata:
      labels:
        tensor-fusion.ai/enabled: "true"
      annotations:
        tensor-fusion.ai/tflops-request: "30"
        tensor-fusion.ai/tflops-limit: "60"
        tensor-fusion.ai/vram-request: "8Gi"
        tensor-fusion.ai/vram-limit: "8Gi"
    spec:
      containers:
        - name: app
          resources: {}   # nvidia.com/gpu removed

Removing isolation: shared puts the workload in the default soft mode, where the limits are enforced and several Pods can share one card. You can canary this change with enabled-replicas in the same way.

If you cannot size the workload by hand, add tensor-fusion.ai/autoscale: "true" and let TensorFusion adjust the values. See Configure AutoScaling.

Roll back

Remove the tensor-fusion.ai/enabled label and the tensor-fusion.ai/* annotations, and restore the nvidia.com/gpu limit if you removed it. The Deployment rolls back to Pods that use the device plugin.

Migrate automatically

With auto migration on, the webhook converts every new Pod that requests a native GPU resource, without any change to the workload. Each converted Pod gets the label tensor-fusion.ai/enabled: "true" and, unless it already has an isolation annotation, tensor-fusion.ai/isolation: shared. The result is the same as step 2 above.

1. Set the scope in the Helm values and upgrade the release:

   dynamicConfig:
     autoMigration:
       enable: true
       scope:
         includes:
           namespaceNames:
             - ml-team-a
         excludes:
           podSelector:
             matchLabels:
               gpu-migration: skip

2. Restart the workloads you want migrated, for example with kubectl rollout restart. Existing Pods are not touched; only newly created Pods are converted.

3. Verify as in step 4.

How the scope is evaluated:

To keep a single Pod out, label it tensor-fusion.ai/enabled: "false".

Next steps

• Configure AutoScaling
• Annotation usage scenarios
• Annotation reference