LogoTensorFusion Docs
LogoTensorFusion Docs
Homepage

Getting Started

OverviewKubernetes InstallVM/Server Install(K3S)Helm Quick InstallHost/GuestVM InstallTensorFusion Architecture

Application Operations

Create WorkloadAnnotation Usage ScenariosConfigure AutoScalingMigrate Existing WorkloadBest Practices

Customize AI Infra

Production-Grade DeploymentConfig QoS and BillingBring Your Own CloudManaging License

Maintenance & Optimization

Upgrade ComponentsSetup AlertsGPU Live MigrationPreload ModelOptimize GPU Efficiency

Troubleshooting

HandbookTracing/ProfilingQuery Metrics & Logs

Reference

Comparison

Compare with NVIDIA vGPUCompare with MIG/MPSCompare with NVIDIA Run:aiCompare with HAMi

GPU Live Migration

What GPU context snapshot and live migration cover today, and how to move a GPU workload without them

GPU live migration moves a running GPU workload to another GPU or node without restarting its process: the GPU context is snapshotted, restored at the destination, and the application carries on. The open-source TensorFusion components do not include it. It is a commercial-edition feature. This page states what is available today and how to move workloads in the meantime.

What is available today

CapabilityStatus
Snapshot and restore of a running GPU contextNot in the open-source controller and hypervisor. The hypervisor API exposes POST /api/v1/workers/<id>/snapshot and POST /api/v1/workers/<id>/resume, but both answer 501 Not Implemented. No CRD field or annotation starts a migration
Moving a workload by restarting itAvailable. Node compaction and defragmentation evict Pods so that the scheduler places them elsewhere. The process restarts and reloads its model
Keeping a node out of automatic movesAvailable, with the tensor-fusion.ai/do-not-disrupt label on the GPUNode

To use live migration, contact us about the commercial edition.

Move a workload without live migration

Without live migration, moving a workload means restarting it somewhere else. Plan for the restart.

  1. Make the workload tolerate a restart. Run at least two replicas behind a Service, or accept a short interruption. Add a PodDisruptionBudget. Defragmentation skips a node if any TensorFusion worker Pod on it has no PodDisruptionBudget.

  2. Choose how the move happens:

    • Automatically. Enable defragmentation on the pool. On a cron schedule, it drains nodes with few GPUs in use, one Pod at a time, and only after a simulation confirms every Pod has a place to go. See Optimize GPU Efficiency.

    • By hand. Cordon the node and delete the Pod that holds the GPU. For a local GPU workload, that is the application Pod. For a remote GPU workload, it is the worker Pod, labeled tensor-fusion.ai/component=worker. The Pod must be owned by a controller that recreates it.

      kubectl cordon <node>
kubectl delete pod <pod> -n <namespace>
kubectl get pods -n <namespace> -o wide
kubectl uncordon <node>

      The replacement Pod appears on a different node.

  3. To keep a node out of compaction and defragmentation, label its GPUNode:

    kubectl label gpunode <node> tensor-fusion.ai/do-not-disrupt=true

Next steps

  • Optimize GPU Efficiency for compaction and defragmentation settings.
  • Preload Model to shorten the restart after a move.

Setup Alerts

Send TensorFusion alerts to your notification channel and tune or add alert rules

Preload Model

What TensorFusion does and does not do to shorten model cold start, and how to cache model weights with standard Kubernetes volumes

Table of Contents

What is available today
Move a workload without live migration
Next steps