LogoTensorFusion Docs
LogoTensorFusion Docs
Homepage

Getting Started

OverviewKubernetes InstallVM/Server Install(K3S)Helm Quick InstallHost/GuestVM InstallTensorFusion Architecture

Application Operations

Create WorkloadAnnotation Usage ScenariosConfigure AutoScalingMigrate Existing WorkloadBest Practices

Customize AI Infra

Production-Grade DeploymentConfig QoS and BillingBring Your Own CloudManaging License

Maintenance & Optimization

Upgrade ComponentsSetup AlertsGPU Live MigrationPreload ModelOptimize GPU Efficiency

Troubleshooting

HandbookTracing/ProfilingQuery Metrics & Logs

Reference

Comparison

Compare with NVIDIA vGPUCompare with MIG/MPSCompare with NVIDIA Run:aiCompare with HAMi

Preload Model

What TensorFusion does and does not do to shorten model cold start, and how to cache model weights with standard Kubernetes volumes

Model preloading means having model weights ready before a Pod needs them, so a new replica starts serving sooner. TensorFusion has no model preloading feature today. No CRD field, annotation or component downloads, caches or warms up models. This page states what TensorFusion covers in a cold start and shows the Kubernetes setup that works next to it.

What TensorFusion covers in a cold start

Phase of a cold startCovered by TensorFusion
Waiting for a GPU nodePartly. capacityConfig.warmResources of a pool keeps spare TFLOPS and VRAM available. Node compaction never removes a node if that would leave less than this. In Provisioned and Karpenter pools, TensorFusion also adds nodes when available capacity drops below it
Pulling the container imageNo
Downloading model weightsNo
Loading weights into VRAMNo

SchedulingConfigTemplate has a hypervisor.autoFreezeAndResume.intelligenceWarmup block, and the chart's default template fills it in. The controller and hypervisor do not read it, so it has no effect.

Cache model weights on a volume

Until a built-in feature exists, keep weights on a volume that outlives the Pod. The TensorFusion webhook adds its own volumes and init containers to a Pod and leaves yours as they are, so this works the same with or without a vGPU.

  1. Create a PersistentVolumeClaim for the cache. Use a storage class that supports ReadWriteMany if several replicas share it.

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: model-cache
    spec:
      accessModes: [ReadWriteMany]
      resources:
        requests:
          storage: 100Gi
  2. Mount it in the workload and point the framework's cache directory at it. The example uses the Hugging Face cache variable HF_HOME:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: llm-server
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: llm-server
      template:
        metadata:
          labels:
            app: llm-server
            tensor-fusion.ai/enabled: "true"
          annotations:
            tensor-fusion.ai/tflops-request: "10"
            tensor-fusion.ai/tflops-limit: "20"
            tensor-fusion.ai/vram-request: "8Gi"
            tensor-fusion.ai/vram-limit: "8Gi"
        spec:
          containers:
            - name: server
              image: <your-inference-image>
              env:
                - name: HF_HOME
                  value: /models
              volumeMounts:
                - name: model-cache
                  mountPath: /models
          volumes:
            - name: model-cache
              persistentVolumeClaim:
                claimName: model-cache
  3. Start the workload once. The first Pod downloads the weights into the volume. Delete it and watch the replacement start:

    kubectl delete pod -l app=llm-server
    kubectl logs -f deploy/llm-server

    The second start skips the download and goes straight to loading the model.

Keep GPU capacity warm

If new replicas wait for a GPU node rather than for weights, raise the pool's warm capacity. Set it on the TensorFusionCluster, under spec.gpuPools[].specTemplate:

capacityConfig:
  warmResources:
    tflops: "100"
    vram: 40Gi

With the Helm chart, the same block is the value cluster.pool.capacityConfig.warmResources.

Next steps

  • Create Workload for the label and annotations used above.
  • Optimize GPU Efficiency for how warm capacity interacts with node compaction.

GPU Live Migration

What GPU context snapshot and live migration cover today, and how to move a GPU workload without them

Optimize GPU Efficiency

Measure how much of a GPU pool is allocated and actually used, then right-size workloads, pack them tighter and release empty nodes

Table of Contents

What TensorFusion covers in a cold start
Cache model weights on a volume
Keep GPU capacity warm
Next steps