Preload Model
What TensorFusion does and does not do to shorten model cold start, and how to cache model weights with standard Kubernetes volumes
Model preloading means having model weights ready before a Pod needs them, so a new replica starts serving sooner. TensorFusion has no model preloading feature today. No CRD field, annotation or component downloads, caches or warms up models. This page states what TensorFusion covers in a cold start and shows the Kubernetes setup that works next to it.
What TensorFusion covers in a cold start
| Phase of a cold start | Covered by TensorFusion |
|---|---|
| Waiting for a GPU node | Partly. capacityConfig.warmResources of a pool keeps spare TFLOPS and VRAM available. Node compaction never removes a node if that would leave less than this. In Provisioned and Karpenter pools, TensorFusion also adds nodes when available capacity drops below it |
| Pulling the container image | No |
| Downloading model weights | No |
| Loading weights into VRAM | No |
SchedulingConfigTemplate has a hypervisor.autoFreezeAndResume.intelligenceWarmup block, and the chart's default template fills it in. The controller and hypervisor do not read it, so it has no effect.
Cache model weights on a volume
Until a built-in feature exists, keep weights on a volume that outlives the Pod. The TensorFusion webhook adds its own volumes and init containers to a Pod and leaves yours as they are, so this works the same with or without a vGPU.
-
Create a
PersistentVolumeClaimfor the cache. Use a storage class that supportsReadWriteManyif several replicas share it.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: model-cache spec: accessModes: [ReadWriteMany] resources: requests: storage: 100Gi -
Mount it in the workload and point the framework's cache directory at it. The example uses the Hugging Face cache variable
HF_HOME:apiVersion: apps/v1 kind: Deployment metadata: name: llm-server spec: replicas: 1 selector: matchLabels: app: llm-server template: metadata: labels: app: llm-server tensor-fusion.ai/enabled: "true" annotations: tensor-fusion.ai/tflops-request: "10" tensor-fusion.ai/tflops-limit: "20" tensor-fusion.ai/vram-request: "8Gi" tensor-fusion.ai/vram-limit: "8Gi" spec: containers: - name: server image: <your-inference-image> env: - name: HF_HOME value: /models volumeMounts: - name: model-cache mountPath: /models volumes: - name: model-cache persistentVolumeClaim: claimName: model-cache -
Start the workload once. The first Pod downloads the weights into the volume. Delete it and watch the replacement start:
kubectl delete pod -l app=llm-server kubectl logs -f deploy/llm-serverThe second start skips the download and goes straight to loading the model.
Keep GPU capacity warm
If new replicas wait for a GPU node rather than for weights, raise the pool's warm capacity. Set it on the TensorFusionCluster, under spec.gpuPools[].specTemplate:
capacityConfig:
warmResources:
tflops: "100"
vram: 40GiWith the Helm chart, the same block is the value cluster.pool.capacityConfig.warmResources.
Next steps
- Create Workload for the label and annotations used above.
- Optimize GPU Efficiency for how warm capacity interacts with node compaction.
TensorFusion Docs