Upgrade Components

A TensorFusion upgrade has two parts. helm upgrade replaces the control plane. The controller then rolls out changes to the per-pool components (hypervisor, worker, client) in batches, following the pool's rolling update policy. This guide covers both parts, how to verify and roll back, and how to rotate the credentials the installation uses.

Upgrading from the 1.x line to 2.x follows a different, stricter procedure. Use the v1 to v2 upgrade guide instead of this page.

What gets upgraded

The chart pins the hypervisor image in cluster.hypervisorImage separately from the controller version. Check that value before every upgrade and set it explicitly.

Prerequisites

• Helm 3 and kubectl with cluster-admin access.
• The release name and namespace of your installation. This guide uses tensor-fusion-sys for both, as in Deploy on Kubernetes.
• The target chart version, and the release notes for it.
• For production: at least two controller replicas (controller.replicaCount: 2), so the webhook and scheduler stay available while the Deployment rolls.

Step 1. Back up values and resources

mkdir -p tf-backup
helm -n tensor-fusion-sys get values tensor-fusion-sys -o yaml > tf-backup/values.yaml
kubectl get tensorfusioncluster,gpupool,gpunode,gpu,workloadprofile,tensorfusionworkload,tensorfusionconnection -A -o yaml \
  > tf-backup/tf-state.yaml

tf-backup/values.yaml now holds every value you set at install time. You pass it back in step 3.

Step 2. Update the CRDs

Helm installs the files in a chart's crds/ directory once and never upgrades them. Pull the target chart and apply its CRDs before upgrading the release:

helm pull tensor-fusion --repo https://download.tensor-fusion.ai \
  --version <chart-version> --untar --untardir tf-chart
kubectl apply --server-side --force-conflicts -f tf-chart/tensor-fusion/crds/

Every CRD is reported as serverside-applied.

Step 3. Upgrade the Helm release

helm upgrade tensor-fusion-sys tensor-fusion \
  --repo https://download.tensor-fusion.ai \
  --namespace tensor-fusion-sys \
  --version <chart-version> \
  -f tf-backup/values.yaml
kubectl -n tensor-fusion-sys rollout status deploy/tensor-fusion-sys-controller --timeout=180s

Always pass your values file. Without it, Helm resets every value to the chart default, including the Console agent settings.

This step also updates the TensorFusionCluster and SchedulingConfigTemplate that the chart manages, the Alertmanager StatefulSet and the standalone GreptimeDB. If you created the TensorFusionCluster yourself (cluster.enabled: false), edit it directly for step 4.

Step 4. Roll out hypervisor, worker and client changes

The controller hashes each component's configuration per pool. When a hash changes, it starts a rollout for that component, provided auto update is enabled in the pool's nodeManagerConfig.nodePoolRollingUpdatePolicy:

nodeManagerConfig:
  nodePoolRollingUpdatePolicy:
    autoUpdateHypervisor: true
    autoUpdateWorker: false
    autoUpdateClient: false
    batchPercentage: 20
    batchInterval: 1s

Set the policy on the TensorFusionCluster, under spec.gpuPools[].specTemplate. The controller overwrites the spec of the derived GPUPool from that template, so edits made to the GPUPool are lost.

With a flag set to false, the controller records the new hash and waits. The component keeps running the old version, and ...ConfigSynced stays false in the pool status. When you switch the flag to true, the pending rollout starts by itself.

Watch the rollout:

kubectl get gpupool -o custom-columns='POOL:.metadata.name,HYPERVISOR:.status.componentStatus.hypervisorUpdateProgress,HV_SYNCED:.status.componentStatus.hypervisorConfigSynced,WORKER:.status.componentStatus.workerUpdateProgress,CLIENT:.status.componentStatus.clientUpdateProgress'
kubectl get pods -n tensor-fusion-sys -l tensor-fusion.ai/component=hypervisor -o wide

Progress goes from 0 to 100. A component is fully updated at 100 with ...ConfigSynced: true. A pool without GPU nodes performs no rollout, so its progress can stay empty.

Verify

kubectl get tensorfusioncluster          # PHASE is Running
kubectl get gpupool                      # PHASE is Running
kubectl get gpunode                      # every node is Running
kubectl -n tensor-fusion-sys get pods -l tensor-fusion.ai/component=hypervisor \
  -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase,IMAGE:.spec.containers[0].image'

Each GPU node has exactly one hypervisor Pod, in Running state and with the new image. Then create a test workload and run a real CUDA call in it, as described in Create Workload. A Pod in Running state alone does not prove the GPU path works.

Roll back

helm -n tensor-fusion-sys history tensor-fusion-sys
helm -n tensor-fusion-sys rollback tensor-fusion-sys <revision>

The rollback restores the previous controller image and the previous TensorFusionCluster. The controller then rolls the hypervisor back through the same batched process. Leave the newer CRDs in place. Helm does not downgrade them, and removing fields from a CRD can drop data from existing resources.

Rotate credentials

Console agent token. The Console shows the token in the cluster's connect command and has no option to regenerate it yet. If the token leaked, ask TensorFusion support for a new one. Then apply it and check that the agent reconnects:

helm upgrade tensor-fusion-sys tensor-fusion \
  --repo https://download.tensor-fusion.ai \
  --namespace tensor-fusion-sys \
  --version <chart-version> \
  -f tf-backup/values.yaml \
  --set agent.enrollToken=<new-token>
kubectl -n tensor-fusion-sys logs deploy/tensor-fusion-sys-controller -c cluster-agent --tail=20

The token is a plain environment variable of the Deployment. Anyone who can read Deployments in the namespace can read it, so restrict that access.

Cloud vendor credentials. The chart creates this Secret only when it does not exist, so changing the Helm values has no effect afterwards. Replace the Secret and restart the controller:

kubectl create secret generic tf-cloud-vendor-credentials -n tensor-fusion-sys \
  --from-literal=ak=<access-key> --from-literal=sk=<secret-key> \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl -n tensor-fusion-sys rollout restart deploy/tensor-fusion-sys-controller

GreptimeDB Cloud password. Change the password in GreptimeDB Cloud, run helm upgrade with the new greptime.password, then restart the components that hold the old one:

kubectl -n tensor-fusion-sys rollout restart deploy/tensor-fusion-sys-controller
# Only if the hypervisor vector sidecar is enabled. The controller recreates the Pods.
kubectl -n tensor-fusion-sys delete pod -l tensor-fusion.ai/component=hypervisor

Limits

• maxDuration and maintenanceWindow exist in the rolling update policy, but the controller does not act on them yet. A rollout starts as soon as the change is detected and runs until it finishes.
• There is no per-node canary or pause. To slow a rollout down, lower batchPercentage and raise batchInterval. To stop it, set the autoUpdate* flag back to false.
• Don't change componentConfig.hypervisor.portNumber as part of an upgrade. Running Pods keep the port they were created with and would no longer reach the hypervisor. Change it separately and recreate the workloads of that pool.
• helm upgrade does not restart the controller when only dynamicConfig or schedulerConfig changed. Run kubectl -n tensor-fusion-sys rollout restart deploy/tensor-fusion-sys-controller to load the new configuration.

Next steps

• Setup Alerts to get notified when allocation fails after an upgrade.
• Handbook if a hypervisor does not come back ready.
• Helm values reference for every chart value.