Production-Grade Deployment

A default helm install gives you one controller replica, a single-node GreptimeDB and one Alertmanager replica. That is fine for a trial and not enough for production. This guide lists the changes to make before production traffic depends on TensorFusion, in the order you would apply them:

1. Run the controller and Alertmanager with multiple replicas.
2. Move metrics to a highly available GreptimeDB.
3. Route alerts to your on-call channel.
4. Pin component versions.
5. Migrate workloads in small steps, with a way back.
6. Review scheduling, update and oversubscription policies.

Prerequisites

• A Kubernetes cluster with GPU nodes, and kubectl and helm access to it. See Kubernetes Install.
• At least two nodes that can run the controller, so the replicas land on different nodes.
• A default StorageClass, for the Alertmanager volumes.
• A GreptimeDB instance outside the chart, or a plan to deploy one (step 2).

The commands below assume the release name tensor-fusion-sys in the namespace tensor-fusion-sys. Put your overrides in one file, called my-values.yaml in this guide, and keep it in version control. All keys are described in Helm Install Values.

Step 1. Make the control plane highly available

Start from the production overlay shipped with the chart and add your own file on top:

helm upgrade --install tensor-fusion-sys tensor-fusion \
  --repo https://download.tensor-fusion.ai \
  --namespace tensor-fusion-sys --create-namespace \
  -f https://download.tensor-fusion.ai/values-production.yaml \
  -f my-values.yaml

If the cluster is connected to the TensorFusion Console, keep the agent.agentId and agent.enrollToken values from the Console's install command in my-values.yaml.

values-production.yaml sets the following:

The overlay removes the built-in GreptimeDB but does not configure a replacement. The install is incomplete until you finish step 2.

The controller uses leader election (-leader-elect in the default command), so one replica is active and the other takes over when it fails. The default pod anti-affinity is preferred. To guarantee that the replicas never share a node, change it to required:

# my-values.yaml
controller:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchExpressions:
              - key: tensor-fusion.ai/component
                operator: In
                values:
                  - operator

Step 2. Use a highly available GreptimeDB

TensorFusion stores metrics and system logs in GreptimeDB. Alert rules, auto-scaling and the Console dashboards all read from it. The built-in instance is a single replica with a 20Gi volume, so pick one of these for production:

• Greptime Cloud: a managed instance. Less to operate.
• GreptimeDB Operator: a self-hosted cluster in your own Kubernetes. Use at least 3 datanodes.

For Greptime Cloud, or any instance that requires a user and password over HTTPS:

# my-values.yaml
greptime:
  installStandalone: false
  isCloud: true
  host: <instance>.<region>.aws.greptime.cloud
  port: 5001
  db: <database>
  user: <user>
  password: <password>

For a self-hosted instance without authentication:

# my-values.yaml
greptime:
  installStandalone: false
  isCloud: false
  host: <greptimedb-frontend-service>.<namespace>.svc.cluster.local
  port: 4002
  db: public

With isCloud: false, the controller queries the MySQL port set in port, while metrics are written to http://<host>:4000 into the public database. Make sure the service exposes both ports.

greptime.password ends up in a Kubernetes Secret, but it also sits in your values file. Pass it with --set greptime.password=... from your secret manager instead of committing it.

View dashboards

With the Console connected, open the cluster in the TensorFusion Console to see GPU, node and workload dashboards. The data stays in your GreptimeDB. The Console queries it through the cluster agent.

Without the Console, connect Grafana to GreptimeDB and build dashboards from the tf_* tables. See Metrics for the table definitions.

Step 3. Set up alerting

The controller evaluates alert rules against GreptimeDB and sends firing alerts to an Alertmanager. You decide which Alertmanager receives them and where it forwards them.

Option A: the Alertmanager from the chart

Enabled by default. Add your receivers and routes:

# my-values.yaml
alert:
  alertManagerConfig:
    global: {}
    receivers:
      - name: default-receiver
        webhook_configs:
          - url: https://<your-alert-webhook>
    route:
      receiver: default-receiver
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 3h

The content follows the Alertmanager configuration format. The default configuration has one receiver with no target, so no notification leaves the cluster until you add one.

Option B: your existing Alertmanager

Turn off the chart's Alertmanager and point the controller at yours. The address must include the scheme. Keep the other flags, because Helm replaces the whole command list:

# my-values.yaml
alert:
  enabled: false
controller:
  command:
    - /manager
    - -metrics-bind-address
    - :9000
    - -leader-elect
    - -enable-auto-scale
    - -enable-alert
    - -enable-auto-expander
    - -alert-manager-addr
    - http://<alertmanager-service>.<namespace>.svc.cluster.local:9093

Configure receivers and routes in that Alertmanager as you do for other alerts.

Add or change alert rules

The chart ships 15 rules covering throttled workers, allocation failures, full or overheated GPUs, low free capacity per node and pool, and idle GPUs. They live in dynamicConfig.alertRules.

To add a rule, copy the whole default alertRules list from the chart's values.yaml into your file, then append yours. Helm replaces lists, so a file that contains only the new rule removes the built-in ones.

# my-values.yaml
dynamicConfig:
  alertRules:
    # ... all built-in rules copied from values.yaml ...

    - name: GPUTemperatureWarning
      query: |
        SELECT node, pool, uuid, avg(temperature) AS avg_temperature
        FROM tf_gpu_usage
        WHERE temperature > {{ .Threshold }} AND {{ .Conditions }}
        GROUP BY node, pool, uuid
      threshold: 80
      evaluationInterval: 30s
      consecutiveCount: 3
      severity: P2
      summary: "GPU temperature above {{ .Threshold }}°C on {{ .node }} {{ .uuid }}"
      description: "GPU {{ .uuid }} on node {{ .node }} in pool {{ .pool }} averages {{ .avg_temperature }}°C"
      alertTargetInstance: "{{ .uuid }}"
      runbookURL: "https://<your-runbook-url>"

The controller reloads rules without a restart. For the meaning of each field, see dynamicConfig.alertRules.

After the change, trigger a test alert, for example a rule with a threshold that is always exceeded, and confirm that it reaches your channel.

Step 4. Pin component versions

Chart 1.8.8 pins most images, but a few still use floating tags. Set every tag explicitly so that a Pod restart never pulls a different version:

The controller and hypervisor images are released together under the same version number. If you set one, check the other. Available tags are listed on Docker Hub.

Also pin the chart itself with --version <chart version> in the helm upgrade command.

Step 5. Migrate workloads gradually

Roll out one workload in steps

Add the annotation tensor-fusion.ai/enabled-replicas to move only some replicas of a workload to TensorFusion. The remaining replicas start unchanged and keep using nvidia.com/gpu.

apiVersion: apps/v1
kind: Deployment
# ...
spec:
  replicas: 10
  template:
    metadata:
      labels:
        tensor-fusion.ai/enabled: "true"
      annotations:
        # 3 of 10 replicas use TensorFusion, the other 7 keep nvidia.com/gpu
        tensor-fusion.ai/enabled-replicas: "3"

Raise the number as you gain confidence, and remove the annotation when all replicas should use TensorFusion.

Coexist with the NVIDIA device plugin

If some Pods in the cluster still request nvidia.com/gpu from the NVIDIA device plugin or GPU Operator, turn on progressive migration so the two allocators do not hand out the same GPU:

# my-values.yaml
controller:
  nvidiaOperatorProgressiveMigration: true

With this flag:

• Pods that request native GPU resources are scheduled by tensor-fusion-scheduler and placed only on nodes where TensorFusion has allocated nothing.
• Nodes with TensorFusion allocations get a PreferNoSchedule taint, which is removed once the node is idle again.
• TensorFusion does not allocate GPUs that the device plugin has already handed out.

Recreate the hypervisor Pods after you change the flag so they pick it up.

The Pod webhook normally sees only Pods labeled tensor-fusion.ai/enabled: "true". With progressive migration on, it sees every Pod creation in the cluster, and the default failurePolicy is Fail. Run at least two controller replicas before you enable it.

Roll back

To return a workload to native GPU usage, set the label to "false" or the enabled replicas to "0", then let the Pods restart:

apiVersion: apps/v1
kind: Deployment
# ...
spec:
  template:
    metadata:
      labels:
        tensor-fusion.ai/enabled: "false"
      annotations:
        tensor-fusion.ai/enabled-replicas: "0"
    spec:
      containers:
        - name: python
          image: ...
          resources:
            limits:
              # keep this during the migration, so the rollback needs no other change
              nvidia.com/gpu: 1 // [!code highlight]

For the full migration procedure, see Migrate Existing Workload.

Step 6. Review the scheduler configuration

TensorFusion Pods are scheduled by tensor-fusion-scheduler, a kube-scheduler with two extra plugins that runs inside the controller. Its configuration comes from the schedulerConfig Helm value and is stored in the ConfigMap tensor-fusion-sys-config under the key scheduler-config.yaml.

The defaults differ from a stock kube-scheduler in two ways:

• It packs nodes. NodeResourcesFit uses RequestedToCapacityRatio, which prefers nodes that are already in use, and the default placement mode is NodeCompactGPULowLoad. Empty nodes stay empty and can be removed.
• GPUs weigh more than CPU and memory. nvidia.com/gpu has weight 5, CPU and memory have weight 1.

The plugin arguments you are most likely to tune:

schedulerConfig:
  profiles:
    - schedulerName: tensor-fusion-scheduler
      # ...
      pluginConfig:
        - name: GPUResourcesFit
          args:
            maxWorkerPerNode: 256
            vramWeight: 0.7
            tflopsWeight: 0.3
        - name: GPUNetworkTopologyAware
          args:
            mode: soft
            topologySource: auto
            maxAllowedTier: 1
            unknownTopologyPolicy: treat-as-worst
            preferLeastDamage: true

schedulerConfig is one object, so copy the whole default block from values.yaml before you edit it. Restart the controller Deployment after a change. Field meanings are in schedulerConfig.

Test a scheduling decision

The controller can simulate scheduling a Pod without creating it:

kubectl port-forward deployment/tensor-fusion-sys-controller 8080:8080 -n tensor-fusion-sys

# simulate scheduling one Pod; the body is a Pod manifest in YAML
curl -X POST http://localhost:8080/api/simulate-schedule \
  -H "Content-Type: application/yaml" \
  -d 'apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: default
  labels:
    tensor-fusion.ai/enabled: "true"
  annotations:
    tensor-fusion.ai/tflops-request: "100"
    tensor-fusion.ai/vram-request: "16Gi"
    tensor-fusion.ai/tflops-limit: "100"
    tensor-fusion.ai/vram-limit: "16Gi"
spec:
  schedulerName: tensor-fusion-scheduler
  containers:
  - name: test-container
    image: nvidia/cuda'

# dump the allocator's in-memory state
curl http://localhost:8080/api/allocation

The response shows the selected node and GPUs, or the filter stage that rejected each candidate.

Step 7. Control how components are updated

When you change the hypervisor, worker or client configuration of a GPU pool, for example a new image, the controller can roll the change out by itself. nodePoolRollingUpdatePolicy on the pool decides whether and how fast:

With a flag set to false, the controller records that the configuration is out of sync and leaves the running Pods alone. They pick up the change when they are next recreated.

For production, update the hypervisor in small batches with time to observe. Leave workers and clients off: updating them restarts your applications, which belongs in your own release process.

apiVersion: tensor-fusion.ai/v1
kind: TensorFusionCluster
# ...
spec:
  gpuPools:
    - name: shared
      isDefault: true
      specTemplate:
        nodeManagerConfig:
          nodePoolRollingUpdatePolicy:
            autoUpdateHypervisor: true
            autoUpdateWorker: false
            autoUpdateClient: false
            batchPercentage: 20
            batchInterval: 5m

The fields maintenanceWindow and maxDuration exist in the CRD, but the current controller does not act on them. Schedule the configuration change itself inside your maintenance window.

Test every update in a non-production cluster first. See Upgrade Components.

Step 8. Set conservative capacity policies

Oversubscription

The default pool lets the scheduler hand out 20 times the physical TFLOPS (tflopsOversellRatio: 2000) and counts 50% of host memory and 60% of host disk as extra VRAM. That suits bursty development workloads. For latency-sensitive production workloads, lower the ratios:

# my-values.yaml
cluster:
  pool:
    capacityConfig:
      oversubscription:
        # schedulable TFLOPS = 3 x physical TFLOPS
        tflopsOversellRatio: 300  // [!code highlight]
        # extra VRAM backed by host memory, in percent of host memory
        vramExpandToHostMem: 40  // [!code highlight]
        # extra VRAM backed by host disk, in percent of host disk
        vramExpandToHostDisk: 20

VRAM expansion uses memory and disk on the GPU node. Size the nodes for it, or set both percentages to 0 to keep VRAM strictly physical.

Requests and limits

Keep the gap between request and limit small for workloads that must not be throttled:

apiVersion: apps/v1
kind: Deployment
# ...
spec:
  template:
    metadata:
      labels:
        tensor-fusion.ai/enabled: "true"
      annotations:
        # compute may burst to 2x the request
        tensor-fusion.ai/tflops-request: "100"
        tensor-fusion.ai/tflops-limit: "200"
        # no burst for VRAM
        tensor-fusion.ai/vram-request: "3Gi"
        tensor-fusion.ai/vram-limit: "3Gi"
    spec:
      containers:
        - name: my-workload
          resources:
            requests:
              memory: 1Gi
              cpu: 1000m
            limits:
              memory: 4Gi
              cpu: 4000m

A workload whose requests equal its limits gets QoS high by default. See Config QoS and Billing.

More recommendations

• Node autoscaling. For clusters that add and remove GPU nodes automatically, set minimum, warm and maximum capacity on the pool so bursts are absorbed and cost stays bounded. See Bring Your Own Cloud.
• Log level. The default pool starts the hypervisor with TF_LOG_LEVEL=info. Keep it there in production. Debug-level logging adds noise and overhead.
• Network for remote vGPU. In remote mode, GPU calls travel between the client Pod and the worker Pod over the network. Use a low-latency, high-bandwidth network between the nodes involved.
• Webhook failure policy. Keep controller.admissionWebhooks.failurePolicy: Fail. With Ignore, a TensorFusion Pod created while the webhook is down starts without its GPU allocation.

Verify the deployment

# 2 controller Pods and 3 Alertmanager Pods, all Running
kubectl get pods -n tensor-fusion-sys -o wide

# the cluster and its pools are Running
kubectl get tensorfusioncluster,gpupool

# GPU nodes and GPUs are discovered
kubectl get gpunode,gpu

# the controller connected to GreptimeDB and reports no errors
kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c controller --tail=100

Then delete the active controller Pod and check that new TensorFusion Pods are still admitted and scheduled while it restarts.

Next steps

• Helm Install Values: every chart key and its default.
• Bring Your Own Cloud: automatic GPU node provisioning.
• Config QoS and Billing: priorities and internal chargeback.
• Troubleshooting Handbook