Optimize GPU Efficiency

A GPU pool wastes money in two ways: workloads reserve more than they use, and the reservations are scattered over more nodes than necessary. This guide measures both from the metrics TensorFusion already collects, then walks through the four settings that fix them: workload autoscaling, placement mode, node compaction and defragmentation.

Prerequisites

• TensorFusion installed with at least one running GPUPool, and workloads that have been running long enough to show a usage pattern. A day is a reasonable minimum.
• Access to GreptimeDB. See Query Metrics & Logs.
• The hypervisor vector sidecar enabled. Actual usage (tf_gpu_usage, tf_worker_usage) and workload autoscaling depend on it. See Ship hypervisor metrics and logs.
• Permission to edit the TensorFusionCluster and SchedulingConfigTemplate.

Pool settings in this guide go on the TensorFusionCluster, under spec.gpuPools[].specTemplate. The controller overwrites the derived GPUPool from that template.

Step 1. Measure allocation and usage

Compare three numbers per node: capacity, how much is allocated, and how much is used.

Allocated share per node over the last day:

SELECT node, pool,
       avg(allocated_tflops_percent) AS tflops_allocated_pct,
       avg(allocated_vram_percent)   AS vram_allocated_pct
FROM tf_node_metrics
WHERE greptime_timestamp >= now() - '1d'::INTERVAL
GROUP BY node, pool
ORDER BY tflops_allocated_pct;

Used share per GPU over the last day:

SELECT node, uuid,
       avg(compute_percentage) AS compute_used_pct,
       max(memory_percentage)  AS vram_peak_pct
FROM tf_gpu_usage
WHERE greptime_timestamp >= now() - '1d'::INTERVAL
GROUP BY node, uuid
ORDER BY compute_used_pct;

Requested versus used per workload. Run the two queries and compare them by workload:

SELECT workload,
       avg(tflops_request)                  AS tflops_request,
       avg(vram_bytes_request) / 1073741824 AS vram_request_gib
FROM tf_worker_resources
WHERE greptime_timestamp >= now() - '1d'::INTERVAL
GROUP BY workload;

SELECT workload,
       avg(compute_tflops)             AS tflops_avg,
       max(compute_tflops)             AS tflops_peak,
       max(memory_bytes) / 1073741824  AS vram_peak_gib
FROM tf_worker_usage
WHERE greptime_timestamp >= now() - '1d'::INTERVAL
GROUP BY workload;

Read the result like this:

For a quick look without SQL, kubectl get gpunode and kubectl get gpu print total and available TFLOPS and VRAM. The built-in alerts IdleGPU and EmptyGPU report the same conditions continuously. See Setup Alerts.

Step 2. Right-size workload requests

The scheduler reserves a workload's request. A request above real usage blocks capacity that nobody uses. You can lower requests by hand from the numbers in step 1, or let the autoscaler follow actual usage. To turn the autoscaler on, add two annotations to the Pod template:

metadata:
  annotations:
    tensor-fusion.ai/autoscale: "true"
    tensor-fusion.ai/autoscale-target: all   # all, compute or vram

The controller then derives a recommendation from the workload's usage history and applies it to the workers. This requires the -enable-auto-scale controller flag, which the chart sets by default. Check what it recommends:

kubectl get tensorfusionworkload <workload> -n <namespace> -o jsonpath='{.status.recommendation}'

For percentiles, bounds, scheduled scaling and pool-wide rules, see Configure Autoscaling.

Step 3. Choose a placement mode

The placement mode decides which GPU a new workload lands on. It is set in the pool's SchedulingConfigTemplate, in spec.placement.mode.

Switch the template that the chart creates:

kubectl patch schedulingconfigtemplate tf-scheduling-config --type=merge \
  -p '{"spec":{"placement":{"mode":"CompactFirst"}}}'

The mode applies to workloads scheduled after the change. Running workloads stay where they are until they restart or defragmentation moves them.

Step 4. Release empty nodes

Node compaction checks the pool on a fixed period and releases GPU nodes that run no TensorFusion workload, as long as the pool keeps enough capacity afterwards.

capacityConfig:
  minResources:
    tflops: "20"
    vram: 10Gi
  warmResources:
    tflops: "10"
    vram: 5Gi
nodeManagerConfig:
  nodeCompaction:
    period: 30m

A node is released only if all of the following hold: no TensorFusion worker runs on it, its GPUNode is Running and older than 5 minutes, it does not carry the label tensor-fusion.ai/do-not-disrupt: "true", and removing it violates neither warmResources nor minResources.

What "released" means depends on the pool's provisioningMode:

To run a check now instead of waiting for the period, annotate the pool with the current time. The controller ignores a timestamp older than 3 seconds:

kubectl annotate gpupool <pool> --overwrite \
  tensor-fusion.ai/manual-compaction="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

In an AutoSelect pool, see which nodes were marked:

kubectl get nodes -l tensor-fusion.ai/should-delete=true

Step 5. Defragment low-utilization nodes

Compaction only releases nodes that are already empty. Defragmentation empties them: on a cron schedule it evicts the TensorFusion workers from nodes where few GPUs are in use, so the scheduler places them on fuller nodes and compaction can release the source node. It is off by default.

1. Add a PodDisruptionBudget for every workload that may be moved. A node is skipped if any TensorFusion worker Pod on it is not covered by one.

2. Enable defragmentation on the pool. timezone is required:

   nodeManagerConfig:
     nodeCompaction:
       period: 30m
       defrag:
         enabled: true
         schedule: "0 3 * * *"
         timezone: "UTC"
         maxDuration: 2h
         utilizationThresholdPercent: 40

   Field	Default	Meaning
   enabled	false	Turns defragmentation on
   schedule	0 3 * * *	Five-field cron expression. Do not add a TZ= prefix
   timezone	none, required	IANA timezone for the schedule, such as UTC or Asia/Shanghai
   maxDuration	2h	Longest time one run may take. Workers younger than this are not evicted either
   utilizationThresholdPercent	40	A node is a candidate when the share of its GPUs that carry at least one worker is at or below this value. Empty nodes are left to compaction

3. After the scheduled time, check what happened in the pool's events:

   kubectl describe gpupool <pool>

   DefragStarted, DefragPodEvicted and DefragFinished mark a normal run. DefragSkipMissingPDB names a Pod that has no PodDisruptionBudget.

Defragmentation is conservative. It evicts one Pod at a time, and only after a scheduling simulation shows that every worker on the node fits on nodes that are at least as full. It skips nodes that are cordoned, labeled tensor-fusion.ai/do-not-disrupt: "true", or running Pods that use GPUs outside TensorFusion.

Verify

Give the changes at least one compaction period, and one defragmentation run if you enabled it. Then rerun the queries from step 1 and compare:

• The allocated share per node is higher on the nodes that remain.
• The node count went down: kubectl get gpunode.
• The WorkerTFlopsThrottled alert did not start firing. If it did, requests or limits are now too low for those workloads.

Limits

• Neither compaction nor defragmentation moves a running process. Both rely on Pods being restarted elsewhere. See GPU Live Migration.
• In AutoSelect pools, TensorFusion only labels nodes. Without a node autoscaler that removes them, nothing is released.
• SchedulingConfigTemplate has a reBalancer block. The controller does not act on it yet.
• The chart has no values for the placement mode or for defragmentation. Check both after each helm upgrade.

Next steps

• Configure Autoscaling for the full autoscaler configuration.
• Setup Alerts to get notified about idle GPUs and exhausted pools.
• Query Metrics & Logs for more tables to analyze.