Best Practices

This page helps you decide which annotation options to use and why. For copy-ready manifests, see Annotation usage scenarios. For exact types and defaults, see the annotation reference. Terms such as TFLOPS and VRAM are defined in Terminology.

Sizing

1. Set a request and a limit for both compute and VRAM

The request is what the scheduler reserves and what decides how many Pods fit on a card. The limit is the ceiling the Pod is held to. Set the request to the steady load and the limit to the peak.

tensor-fusion.ai/tflops-request: "10"
tensor-fusion.ai/tflops-limit: "20"
tensor-fusion.ai/vram-request: "4Gi"
tensor-fusion.ai/vram-limit: "4Gi"

If you set only a limit, the request takes the same value. That is safe but reserves the peak all the time.

2. Keep the VRAM request equal to the VRAM limit for services

Unused compute can go to other Pods and be handed back later. Memory cannot: once a process holds it, it stays held. A VRAM limit above the request therefore means the Pod may fail to allocate the extra memory when the card is full. Leave a gap only for jobs that can be retried.

3. Prefer TFLOPS to percentages

compute-percent-* is convenient, but 30% of a T4 and 30% of an A100 are very different amounts, and percentages are not counted against namespace GPU quotas. Use TFLOPS whenever the pool has more than one GPU model or you rely on quotas. Never combine the two on one workload.

4. Remove the native GPU limit once you use fractions

A leftover nvidia.com/gpu: 1 in resources.limits still influences the request. When gpu-count is not set, it decides the GPU count, and with no tflops-* annotation present it also asks for 100% of a card. After you move a workload to TFLOPS and VRAM annotations, delete the native limit and state gpu-count explicitly.

5. Measure, then let autoscaling maintain it

Size from a real run, not from the model card. If you cannot measure, start high and add tensor-fusion.ai/autoscale: "true". To see what the autoscaler would do before it changes anything, use a WorkloadProfile with autoSetResources.enable: true and no targetResource; the recommendation then appears in the workload status only. See Configure AutoScaling.

Mode and isolation

6. Stay in local mode unless you need remote

Local GPU mode is the pool default and has the shortest path to the card. Choose remote vGPU mode (is-local-gpu: "false") when the application has to stay on CPU nodes or when GPU nodes are too scarce to host the application Pods themselves. State the mode explicitly in manifests you share between clusters, because the default comes from each pool.

7. Choose the isolation mode by how much you trust the neighbors

Two consequences to plan for:

• hard limits are fixed when the Pod starts, so use soft for workloads you want to autoscale.
• With the default installation a GPU serves one mode at a time. Mixing soft and hard freely in one pool leaves cards half-used. Standardize on one mode per pool where you can.

8. Do not set sidecar-worker by hand

Hard isolation in local mode adds the worker sidecar on its own, so you don't need the annotation. It also forces local mode, which is a surprise on a workload you meant to run remotely.

Priority

9. Set QoS explicitly, and keep critical rare

When qos is absent, a Pod whose requests equal its limits becomes high; other Pods get the pool default or medium. An implicit rule like this is easy to forget, so write the level down. high and critical Pods can preempt lower levels; if everything is critical, nothing is.

How to choose a level is covered in Create Workload.

Placement

10. Constrain placement only as much as you must

Every constraint shrinks the set of cards the scheduler can use and makes Pending Pods more likely.

• Use gpu-model when the workload truly depends on a model. Copy the value from status.gpuModel of a GPU resource; the match is exact.
• Use vendor in pools with more than one accelerator vendor.
• Keep gpu-indices for debugging and benchmarks. It is a hard filter on card numbers and defeats bin packing.
• For zone or node constraints, use the normal nodeSelector and affinity fields.

11. Always name the GPU containers in multi-container Pods

A Pod with more than one container is rejected without inject-container. Even where it could be inferred, list the containers yourself so that adding a sidecar later does not change which containers see the GPU. If several containers need separate cards, add container-gpu-count.

Operations

12. Keep shared settings in a WorkloadProfile

Put the resources, QoS and autoscaling settings a team uses into a WorkloadProfile and override single values with annotations. Keep in mind that the pool, the local or remote mode and the isolation mode are not read from the profile; set those as annotations.

13. Roll out in stages and test the GPU, not the Pod status

Use tensor-fusion.ai/enabled-replicas to move a few replicas first. At each stage run a real CUDA call in a migrated Pod; a Running Pod alone does not prove the GPU path. The procedure is in Migrate Existing Workload.

14. Read the effective configuration from the Pod

The webhook writes the resolved values back to the Pod, and the scheduler adds the assigned GPUs. When behavior is not what you expect, look there first:

kubectl describe pod <pod-name> | grep tensor-fusion.ai

GPU compute reference

To turn utilization into TFLOPS you need the card's total. The authoritative value for your cluster is the capacity TensorFusion reports:

kubectl get gpu -o custom-columns='NAME:.metadata.name,MODEL:.status.gpuModel,CAPACITY:.status.capacity'

For planning before the hardware exists, these are the FP16 TFLOPS values TensorFusion ships for common NVIDIA cards:

VRAM is read from the device and is not part of this table.

Next steps

• Annotation usage scenarios
• Configure AutoScaling
• Annotation reference