TensorFusion DocsHandbook
Symptom-based troubleshooting for TensorFusion, from undiscovered GPUs and pending Pods to limits that are not enforced
This handbook is organized by symptom. Find the section that matches what you see, run the check, then apply the fix for the cause that matches. All commands assume the Helm release and namespace are both tensor-fusion-sys. Adjust them if yours differ.
Collect the basics
Most problems show up in one of these five outputs. Run them first:
kubectl get tensorfusioncluster,gpupool,gpunode,gpu
kubectl get pods -n tensor-fusion-sys -o wide
kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c controller --tail=200
kubectl logs -n tensor-fusion-sys hypervisor-<node> --all-containers --tail=200
kubectl describe pod <pod> -n <namespace>In a healthy installation the cluster and its pools are Running, every GPU node has a GPUNode in Running phase and one hypervisor-<node> Pod, and every card has a GPU resource with capacity and available values.
Controller keeps restarting
Check
kubectl get pods -n tensor-fusion-sys -l tensor-fusion.ai/component=operator
kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c controller --previous
kubectl describe pod -n tensor-fusion-sys -l tensor-fusion.ai/component=operator| What you see | Cause | Fix |
|---|---|---|
Log ends with unable to load global config file | dynamicConfig in the Helm values is not valid | Fix the values and run helm upgrade |
Log ends with invalid isolation mode policy | controller.isolationModePolicy has an unsupported value | Set it to dynamic or static |
Pod stays in ContainerCreating, events mention the secret tensor-fusion-webhook-secret | The webhook certificate was not created. A Helm hook job creates it, and hooks do not run with helm template piped to kubectl apply | Install or upgrade with helm upgrade --install |
| Errors about unknown kinds or fields right after an upgrade | The CRDs are older than the controller. Helm does not upgrade CRDs | Apply the chart's CRDs. See Upgrade Components |
Last state is OOMKilled | The memory limit is too low for the cluster size | Raise controller.resources.limits.memory |
GPUs are not discovered
Symptom: kubectl get gpu is empty or misses cards, or a pool reports zero capacity.
Discovery is a chain: a labeled Kubernetes Node gets a GPUNode, the GPUNode gets a hypervisor Pod, and the hypervisor reports each card as a GPU. Find the first link that is missing.
Check
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl get gpunode
kubectl get pods -n tensor-fusion-sys -l tensor-fusion.ai/component=hypervisor -o wide
kubectl get gpu -o custom-columns='NAME:.metadata.name,MODEL:.status.gpuModel,PHASE:.status.phase,USED_BY:.status.usedBy,CAPACITY:.status.capacity'| What you see | Cause | Fix |
|---|---|---|
| The node is not listed by the first command | The node lacks the label that TensorFusion selects on. The controller watches only nodes that match initialGpuNodeLabelSelector (default nvidia.com/gpu.present=true), and the pool selects nodes with nodeManagerConfig.nodeSelector | Label the node: kubectl label node <node> nvidia.com/gpu.present=true. Or set initialGpuNodeLabelSelector and cluster.pool.nodeSelectorKey to a label your GPU nodes already have |
The node is labeled, but has no GPUNode | The pool's node selector does not match the node | Compare kubectl get gpupool <pool> -o jsonpath='{.spec.nodeManagerConfig.nodeSelector}' with the node's labels |
The GPUNode exists, but there is no hypervisor Pod, or it is not Running | The hypervisor cannot start | See Hypervisor Pod is not ready |
The hypervisor is Running, but no GPU appears | The hypervisor found no device. The vendor driver is missing or not visible in the container | Run nvidia-smi on the node. Read the hypervisor log and its device list, see Inspect a node's hypervisor |
A GPU exists, but USED_BY is not tensor-fusion | A Pod that requested the card natively, for example through nvidia.com/gpu, holds it. TensorFusion does not allocate such a card | Move that workload to TensorFusion, see Migrate Existing Workload, or leave the card to it |
TFLOPS in CAPACITY is missing or implausible | The GPU model is not in the vendor's ProviderConfig, so capacity is estimated at runtime | Add the model to providerConfigs.nvidia.hardwareMetadata in the Helm values. fullModelName must equal the MODEL column |
Hypervisor Pod is not ready
Symptom: hypervisor-<node> is in Init, CrashLoopBackOff or Pending, the GPUNode stays Pending, or Pods fail to schedule with node tensor-fusion.ai/index_0 allocatable is <= 0, hypervisor likely unhealthy.
Check
kubectl describe pod -n tensor-fusion-sys hypervisor-<node>
kubectl logs -n tensor-fusion-sys hypervisor-<node> --all-containers --tail=200
kubectl logs -n tensor-fusion-sys hypervisor-<node> --all-containers --previous
kubectl describe gpunode <node>| What you see | Cause | Fix |
|---|---|---|
Stuck in the init container init-runtime or init-shm with ImagePullBackOff | The GPU node cannot pull the vendor provider image (providerConfigs.nvidia.images.middleware) or the hypervisor image | See Images cannot be pulled |
Stuck in the init container toolkit-validation, which logs waiting for nvidia container stack to be setup | controller.compatibleWithNvidiaContainerToolkit is true, so the hypervisor waits for the file /run/nvidia/validations/toolkit-ready. NVIDIA GPU Operator writes that file | Fix GPU Operator on the node. If you don't run GPU Operator, set the value to false |
Log contains failed to load accelerator library | The vendor library is not at the path in TF_ACCELERATOR_LIB_PATH, or the GPU driver libraries are not visible in the container | Check that nvidia-smi works on the node and that the NVIDIA Container Toolkit is installed and configured for the container runtime |
The Pod is deleted and recreated again and again, and the GPUNode has HypervisorCrashLoopRecreate events | The controller replaces a hypervisor Pod once a container has restarted 5 times (dynamicConfig.hypervisorMaxCrashCount) | Read the log with --previous before the Pod is replaced, and fix the crash it shows |
| The controller log shows the Pod creation is forbidden by Pod Security | The hypervisor runs privileged, with host PID and host network. The namespace enforces a stricter Pod Security level | Allow the privileged level for the TensorFusion namespace |
| Log shows the HTTP port is already in use | The hypervisor uses the host network. Another process on the node holds port 8001 | Free the port, or set componentConfig.hypervisor.portNumber on the pool and recreate the pool's workloads |
After the fix, the Pod becomes Running and the GPUNode moves to Running.
Webhook did not inject the Pod
Symptom: the Pod runs, but it sees the whole GPU or no GPU, and has none of the TensorFusion init containers.
Check
kubectl get pod <pod> -n <namespace> --show-labels
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.schedulerName}{"\n"}{.spec.initContainers[*].name}{"\n"}'
kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c controller | grep "Mutating pod"An injected Pod carries the label tensor-fusion.ai/gpupool=<pool>. A local GPU Pod uses the scheduler tensor-fusion-scheduler. In soft mode it has the init containers init-soft-limiter and tf-soft-preload-0. With remote GPU or hard isolation it has inject-lib. shared and partitioned Pods get no init container.
| Cause | Fix |
|---|---|
The label tensor-fusion.ai/enabled: "true" is missing. The webhook only receives Pods that carry it | Add it under spec.template.metadata.labels |
| The label or the annotations are on the Deployment's own metadata instead of the Pod template | Move them to spec.template.metadata |
| The Pod was created before you added the label. The webhook acts only when a Pod is created | Recreate the Pod: kubectl rollout restart deployment/<name> |
tensor-fusion.ai/inject-container names a container that does not exist. The webhook then skips the Pod without an error | Correct the container name |
tensor-fusion.ai/enabled-replicas is set and that many Pods of the workload already use TensorFusion | Expected during a gradual rollout. Raise the number to cover more replicas |
| The webhook is not registered | kubectl get mutatingwebhookconfiguration tensor-fusion-sys-mutating-webhook must exist. If not, run helm upgrade again |
Pod creation is rejected by the webhook
Symptom: the Deployment has fewer Pods than replicas. The error is on the ReplicaSet, not on a Pod.
Check
kubectl describe replicaset -n <namespace> -l <your-pod-label>| Error contains | Cause | Fix |
|---|---|---|
gpu pool(...) does not exist | tensor-fusion.ai/gpupool names a pool that does not exist, or the annotation is absent and no pool is the default | Use a name from kubectl get gpupool, or mark one pool with isDefault: true in the TensorFusionCluster |
tflops request is not set | The Pod has no resource request | Add tensor-fusion.ai/tflops-request or tensor-fusion.ai/compute-percent-request, and tensor-fusion.ai/vram-request |
parse tf resources | An annotation has an invalid value | Compare with the annotation reference |
failed calling webhook "mpod.tensor-fusion.ai" | No controller replica is ready, or its certificate is invalid. The webhook's failure policy is Fail, so labeled Pods cannot be created meanwhile | See Controller keeps restarting |
Pod stays Pending
Symptom: a workload Pod, or the worker Pod of a remote GPU workload, is not scheduled.
First find the Pod that needs the GPU. With a local GPU it is the application Pod. With a remote GPU the application Pod waits for a worker Pod, which the controller creates in the same namespace:
kubectl get tensorfusionworkload,tensorfusionconnection -n <namespace>
kubectl get pods -n <namespace> -l tensor-fusion.ai/component=workerCheck
kubectl describe pod <pending-pod> -n <namespace>
kubectl get gpu -o custom-columns='NAME:.metadata.name,MODEL:.status.gpuModel,PHASE:.status.phase,USED_BY:.status.usedBy,MODE:.status.isolationMode,ACTIVE_MODE:.status.activeIsolationMode,AVAILABLE:.status.available'
kubectl get gpuresourcequota -n <namespace>| Event or message | Cause | Fix |
|---|---|---|
GPUQuotaOrCapacityNotEnough | No GPU has enough available TFLOPS and VRAM for the request, or the namespace's GPUResourceQuota is used up | Compare the request with the AVAILABLE column. Lower the request, free capacity, add GPU nodes, or raise the quota |
GPU not fit on every node | GPUs with enough capacity exist, but none passes the other filters: gpu-model, gpu-indices, vendor or isolation mode | Check those annotations against the MODEL, MODE and ACTIVE_MODE columns. A GPU serves one isolation mode at a time |
The Pod requests tensor-fusion.ai/isolation: shared | shared takes a whole GPU and needs one with nothing allocated on it | Free a card, or use soft |
hypervisor likely unhealthy | The hypervisor on that node is down | See Hypervisor Pod is not ready |
gang scheduling timeout, or a GangWaitingForMembers or GangSchedulingTimeout event | The Pod is part of a gang and not all members fit at once | Free capacity for the whole group, or lower tensor-fusion.ai/gang-min-members |
| Only standard Kubernetes reasons, such as taints, node affinity, or insufficient CPU or memory | The GPU fits, the node does not | Fix tolerations, selectors or CPU and memory requests as for any Pod |
| No scheduling event at all | The Pod uses tensor-fusion-scheduler, which runs inside the controller, and no controller replica is ready | See Controller keeps restarting |
The pool keeps a count of failed allocations in the tf_system_metrics table. See Query Metrics & Logs.
Pod is stuck in init
Symptom: the Pod is scheduled but stays in Init:... or Init:ImagePullBackOff.
Check
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <init-container>| Init container | Cause | Fix |
|---|---|---|
init-soft-limiter | The vendor provider image cannot be pulled | See Images cannot be pulled |
inject-lib | The client image cannot be pulled | Same |
tf-soft-preload-0 | This container runs in your application image and needs /bin/sh, cat and chmod. Distroless and scratch images lack them | Use an application image that has a shell, or choose another isolation mode |
Limits are not enforced
Symptom: a workload with soft isolation uses more VRAM or compute than its limit, or nvidia-smi inside the container shows the full memory of the card.
Check
kubectl exec <pod> -n <namespace> -c <container> -- sh -c '
cat /etc/ld.so.preload
env | grep -E "^TF_(ISOLATION_MODE|SHM_PATH)="
ls -l "$TF_SHM_PATH"
nvidia-smi'When limiting works, /etc/ld.so.preload contains /tensor-fusion-limiter/libcuda_limiter.so, TF_ISOLATION_MODE is soft, the file at TF_SHM_PATH exists, and nvidia-smi reports the VRAM limit as the total memory.
| What you see | Cause | Fix |
|---|---|---|
No /etc/ld.so.preload, or the limiter is not in it | The Pod was not injected, or its isolation mode is not soft | See Webhook did not inject the Pod |
The isolation mode is shared | By design. shared allocates a whole card and applies no limit | Use soft or hard to limit a workload |
The file at TF_SHM_PATH is missing | The hypervisor on the node has not published limits for this Pod | Get the hypervisor healthy, then recreate the Pod |
| Everything is in place, but the process is not limited | The process does not load /etc/ld.so.preload. This happens with statically linked binaries and with images based on musl, such as Alpine | Use a glibc-based image. To confirm, look for the limiter in /proc/<pid>/maps of the application process |
| Compute usage goes above the limit for short moments | soft mode limits compute with a rate limiter, so short bursts pass and the average converges to the limit | Use hard isolation if you need a strict cap |
Clearing LD_PRELOAD does not disable the limiter, because it is loaded through /etc/ld.so.preload.
With hard isolation the worker enforces the limits. Check that the worker container has TF_CUDA_SM_PERCENT_LIMIT and TF_GPU_MEMORY_LIMIT:
kubectl get pod <worker-pod> -n <namespace> -o yaml | grep -A1 -E 'TF_CUDA_SM_PERCENT_LIMIT|TF_GPU_MEMORY_LIMIT'CUDA out of memory
Symptom: the application fails with CUDA out of memory or CUDA_ERROR_OUT_OF_MEMORY.
Check
kubectl exec <pod> -n <namespace> -c <container> -- nvidia-smi
kubectl get pod <pod> -n <namespace> -o jsonpath='{.metadata.annotations}'
kubectl describe pod <pod> -n <namespace>| What you see | Cause | Fix |
|---|---|---|
nvidia-smi in the container shows usage at the VRAM limit, while the card itself has free memory | The workload reached its tensor-fusion.ai/vram-limit. In soft mode the limiter fails the allocation with an out-of-memory error | Raise vram-request and vram-limit, lower the batch size or context length, or turn on autoscaling |
| The error appears at startup, before any real work | The framework reserves a fixed share of GPU memory up front | Limit the reservation in the framework. TensorFlow: set TF_FORCE_GPU_ALLOW_GROWTH=true. vLLM: lower --gpu-memory-utilization |
| The workload is below its limit, but the physical card is full | Workloads on the card have VRAM limits above their requests, and together they use more than the card has | Set vram-request equal to vram-limit for workloads that need the memory guaranteed |
The Pod is OOMKilled or Evicted with The node was low on resource: memory | This is host memory, not GPU memory | Set memory requests and limits on the container that match what the model needs while loading |
CUDA or NVML library errors
Symptom: NVML Function Not Found, libnvidia-ml.so.1: cannot open shared object file, a worker assertion such as cuInit_fn != nullptr, or PyTorch reporting device >= 0 && device < num_gpus INTERNAL ASSERT FAILED.
These errors mean the process cannot find the driver's libcuda.so.1 or libnvidia-ml.so.1, or finds them at a different path than TensorFusion expects.
Check on the GPU node, then inside the failing container:
nvidia-smi
ldconfig -p | grep -E 'libcuda.so.1|libnvidia-ml.so.1'
cat /etc/ld.so.preload| Cause | Fix |
|---|---|
| The driver is not installed or not loaded on the node | lsmod | grep nvidia shows nothing. Install the NVIDIA driver |
| The driver libraries are on the node but not in the container | Install and configure the NVIDIA Container Toolkit for your container runtime |
The libraries are in a different directory than the worker expects. The chart's worker template points to /usr/lib/x86_64-linux-gnu. RHEL-family distributions use /usr/lib64 | Set TF_CUDA_LIB_PATH and TF_NVML_LIB_PATH on the failing container to the .so.1 paths that exist in it. Don't use a path that contains a driver version number, because it changes with every driver upgrade |
| PyTorch sees no device at all | The TensorFusion library is not loaded in the application process. See Webhook did not inject the Pod |
Remote GPU workloads
For a remote GPU workload, the application Pod connects to a worker Pod through a TensorFusionConnection. Check both ends first:
kubectl get tensorfusionconnection -n <namespace>
kubectl logs <worker-pod> -n <namespace> --tail=200| Symptom | Cause | Fix |
|---|---|---|
| The connection has no worker or connection URL | The worker Pod is not running yet | See Pod stays Pending |
| The worker Pod keeps restarting | Read its log. For more detail, raise the log level, see Tracing/Profiling | Fix what the log shows. Include the log when you ask for help |
CUDA driver error: named symbol not found in the application | The application calls a CUDA API that remote mode does not support. The worker log names it, for example call api=cuMemAddressReserve failed. PyTorch triggers this one with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | Remove the setting that triggers the call, or run the workload with a local GPU |
Invalid Memory Address errors in the application | A rare conflict with the worker's GPU memory manager | Turn the memory manager off for the workload with the annotation tensor-fusion.ai/disable-features: mem-manager. Features that depend on it stop working for that workload |
Images cannot be pulled
TensorFusion pulls images on two kinds of nodes. The controller, Alertmanager and GreptimeDB images are pulled where those Pods run. The hypervisor, vendor provider, client and worker images are pulled on GPU nodes and on the nodes of your workloads.
Check
kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull'
kubectl describe pod <pod> -n <namespace>| Cause | Fix |
|---|---|
| The cluster cannot reach Docker Hub and other public registries | Configure registry mirrors in the container runtime of every node. Or copy the images to your own registry and override them in the Helm values: controller.image.repository, cluster.hypervisorImage, providerConfigs.nvidia.images.*, greptime.image.repository, alert.image.repository |
| The cluster is in mainland China | Add the values-cn.yaml overlay. See Deploy on Kubernetes |
| A private registry needs credentials | Set imagePullSecrets in the Helm values |
Pods that request nvidia.com/gpu
TensorFusion does not register the nvidia.com/gpu resource. A Pod that requests it without the TensorFusion label is left untouched by the webhook, and it is scheduled only if the NVIDIA device plugin is installed:
helm upgrade --install --create-namespace --namespace nvidia-device-plugin \
--repo https://nvidia.github.io/k8s-device-plugin/ nvdp nvidia-device-pluginTo move such workloads to TensorFusion instead, see Migrate Existing Workload.
Harmless messages
Error in cpuinfo: prctl(PR_SVE_GET_VL) failed appears when an ARM container image runs on a CPU without SVE support, for example on macOS. It can be ignored in most cases.
Get help
If none of the sections match, open an issue at github.com/NexusGPU/tensor-fusion or write to support@tensor-fusion.com. Attach the output of the commands in Collect the basics. Remove tokens, keys and internal addresses before you share logs.
For deeper inspection, see Tracing/Profiling and Query Metrics & Logs.