Tracing/Profiling

When Pod status and Kubernetes events don't explain a problem, three tools go deeper. Component logs show what the TensorFusion libraries do inside a workload. The hypervisor API and its terminal UI show the devices, workers and limits on one node. NVIDIA Nsight Compute profiles the CUDA kernels themselves. Start with the Handbook for common symptoms, and come here when it points you to logs or to the hypervisor.

Enable TensorFusion logs in a workload

The libraries that TensorFusion injects into a workload, and the worker, are silent by default. Three environment variables control their logging:

1. Add the variables to the GPU container of the workload and let the Pod restart:

   env:
     - name: TF_ENABLE_LOG
       value: "1"
     - name: TF_LOG_LEVEL
       value: "debug"

   For a remote GPU workload, the worker Pod runs the GPU calls. Set the same variables on the worker container, in componentConfig.worker.podTemplate of the pool. The cluster created by the Helm chart already sets TF_ENABLE_LOG=1 there.

2. Read the output:

   kubectl logs <pod> -n <namespace> -c <container>

   A log line has this shape:

   [2026-01-01 12:00:00.000] [147] [152] [TF] Error Func:Consume Line:80 call api=cuMemAddressReserve failed

   The fields are the time, process ID, thread ID, level, source function and line, then the message.

3. Remove the variables when you are done. debug and trace are verbose and slow the workload down.

A fatal error in the injected library is always printed to stderr with the prefix [TF panic], even when logging is off.

Logs of Pods labeled tensor-fusion.ai/component=hypervisor or worker are also stored in the tf_system_log table when the hypervisor vector sidecar is enabled. See Query Metrics & Logs.

Inspect a node's hypervisor

Each GPU node runs one hypervisor Pod named hypervisor-<node>. It serves an HTTP API on port 8001, unless the pool sets componentConfig.hypervisor.portNumber. Forward the port to your machine:

kubectl -n tensor-fusion-sys port-forward pod/hypervisor-<node> 8001:8001

With curl

curl -s http://localhost:8001/healthz
curl -s http://localhost:8001/api/v1/devices
curl -s http://localhost:8001/api/v1/workers

/api/v1/devices lists the GPUs the hypervisor discovered on the node. /api/v1/workers lists the workers it tracks, with their allocation. An empty device list means discovery failed. See GPUs are not discovered.

With the terminal UI

The terminal UI shows the same data as a live view, with usage charts. It is a separate binary that is not part of the hypervisor image, so build it from source. This requires Go:

git clone https://github.com/NexusGPU/tensor-fusion.git
cd tensor-fusion
make build-hypervisor-tui
./bin/hypervisor-tui -host localhost -port 8001

Profile CUDA kernels with Nsight Compute

To see where GPU time goes inside your application, use NVIDIA Nsight Compute (ncu). It ships with the CUDA Toolkit, not with TensorFusion.

Profile in a Pod with tensor-fusion.ai/isolation: shared. In that mode TensorFusion allocates a whole GPU and injects nothing into the container, so the profile shows your application only.

ncu --config-file off \
    --export profile-$(date +%Y%m%d-%H%M%S) \
    --call-stack \
    --force-overwrite \
    python3 main.py

The run writes a .ncu-rep report file. Open it in the Nsight Compute UI.

Next steps

• Handbook for symptom-based troubleshooting.
• Query Metrics & Logs for historical usage and component logs.