TensorFusion DocsCommand Line Reference
This page lists the command-line flags and environment variables of the TensorFusion binaries: the controller, the hypervisor, the hypervisor TUI, and the worker and client pieces that are injected into Pods. Use it when you customize the Helm chart or a component's Pod template.
All Go binaries accept flags with one or two dashes (-leader-elect and --leader-elect are the same). Boolean flags take the form -flag or -flag=false.
Controller
Binary /manager in the tensor-fusion-operator image. One process runs the CRD controllers, the Pod mutating webhook, the scheduler and an HTTP API.
The Helm chart starts it with:
/manager -metrics-bind-address :9000 -leader-elect -enable-auto-scale -enable-alert -enable-auto-expanderChange the command with the controller.command value.
Flags
| Flag | Default | Description |
|---|---|---|
-leader-elect | false | Enable leader election so that only one replica is active. |
-enable-auto-scale | false | Adjust workers' TFLOPS and VRAM automatically from observed usage. |
-enable-alert | false | Evaluate the alert rules in dynamicConfig.alertRules and send alerts to Alertmanager. |
-enable-auto-expander | false | Add GPU nodes when Pods are pending for lack of GPU resources. |
-alert-manager-addr | http://alert-manager.tensor-fusion-sys.svc.cluster.local:9093 | Alertmanager address, including the scheme. |
-compatible-with-nvidia-container-toolkit | false | Wait for the NVIDIA Container Toolkit toolkit-ready validation before node discovery starts. Set by the chart value controller.compatibleWithNvidiaContainerToolkit. |
-gpu-info-config | /etc/tensor-fusion/gpu-info.yaml | Path to the GPU model catalog (TFLOPS and price per model). |
-dynamic-config | /etc/tensor-fusion/config.yaml | Path to the dynamic config file, rendered from the dynamicConfig Helm value. |
-scheduler-config | /etc/tensor-fusion/scheduler-config.yaml | Path to the scheduler configuration, in KubeSchedulerConfiguration format. |
-metrics-path | /logs/metrics.log | File the controller appends metrics to. Vector tails this file. |
-host-port-range | 40000-42000 | Port range for host ports assigned to pre-scheduled Pods such as vGPU workers. |
-cluster-host-port-range | 42000-62000 | Port range for Pods annotated with tensor-fusion.ai/host-port: auto. |
-health-probe-bind-address | :8081 | Address of the /healthz and /readyz probes. |
-metrics-bind-address | 0 | Address of the controller-runtime metrics endpoint. 0 disables it. |
-metrics-secure | false | Serve the metrics endpoint over HTTPS. |
-enable-http2 | false | Enable HTTP/2 on the metrics and webhook servers. |
-v | 0 | Log verbosity (klog). Higher numbers log more. |
The controller logs through klog, so the other standard klog flags (-logtostderr, -log_file, -vmodule and so on) also apply. The -zap-* flags of earlier releases no longer exist.
Environment variables
| Variable | Default | Description |
|---|---|---|
OPERATOR_NAMESPACE | Set by the chart to the release namespace | Namespace the controller runs in. |
TF_ISOLATION_MODE_POLICY | static (the chart sets dynamic) | static or dynamic. Decides whether a GPU is fixed to one isolation mode or may switch. The controller passes the same value to the hypervisors. |
INITIAL_GPU_NODE_LABEL_SELECTOR | Chart value initialGpuNodeLabelSelector | One key=value label. Nodes that carry it are registered as GPU nodes. |
TSDB_MYSQL_HOST | 127.0.0.1 | GreptimeDB host (MySQL protocol). Used by alerting and autoscaling. |
TSDB_MYSQL_PORT | 4002 | GreptimeDB MySQL port. |
TSDB_MYSQL_USER | root | GreptimeDB user. |
TSDB_MYSQL_PASSWORD | empty | GreptimeDB password. |
TSDB_MYSQL_DATABASE | public | GreptimeDB database. |
ENABLE_WEBHOOKS | enabled | Set to false to run without the Pod webhook. |
ENABLE_SCHEDULER | enabled | Set to false to run without the scheduler. |
ENABLE_CR_CONTROLLER | enabled | Set to false to run without the CRD controllers. |
NVIDIA_OPERATOR_PROGRESSIVE_MIGRATION | unset | Set to true to migrate step by step from the NVIDIA device plugin. Set by the chart value controller.nvidiaOperatorProgressiveMigration. |
ENABLE_POD_RESOURCES_PROXY | unset | Set to true to have hypervisors expose a kubelet PodResources proxy socket, which lets DCGM exporter attribute GPU metrics to TensorFusion Pods. |
KUBECONFIG | unset | Path to a kubeconfig file. Needed only when the controller runs outside the cluster. |
Hypervisor
Binary hypervisor in the tensor-fusion-hypervisor image. The controller creates one hypervisor Pod per GPU node. Customize it through componentConfig.hypervisor.podTemplate in the GPUPool.
Flags
| Flag | Default | Description |
|---|---|---|
-vendor | NVIDIA | Accelerator vendor. Overridden by TF_HARDWARE_VENDOR. |
-accelerator-lib | ./provider/build/libaccelerator_example.so | Path to the vendor accelerator library. Overridden by TF_ACCELERATOR_LIB_PATH. |
-isolation-mode | soft | Default isolation mode of the node: shared, soft, hard or partitioned. Overridden by TF_ISOLATION_MODE. |
-isolation-policy | static | static or dynamic. Overridden by TF_ISOLATION_MODE_POLICY. |
-backend-type | kubernetes | kubernetes, or simple for a single machine without Kubernetes. |
-discovery-interval | 12h | How often devices are rediscovered. |
-metrics-output-path | metrics.log | File the hypervisor appends metrics to. |
-metrics-interval | 60s | How often GPU and worker usage is sampled. |
-port | 8001 | Port of the hypervisor HTTP API. Overridden by HYPERVISOR_PORT. |
-v | 0 | Log verbosity (klog). |
For compatibility with v1 Pod templates, a leading daemon argument is accepted and ignored.
Subcommand mount-shm
hypervisor mount-shm mounts the tmpfs that holds the shared-memory files of soft isolation. The controller runs it in the privileged init-shm init container of every hypervisor Pod; you normally never call it yourself.
| Flag | Required | Description |
|---|---|---|
--mount-point | Yes | Directory to mount the tmpfs on. The controller passes /run/tensor-fusion/shm. |
--size | Yes | Size in MB. The controller passes 2048. |
Environment variables
The controller sets the variables marked "injected". Set the others in the hypervisor Pod template when you need them.
| Variable | Source | Description |
|---|---|---|
GPU_NODE_NAME | Injected | Name of the node, used to report GPUNode and GPU resources. |
TENSOR_FUSION_POOL_NAME | Injected | Name of the GPUPool the node belongs to. |
HYPERVISOR_PORT | Injected | HTTP API port. Takes precedence over -port. |
TF_ISOLATION_MODE_POLICY | Injected | static or dynamic, copied from the controller. |
TF_HYPERVISOR_METRICS_FORMAT | Injected | Metrics line format, from dynamicConfig.metricsFormat. |
DETECT_IN_USED_GPUS | Injected | true during progressive migration, so GPUs already used by non-TensorFusion Pods are detected. |
TF_HARDWARE_VENDOR | Optional | Accelerator vendor, for example NVIDIA or Ascend. |
TF_ACCELERATOR_LIB_PATH | Optional | Full path to the accelerator library. The chart sets /run/tensor-fusion/libaccelerator_nvidia.so. |
TF_ISOLATION_MODE | Optional | Default isolation mode of the node. |
KUBELET_CHECKPOINT_PATH | Optional | Path to the kubelet device-manager checkpoint file, if it is not in the default location. |
KUBECONFIG | Optional | Path to a kubeconfig file, for running outside a Pod. |
DISABLE_TENSOR_FUSION_TELEMETRY | Optional | Set to any value to turn off anonymous usage telemetry. See the security whitepaper. |
Hypervisor TUI
hypervisor-tui is a terminal UI for inspecting one hypervisor: devices, workers and live usage. It talks to the hypervisor HTTP API. It is not shipped in the images; build it from the tensor-fusion repository with make build-hypervisor-tui.
| Flag | Default | Description |
|---|---|---|
-host | localhost | Hypervisor host. |
-port | 8001 | Hypervisor HTTP port. |
make build-hypervisor-tui
./bin/hypervisor-tui -host <node-ip> -port 8001Worker
The worker serves GPU calls for remote vGPU and for hard isolation. The controller builds its command; you only see it when you read a worker Pod's spec.
| Scenario | Command set by the controller |
|---|---|
| Remote worker, TCP | ./tensor-fusion-worker -p 8000 |
| Sidecar worker in the same Pod, shared memory | ./tensor-fusion-worker -n shmem -m tf_shm -M 2048 |
| Flag | Description |
|---|---|
-p | TCP port the worker listens on. |
-n | Transport. shmem selects shared memory. |
-m | Name of the shared-memory segment. |
-M | Size of the shared-memory segment in MB. |
| Variable | Description |
|---|---|
TF_ENABLE_LOG | 1 enables worker logs. The controller sets it by default. |
TF_CUDA_SM_PERCENT_LIMIT | Compute limit in percent, set for hard isolation. When the workload gives its limit in TFLOPS, the hypervisor converts it to a percentage of the allocated GPU. |
TF_GPU_MEMORY_LIMIT | VRAM limit in MB, set for hard isolation. TF_CUDA_MEMORY_LIMIT carries the same value for older worker builds. |
Client and limiter injection
The Pod webhook injects GPU libraries into containers that request TensorFusion resources. Nothing is preloaded through the LD_PRELOAD variable in application containers; the webhook mounts a generated /etc/ld.so.preload file instead, so every process in the container loads the library.
| Workload | What is injected |
|---|---|
Local GPU, soft isolation | Init container init-soft-limiter copies the vendor limiter (libcuda_limiter.so for NVIDIA) to /tensor-fusion-limiter. A second init container appends it to the image's own /etc/ld.so.preload. |
| Remote vGPU, or local GPU with a worker sidecar | Init container inject-lib copies the client libraries to /tensor-fusion and generates /etc/ld.so.preload and /etc/ld.so.conf.d/zz_tensor-fusion.conf. |
shared and partitioned isolation | Nothing. |
Variables the webhook sets on application containers:
| Variable | Value | Description |
|---|---|---|
TF_PREPEND_PATH | /tensor-fusion | Directory prepended to PATH. |
TF_LD_LIBRARY_PATH | /tensor-fusion | Directory added to the library search path. |
TENSOR_FUSION_OPERATOR_GET_CONNECTION_URL | Controller URL | Where the client asks for its worker's address (remote vGPU). |
TENSOR_FUSION_CONNECTION_NAME, TENSOR_FUSION_CONNECTION_NAMESPACE | From the Pod | The TensorFusionConnection that pairs the Pod with a worker (remote vGPU). |
HYPERVISOR_IP, HYPERVISOR_PORT | Node IP, 8001 | Address of the hypervisor on the same node (local GPU). |
TF_ISOLATION_MODE | soft | Read by the soft limiter. |
TF_SHM_PATH | /run/tensor-fusion/shm/shm | Shared-memory file the soft limiter reads its limits from. |
POD_NAME, POD_NAMESPACE, CONTAINER_NAME | From the Pod | Identify the container to the hypervisor. |
To switch a built-in feature off for one Pod, use the tensor-fusion.ai/disable-features annotation with a comma-separated list:
| Value | Effect |
|---|---|
gpu-limiter | Do not load the GPU limiter. |
gpu-opt | Turn off CUDA call optimization in the client. |
mem-manager | Turn off the VRAM manager. |
start-worker | Start the worker container with sleep infinity instead of the worker process, for debugging. |
Removed components
Earlier releases documented a separate node discovery binary (--hostname, --gpu-info-config, NODE_DISCOVERY_REPORT_GPU_NODE) and hypervisor flags --sock_path, --gpu_metrics_file and --scheduler. Device discovery now runs inside the hypervisor, and those flags no longer exist.