TensorFusion Metrics Reference

TensorFusion writes its metrics and component logs to GreptimeDB as nine tables, all prefixed with tf_. This page lists every table, its tags and its fields, so you can write SQL for dashboards, alert rules and cost reports.

How metrics are collected

The controller and the hypervisor append metrics to a local file. A Vector sidecar tails the file and posts the lines to GreptimeDB over the InfluxDB line protocol. Metrics stay inside your cluster unless you point the chart at an external GreptimeDB.

Three keys under dynamicConfig in the Helm values control this pipeline:

Conventions

• Tags identify a time series and are indexed. Fields are the measured values.
• Every table has a time index column, greptime_timestamp, written with millisecond precision. Tables created by the start-up DDL of earlier releases use ts instead; see which one a cluster has with SELECT table_name, column_name FROM information_schema.columns WHERE semantic_type = 'TIMESTAMP';.
• TFLOPS values are FP16 TFLOPS. VRAM values are bytes. Fields whose name contains percent or percentage range from 0 to 100.
• "Allocated" means the sum of workload requests. "Limited" means the sum of workload limits.
• "Virtual capacity" is the physical capacity multiplied by the pool's oversubscription ratio.

Cluster and pool tables

tf_system_metrics

Scheduling and scaling counters, one row per pool. The counters are cumulative since the controller started and reset when it restarts, so query them as a difference over a time window.

The four total_topo_* fields are written only after the scheduler has handled at least one allocation for the pool.

tf_pool_metrics

Capacity and allocation summary per pool.

Node and GPU tables

tf_node_metrics

Allocation and hardware cost per GPU node.

tf_gpu_metrics

Capacity and allocation per GPU, as seen by the scheduler.

tf_gpu_usage

Hardware readings per physical GPU, reported by the hypervisor through the vendor library.

The table also has the columns graphics_clock_mhz, sm_clock_mhz, memory_clock_mhz, video_clock_mhz, nvlink_rx and nvlink_tx. They are filled only when the vendor library reports them as extra metrics; the bundled NVIDIA library does not, so they are normally empty.

Workload tables

tf_worker_resources

Requests, limits and billed cost per worker Pod, recorded by the controller.

raw_cost is an increment, not a running total. Sum it over a time range to get the cost for that period.

tf_worker_usage

Actual GPU usage per worker and per GPU, measured by the hypervisor. A worker that uses two GPUs produces two rows per sample.

The columns compute_throttled_cnt, vram_freezed_cnt and vram_resumed_cnt exist in the table schema, but the current hypervisor does not write them.

Note that worker holds the Pod name in tf_worker_resources and the Pod UID in tf_worker_usage. To join the two tables, match tf_worker_resources.worker with tf_worker_usage.pod_name.

tf_gpu_allocation

Which GPUs each Pod holds, one row per Pod.

Log table

tf_system_log

Container logs of the controller, the hypervisors and the workers, collected by Vector.

Example queries

Requested versus used VRAM per workload over the last hour:

SELECT r.workload,
       max(r.vram_bytes_request) AS requested_bytes,
       max(u.memory_bytes)       AS peak_used_bytes
FROM tf_worker_resources r
JOIN tf_worker_usage u ON r.worker = u.pod_name
WHERE r.greptime_timestamp > now() - INTERVAL '1 hour'
  AND u.greptime_timestamp > now() - INTERVAL '1 hour'
GROUP BY r.workload;

Cost per namespace over the last day:

SELECT namespace, sum(raw_cost) AS cost_usd
FROM tf_worker_resources
WHERE greptime_timestamp > now() - INTERVAL '1 day'
GROUP BY namespace;

Next steps

• Query metrics and logs shows how to connect to GreptimeDB.
• Set up alerts covers alert rules built on these tables.