Query Metrics & Logs

TensorFusion stores its metrics and component logs in GreptimeDB as ordinary tables. When a dashboard or an alert doesn't tell you enough, query those tables with SQL. This page shows how to connect, which tables exist, and a few queries to start from.

Prerequisites

kubectl access to the namespace GreptimeDB runs in. By default the Helm chart installs a standalone GreptimeDB into the greptimedb namespace (greptime.installStandalone: true).
A mysql client, if you want to query from a terminal instead of the browser.

If you pointed TensorFusion at GreptimeDB Cloud or at your own instance (greptime.isCloud: true or greptime.installStandalone: false), run the same queries in that instance. The tables are identical.

Connect to GreptimeDB

Use the built-in dashboard for ad-hoc queries, and the MySQL protocol for scripts.

Dashboard

kubectl port-forward -n greptimedb svc/greptimedb-standalone 4000:4000

Open http://localhost:4000/dashboard and select the public database.

MySQL client

kubectl port-forward -n greptimedb svc/greptimedb-standalone 4002:4002
mysql -h 127.0.0.1 -P 4002 -u root public

The standalone instance has no password. To confirm the connection, list the tables:

SHOW TABLES;

You should see the tf_* tables described below.

Tables

Table	Written by	One row describes
`tf_gpu_usage`	hypervisor	Actual usage of one physical GPU: compute and VRAM percentage, temperature, power, PCIe and NVLink traffic
`tf_worker_usage`	hypervisor	Actual usage of one worker on one GPU, plus counters for compute throttling and VRAM freeze and resume
`tf_worker_resources`	controller	Requests and limits of one worker, its QoS level and raw cost
`tf_gpu_metrics`	controller	Capacity, allocated and available TFLOPS and VRAM of one GPU
`tf_gpu_allocation`	controller	Which GPUs a Pod was allocated
`tf_node_metrics`	controller	Allocated TFLOPS and VRAM of one GPU node, as values and as percentages
`tf_pool_metrics`	controller	The same allocation figures for a whole pool
`tf_system_metrics`	controller	Counters per pool: workers, nodes, allocation successes and failures, scale-ups and scale-downs
`tf_system_log`	vector	One log line of a TensorFusion component

The time column is greptime_timestamp in every table. Tables created by earlier releases may use ts instead; DESC TABLE <table> shows which. Each component writes metrics about once a minute. Rows are kept for 30 days by default (dynamicConfig.metricsTTL in the Helm values).

For every column, see the metrics reference.

Ship hypervisor metrics and logs

The controller Pod always runs a vector sidecar, so the controller tables and the controller's own logs arrive without extra setup.

tf_gpu_usage, tf_worker_usage and the logs of hypervisor and worker Pods come from a vector sidecar in each hypervisor Pod. That sidecar exists only when the pool sets componentConfig.hypervisor.enableVector: true and declares a second container for it. The TensorFusionCluster that the chart creates does not, so on a default install these two tables stay empty.

To turn the sidecar on for the first pool of the chart-created cluster:

kubectl patch tensorfusioncluster tensor-fusion --type=json -p='[
  {"op":"add","path":"/spec/gpuPools/0/specTemplate/componentConfig/hypervisor/enableVector","value":true},
  {"op":"add","path":"/spec/gpuPools/0/specTemplate/componentConfig/hypervisor/podTemplate/template/spec/containers/-","value":{"name":"vector"}}
]'

Both operations are required. With enableVector: true and only one container in the template, the controller discards the hypervisor container settings from the template.

The change alters the hypervisor Pod template, so the controller replaces the hypervisor Pods batch by batch. See Upgrade Components. When the rollout is done, each hypervisor Pod has two containers:

kubectl get pods -n tensor-fusion-sys -l tensor-fusion.ai/component=hypervisor

The chart has no value for this setting. Check it again after each helm upgrade.

Example queries

GPU usage over the last 10 minutes:

SELECT node, uuid,
       avg(compute_percentage) AS compute_pct,
       avg(memory_percentage)  AS vram_pct,
       max(temperature)        AS max_temp
FROM tf_gpu_usage
WHERE greptime_timestamp >= now() - '10m'::INTERVAL
GROUP BY node, uuid;

How much of each node is allocated:

SELECT node, pool,
       avg(allocated_tflops_percent) AS tflops_allocated_pct,
       avg(allocated_vram_percent)   AS vram_allocated_pct
FROM tf_node_metrics
WHERE greptime_timestamp >= now() - '10m'::INTERVAL
GROUP BY node, pool;

Workers that hit their compute limit in the last hour:

SELECT workload, worker, node, uuid,
       max(compute_throttled_cnt) - min(compute_throttled_cnt) AS throttled
FROM tf_worker_usage
WHERE greptime_timestamp >= now() - '1h'::INTERVAL
GROUP BY workload, worker, node, uuid
HAVING throttled > 0;

Scheduling failures per pool in the last hour:

SELECT pool,
       max(total_allocation_fail_cnt) - min(total_allocation_fail_cnt) AS failures
FROM tf_system_metrics
WHERE greptime_timestamp >= now() - '1h'::INTERVAL
GROUP BY pool;

Recent controller log lines that mention an error:

SELECT greptime_timestamp, pod, message
FROM tf_system_log
WHERE component = 'operator'
  AND greptime_timestamp >= now() - '30m'::INTERVAL
  AND message LIKE '%error%'
ORDER BY greptime_timestamp DESC
LIMIT 50;

component is the value of the Pod label tensor-fusion.ai/component: operator, hypervisor or worker.

If a query returns nothing

What you see	Cause	Fix
`tf_gpu_usage` and `tf_worker_usage` are empty or missing, other tables have rows	The hypervisor vector sidecar is off	Ship hypervisor metrics and logs
No `tf_*` table has recent rows	The controller's vector sidecar can't reach GreptimeDB	Check `kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c vector` and the `greptime.host` Helm value
Rows before a certain date are gone	They passed the retention period in `dynamicConfig.metricsTTL`	Raise the value and upgrade the release

Next steps

Setup Alerts: alert rules are SQL queries over these tables.
Optimize GPU Efficiency: use the numbers to right-size workloads and reclaim nodes.
Handbook: symptom-based troubleshooting.

Prerequisites

kubectl access to the namespace GreptimeDB runs in. By default the Helm chart installs a standalone GreptimeDB into the greptimedb namespace (greptime.installStandalone: true).
A mysql client, if you want to query from a terminal instead of the browser.

Connect to GreptimeDB

Use the built-in dashboard for ad-hoc queries, and the MySQL protocol for scripts.

Dashboard

kubectl port-forward -n greptimedb svc/greptimedb-standalone 4000:4000

Open http://localhost:4000/dashboard and select the public database.

MySQL client

kubectl port-forward -n greptimedb svc/greptimedb-standalone 4002:4002
mysql -h 127.0.0.1 -P 4002 -u root public

The standalone instance has no password. To confirm the connection, list the tables:

SHOW TABLES;

You should see the tf_* tables described below.

Tables

Table	Written by	One row describes
`tf_gpu_usage`	hypervisor	Actual usage of one physical GPU: compute and VRAM percentage, temperature, power, PCIe and NVLink traffic
`tf_worker_usage`	hypervisor	Actual usage of one worker on one GPU, plus counters for compute throttling and VRAM freeze and resume
`tf_worker_resources`	controller	Requests and limits of one worker, its QoS level and raw cost
`tf_gpu_metrics`	controller	Capacity, allocated and available TFLOPS and VRAM of one GPU
`tf_gpu_allocation`	controller	Which GPUs a Pod was allocated
`tf_node_metrics`	controller	Allocated TFLOPS and VRAM of one GPU node, as values and as percentages
`tf_pool_metrics`	controller	The same allocation figures for a whole pool
`tf_system_metrics`	controller	Counters per pool: workers, nodes, allocation successes and failures, scale-ups and scale-downs
`tf_system_log`	vector	One log line of a TensorFusion component

For every column, see the metrics reference.

Ship hypervisor metrics and logs

The controller Pod always runs a vector sidecar, so the controller tables and the controller's own logs arrive without extra setup.

To turn the sidecar on for the first pool of the chart-created cluster:

kubectl patch tensorfusioncluster tensor-fusion --type=json -p='[
  {"op":"add","path":"/spec/gpuPools/0/specTemplate/componentConfig/hypervisor/enableVector","value":true},
  {"op":"add","path":"/spec/gpuPools/0/specTemplate/componentConfig/hypervisor/podTemplate/template/spec/containers/-","value":{"name":"vector"}}
]'

Both operations are required. With enableVector: true and only one container in the template, the controller discards the hypervisor container settings from the template.

The change alters the hypervisor Pod template, so the controller replaces the hypervisor Pods batch by batch. See Upgrade Components. When the rollout is done, each hypervisor Pod has two containers:

kubectl get pods -n tensor-fusion-sys -l tensor-fusion.ai/component=hypervisor

The chart has no value for this setting. Check it again after each helm upgrade.

Example queries

GPU usage over the last 10 minutes:

SELECT node, uuid,
       avg(compute_percentage) AS compute_pct,
       avg(memory_percentage)  AS vram_pct,
       max(temperature)        AS max_temp
FROM tf_gpu_usage
WHERE greptime_timestamp >= now() - '10m'::INTERVAL
GROUP BY node, uuid;

How much of each node is allocated:

SELECT node, pool,
       avg(allocated_tflops_percent) AS tflops_allocated_pct,
       avg(allocated_vram_percent)   AS vram_allocated_pct
FROM tf_node_metrics
WHERE greptime_timestamp >= now() - '10m'::INTERVAL
GROUP BY node, pool;

Workers that hit their compute limit in the last hour:

SELECT workload, worker, node, uuid,
       max(compute_throttled_cnt) - min(compute_throttled_cnt) AS throttled
FROM tf_worker_usage
WHERE greptime_timestamp >= now() - '1h'::INTERVAL
GROUP BY workload, worker, node, uuid
HAVING throttled > 0;

Scheduling failures per pool in the last hour:

SELECT pool,
       max(total_allocation_fail_cnt) - min(total_allocation_fail_cnt) AS failures
FROM tf_system_metrics
WHERE greptime_timestamp >= now() - '1h'::INTERVAL
GROUP BY pool;

Recent controller log lines that mention an error:

SELECT greptime_timestamp, pod, message
FROM tf_system_log
WHERE component = 'operator'
  AND greptime_timestamp >= now() - '30m'::INTERVAL
  AND message LIKE '%error%'
ORDER BY greptime_timestamp DESC
LIMIT 50;

component is the value of the Pod label tensor-fusion.ai/component: operator, hypervisor or worker.

If a query returns nothing

What you see	Cause	Fix
`tf_gpu_usage` and `tf_worker_usage` are empty or missing, other tables have rows	The hypervisor vector sidecar is off	Ship hypervisor metrics and logs
No `tf_*` table has recent rows	The controller's vector sidecar can't reach GreptimeDB	Check `kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c vector` and the `greptime.host` Helm value
Rows before a certain date are gone	They passed the retention period in `dynamicConfig.metricsTTL`	Raise the value and upgrade the release

Next steps

Setup Alerts: alert rules are SQL queries over these tables.
Optimize GPU Efficiency: use the numbers to right-size workloads and reclaim nodes.
Handbook: symptom-based troubleshooting.

Prerequisites

Connect to GreptimeDB

Tables

Ship hypervisor metrics and logs

Example queries

If a query returns nothing

Next steps

Table of Contents

Query Metrics & Logs

Prerequisites

Connect to GreptimeDB

Tables

Ship hypervisor metrics and logs

Example queries

If a query returns nothing

Next steps

Table of Contents