Setup Alerts

Send TensorFusion alerts to your notification channel and tune or add alert rules

TensorFusion ships with alert rules for GPU saturation, allocation failures, low pool capacity and idle GPUs. The controller evaluates the rules and sends firing alerts to Alertmanager, which the Helm chart installs. Out of the box, Alertmanager has a receiver that notifies no one. This guide connects it to your channel, then shows how to change thresholds and add rules.

How alerting works

Each rule is a SQL query over the metrics tables in GreptimeDB. The controller runs it every evaluationInterval.
Every row the query returns is one alert candidate. A candidate becomes a firing alert after it shows up in consecutiveCount evaluations in a row.
The controller posts firing and resolved alerts to Alertmanager. Alertmanager groups them, removes duplicates and notifies your receivers.

Rules live in the Helm value dynamicConfig.alertRules. Alertmanager's configuration lives in alert.alertManagerConfig.

Prerequisites

TensorFusion installed with Helm. The chart enables rule evaluation (-enable-alert in controller.command) and Alertmanager (alert.enabled: true) by default.
The values file of your release. Export it with helm -n tensor-fusion-sys get values tensor-fusion-sys -o yaml > values.yaml.
An endpoint that Alertmanager can notify: a webhook URL, Slack, PagerDuty, email, or any other Alertmanager receiver.
For rules on actual GPU and worker usage: the hypervisor vector sidecar. See Ship hypervisor metrics and logs. The rule table marks which rules need it.

Step 1. Configure receivers

Create alert-values.yaml. The example sends every alert to a webhook and routes P0 alerts to a second one:

alert:
  alertManagerConfig:
    global: {}
    route:
      receiver: ops-webhook
      group_by: [alertname, instance]
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 3h
      routes:
        - matchers: ['severity="P0"']
          receiver: oncall-webhook
    receivers:
      - name: ops-webhook
        webhook_configs:
          - url: https://alerts.example.com/tensor-fusion
      - name: oncall-webhook
        webhook_configs:
          - url: https://alerts.example.com/oncall
    templates:
      - /etc/alertmanager/*.tmpl

The content of alertManagerConfig is written as is to alertmanager.yml, so any valid Alertmanager configuration works. It is stored in a ConfigMap. Keep that in mind before you put an API key or SMTP password in it.

Every alert carries these labels and annotations, which you can use in routes and templates:

Kind	Key	Value
Label	`alertname`	Rule name, for example `GPUVRAMFull`
Label	`severity`	The rule's `severity`: `P0`, `P1` or `P2` in the built-in rules
Label	`job`	`tensor-fusion`
Label	`instance`	The rule's `alertTargetInstance`, for example a GPU UUID, node name or pool name
Annotation	`summary`, `description`	Rendered from the rule's templates
Annotation	`runbook_url`	The rule's `runbookURL`

Step 2. Adjust or add rules (optional)

dynamicConfig.alertRules is a list, and Helm replaces lists as a whole. To change one rule, copy the complete list from the chart's values.yaml into your values file and edit it there. If you copy only one rule, all other rules disappear.

A rule looks like this:

dynamicConfig:
  alertRules:
    - name: GPUVRAMFull
      query: |
        SELECT node, pool, uuid, avg(memory_percentage) AS memory_used
        FROM tf_gpu_usage
        WHERE memory_percentage > {{ .Threshold }} AND {{ .Conditions }}
        GROUP BY node, pool, uuid
      threshold: 97
      evaluationInterval: 30s
      consecutiveCount: 2
      severity: P1
      summary: "GPU VRAM Full, used {{ .memory_used }}% on {{ .node }} {{ .uuid }}"
      description: "GPU {{ .uuid }} on Node {{ .node }} in Pool {{ .pool }} has VRAM usage above {{ .Threshold }}%"
      alertTargetInstance: "{{ .uuid }}"
      runbookURL: "https://tensor-fusion.ai/docs/tensor-fusion-engine/troubleshooting/handbook"
    # ... all other rules

Field	Meaning
`name`	Rule name. Becomes the `alertname` label
`query`	SQL query, written as a Go template. `{{ .Threshold }}` is the rule's threshold. `{{ .Conditions }}` expands to a time filter that covers the last `evaluationInterval`, in the form `ts >= now() - '30s'::INTERVAL`. `{{ .Severity }}` and `{{ .Name }}` are also available
`threshold`	A number passed into the query. The query does the comparison, not the controller
`evaluationInterval`	How often the query runs, as a Go duration such as `30s` or `5m`. It is also the time window of `{{ .Conditions }}`
`consecutiveCount`	How many evaluations in a row must return the row before the alert fires
`severity`	Free text. Becomes the `severity` label
`summary`, `description`	Go templates. Every column of the returned row is available by name, plus `{{ .Threshold }}`
`alertTargetInstance`	Go template for the `instance` label. It must identify the object the alert is about, because Alertmanager uses the labels to tell alerts apart
`runbookURL`	Link attached to the alert

Test a new query in GreptimeDB first, with the template variables replaced by real values. See Query Metrics & Logs.

Step 3. Apply the change

helm upgrade tensor-fusion-sys tensor-fusion \
  --repo https://download.tensor-fusion.ai \
  --namespace tensor-fusion-sys \
  --version <chart-version> \
  -f values.yaml -f alert-values.yaml

# Alertmanager reads its configuration at startup
kubectl -n tensor-fusion-sys rollout restart statefulset/tensor-fusion-sys-alert-manager

# Only if you changed dynamicConfig.alertRules
kubectl -n tensor-fusion-sys rollout restart deploy/tensor-fusion-sys-controller

Use the chart version you already run, so that this command changes nothing else. helm -n tensor-fusion-sys list shows it.

Verify

Open the Alertmanager UI:
```
kubectl -n tensor-fusion-sys port-forward svc/alert-manager 9093:9093
```
Browse to http://localhost:9093. The Status page shows the configuration you applied.

Send a test alert through the same API the controller uses:

curl -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"P1","job":"tensor-fusion","instance":"test"},"annotations":{"summary":"Test alert from TensorFusion setup"}}]'

The alert appears on the Alerts page, and your receiver gets a notification after group_wait.

Check that the controller evaluates the rules without errors:
```
kubectl -n tensor-fusion-sys logs deploy/tensor-fusion-sys-controller -c controller | grep -i "failed to evaluate rule"
```
No output means every query ran. Only the leader evaluates rules, so with several controller replicas run the command against each Pod.

Built-in rules

Rule	Fires when	Severity	Needs hypervisor metrics
`WorkerTFlopsThrottled`	A worker was throttled at its compute limit in 3 consecutive 15s windows	P1	Yes
`WorkerVRAMSwitchCountIncreasing`	A worker's VRAM was frozen and resumed within 2 minutes	P1	Yes
`WorkerAllocationFailed`	A pool had at least one failed GPU allocation in the last 30s	P1	No
`GPUVRAMFull`	VRAM usage of a GPU is above 97% in 2 consecutive 30s windows	P1	Yes
`GPUTFlopsFull`	Compute usage of a GPU is above 97% in 4 consecutive 30s windows	P1	Yes
`GPUTemperatureHigh`	A GPU is hotter than 90°C in 3 consecutive 30s windows	P1	Yes
`NodeTFlopsAllocationCritical`	Less than 5% of a node's TFLOPS is unallocated for 2 minutes	P0	No
`NodeTFlopsAllocationWarning`	Less than 10% of a node's TFLOPS is unallocated for 2 minutes	P1	No
`PoolTotalTFlopsAllocationCritical`	Less than 5% of a pool's TFLOPS is unallocated for 2 minutes	P0	No
`PoolTotalTFlopsAllocationWarning`	Less than 10% of a pool's TFLOPS is unallocated for 2 minutes	P1	No
`NodeVRAMAllocationCritical`	Less than 5% of a node's VRAM is unallocated for 2 minutes	P1	No
`NodeVRAMAllocationWarning`	Less than 10% of a node's VRAM is unallocated for 2 minutes	P1	No
`PoolVRAMAllocationWarning`	Less than 10% of a pool's VRAM is unallocated for 2 minutes	P1	No
`EmptyGPU`	A GPU node reported no worker usage in 2 consecutive 5-minute windows	P2	Yes
`IdleGPU`	A GPU stays below 1% compute and 5% VRAM usage in 3 consecutive 10-minute windows	P2	Yes

The chart's values.yaml is the source of truth for the queries and thresholds.

Use your own Alertmanager

To send alerts to an Alertmanager you already run, turn the bundled one off and pass its address to the controller. controller.command is a list, so repeat the default flags and append yours:

alert:
  enabled: false
controller:
  command:
    - /manager
    - -metrics-bind-address
    - :9000
    - -leader-elect
    - -enable-auto-scale
    - -enable-alert
    - -enable-auto-expander
    - -alert-manager-addr=http://alertmanager.monitoring.svc.cluster.local:9093

The address must start with http:// or https://. The controller appends /api/v2/alerts itself.

The same flag is needed when TensorFusion is not installed in the tensor-fusion-sys namespace. The default address is http://alert-manager.tensor-fusion-sys.svc.cluster.local:9093, so in any other namespace the controller cannot reach the bundled Alertmanager until you set it.

To turn alerting off completely, also remove -enable-alert from the list.

Limits

Rule state is kept in the controller's memory. After a controller restart, the consecutive counts start from zero, and alerts that were firing are not explicitly resolved. Alertmanager expires them on its own.
The controller picks up rule changes only at startup when installed with the chart. Restart it after every change, as in step 3.
The hypervisor writes usage metrics about once a minute. Rules on tf_gpu_usage or tf_worker_usage with a shorter evaluationInterval can see windows without any row. If such a rule doesn't fire when you expect it to, raise its evaluationInterval to 1m or more.
Without hypervisor metrics, the rules marked "Yes" above do not work. EmptyGPU can then report every GPU node. Enable the sidecar or remove those rules.
Rules only query GreptimeDB. Events such as a crashing hypervisor Pod are not covered. Watch Kubernetes events and Pod status with your cluster monitoring.

Next steps

Query Metrics & Logs to explore the tables a rule can query.
Metrics reference for every column.
Handbook for what to do when an alert fires.

How alerting works

Each rule is a SQL query over the metrics tables in GreptimeDB. The controller runs it every evaluationInterval.
Every row the query returns is one alert candidate. A candidate becomes a firing alert after it shows up in consecutiveCount evaluations in a row.
The controller posts firing and resolved alerts to Alertmanager. Alertmanager groups them, removes duplicates and notifies your receivers.

Rules live in the Helm value dynamicConfig.alertRules. Alertmanager's configuration lives in alert.alertManagerConfig.

Prerequisites

TensorFusion installed with Helm. The chart enables rule evaluation (-enable-alert in controller.command) and Alertmanager (alert.enabled: true) by default.
The values file of your release. Export it with helm -n tensor-fusion-sys get values tensor-fusion-sys -o yaml > values.yaml.
An endpoint that Alertmanager can notify: a webhook URL, Slack, PagerDuty, email, or any other Alertmanager receiver.
For rules on actual GPU and worker usage: the hypervisor vector sidecar. See Ship hypervisor metrics and logs. The rule table marks which rules need it.

Step 1. Configure receivers

Create alert-values.yaml. The example sends every alert to a webhook and routes P0 alerts to a second one:

alert:
  alertManagerConfig:
    global: {}
    route:
      receiver: ops-webhook
      group_by: [alertname, instance]
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 3h
      routes:
        - matchers: ['severity="P0"']
          receiver: oncall-webhook
    receivers:
      - name: ops-webhook
        webhook_configs:
          - url: https://alerts.example.com/tensor-fusion
      - name: oncall-webhook
        webhook_configs:
          - url: https://alerts.example.com/oncall
    templates:
      - /etc/alertmanager/*.tmpl

Every alert carries these labels and annotations, which you can use in routes and templates:

Kind	Key	Value
Label	`alertname`	Rule name, for example `GPUVRAMFull`
Label	`severity`	The rule's `severity`: `P0`, `P1` or `P2` in the built-in rules
Label	`job`	`tensor-fusion`
Label	`instance`	The rule's `alertTargetInstance`, for example a GPU UUID, node name or pool name
Annotation	`summary`, `description`	Rendered from the rule's templates
Annotation	`runbook_url`	The rule's `runbookURL`

Step 2. Adjust or add rules (optional)

A rule looks like this:

dynamicConfig:
  alertRules:
    - name: GPUVRAMFull
      query: |
        SELECT node, pool, uuid, avg(memory_percentage) AS memory_used
        FROM tf_gpu_usage
        WHERE memory_percentage > {{ .Threshold }} AND {{ .Conditions }}
        GROUP BY node, pool, uuid
      threshold: 97
      evaluationInterval: 30s
      consecutiveCount: 2
      severity: P1
      summary: "GPU VRAM Full, used {{ .memory_used }}% on {{ .node }} {{ .uuid }}"
      description: "GPU {{ .uuid }} on Node {{ .node }} in Pool {{ .pool }} has VRAM usage above {{ .Threshold }}%"
      alertTargetInstance: "{{ .uuid }}"
      runbookURL: "https://tensor-fusion.ai/docs/tensor-fusion-engine/troubleshooting/handbook"
    # ... all other rules

Field	Meaning
`name`	Rule name. Becomes the `alertname` label
`query`	SQL query, written as a Go template. `{{ .Threshold }}` is the rule's threshold. `{{ .Conditions }}` expands to a time filter that covers the last `evaluationInterval`, in the form `ts >= now() - '30s'::INTERVAL`. `{{ .Severity }}` and `{{ .Name }}` are also available
`threshold`	A number passed into the query. The query does the comparison, not the controller
`evaluationInterval`	How often the query runs, as a Go duration such as `30s` or `5m`. It is also the time window of `{{ .Conditions }}`
`consecutiveCount`	How many evaluations in a row must return the row before the alert fires
`severity`	Free text. Becomes the `severity` label
`summary`, `description`	Go templates. Every column of the returned row is available by name, plus `{{ .Threshold }}`
`alertTargetInstance`	Go template for the `instance` label. It must identify the object the alert is about, because Alertmanager uses the labels to tell alerts apart
`runbookURL`	Link attached to the alert

Test a new query in GreptimeDB first, with the template variables replaced by real values. See Query Metrics & Logs.

Step 3. Apply the change

helm upgrade tensor-fusion-sys tensor-fusion \
  --repo https://download.tensor-fusion.ai \
  --namespace tensor-fusion-sys \
  --version <chart-version> \
  -f values.yaml -f alert-values.yaml

# Alertmanager reads its configuration at startup
kubectl -n tensor-fusion-sys rollout restart statefulset/tensor-fusion-sys-alert-manager

# Only if you changed dynamicConfig.alertRules
kubectl -n tensor-fusion-sys rollout restart deploy/tensor-fusion-sys-controller

Use the chart version you already run, so that this command changes nothing else. helm -n tensor-fusion-sys list shows it.

Verify

Open the Alertmanager UI:
```
kubectl -n tensor-fusion-sys port-forward svc/alert-manager 9093:9093
```
Browse to http://localhost:9093. The Status page shows the configuration you applied.

Send a test alert through the same API the controller uses:

curl -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"P1","job":"tensor-fusion","instance":"test"},"annotations":{"summary":"Test alert from TensorFusion setup"}}]'

The alert appears on the Alerts page, and your receiver gets a notification after group_wait.

Check that the controller evaluates the rules without errors:
```
kubectl -n tensor-fusion-sys logs deploy/tensor-fusion-sys-controller -c controller | grep -i "failed to evaluate rule"
```
No output means every query ran. Only the leader evaluates rules, so with several controller replicas run the command against each Pod.

Built-in rules

Rule	Fires when	Severity	Needs hypervisor metrics
`WorkerTFlopsThrottled`	A worker was throttled at its compute limit in 3 consecutive 15s windows	P1	Yes
`WorkerVRAMSwitchCountIncreasing`	A worker's VRAM was frozen and resumed within 2 minutes	P1	Yes
`WorkerAllocationFailed`	A pool had at least one failed GPU allocation in the last 30s	P1	No
`GPUVRAMFull`	VRAM usage of a GPU is above 97% in 2 consecutive 30s windows	P1	Yes
`GPUTFlopsFull`	Compute usage of a GPU is above 97% in 4 consecutive 30s windows	P1	Yes
`GPUTemperatureHigh`	A GPU is hotter than 90°C in 3 consecutive 30s windows	P1	Yes
`NodeTFlopsAllocationCritical`	Less than 5% of a node's TFLOPS is unallocated for 2 minutes	P0	No
`NodeTFlopsAllocationWarning`	Less than 10% of a node's TFLOPS is unallocated for 2 minutes	P1	No
`PoolTotalTFlopsAllocationCritical`	Less than 5% of a pool's TFLOPS is unallocated for 2 minutes	P0	No
`PoolTotalTFlopsAllocationWarning`	Less than 10% of a pool's TFLOPS is unallocated for 2 minutes	P1	No
`NodeVRAMAllocationCritical`	Less than 5% of a node's VRAM is unallocated for 2 minutes	P1	No
`NodeVRAMAllocationWarning`	Less than 10% of a node's VRAM is unallocated for 2 minutes	P1	No
`PoolVRAMAllocationWarning`	Less than 10% of a pool's VRAM is unallocated for 2 minutes	P1	No
`EmptyGPU`	A GPU node reported no worker usage in 2 consecutive 5-minute windows	P2	Yes
`IdleGPU`	A GPU stays below 1% compute and 5% VRAM usage in 3 consecutive 10-minute windows	P2	Yes

The chart's values.yaml is the source of truth for the queries and thresholds.

Use your own Alertmanager

To send alerts to an Alertmanager you already run, turn the bundled one off and pass its address to the controller. controller.command is a list, so repeat the default flags and append yours:

alert:
  enabled: false
controller:
  command:
    - /manager
    - -metrics-bind-address
    - :9000
    - -leader-elect
    - -enable-auto-scale
    - -enable-alert
    - -enable-auto-expander
    - -alert-manager-addr=http://alertmanager.monitoring.svc.cluster.local:9093

The address must start with http:// or https://. The controller appends /api/v2/alerts itself.

To turn alerting off completely, also remove -enable-alert from the list.

Limits

Rule state is kept in the controller's memory. After a controller restart, the consecutive counts start from zero, and alerts that were firing are not explicitly resolved. Alertmanager expires them on its own.
The controller picks up rule changes only at startup when installed with the chart. Restart it after every change, as in step 3.
The hypervisor writes usage metrics about once a minute. Rules on tf_gpu_usage or tf_worker_usage with a shorter evaluationInterval can see windows without any row. If such a rule doesn't fire when you expect it to, raise its evaluationInterval to 1m or more.
Without hypervisor metrics, the rules marked "Yes" above do not work. EmptyGPU can then report every GPU node. Enable the sidecar or remove those rules.
Rules only query GreptimeDB. Events such as a crashing hypervisor Pod are not covered. Watch Kubernetes events and Pod status with your cluster monitoring.

Next steps

Query Metrics & Logs to explore the tables a rule can query.
Metrics reference for every column.
Handbook for what to do when an alert fires.

How alerting works

Prerequisites

Step 1. Configure receivers

Step 2. Adjust or add rules (optional)

Step 3. Apply the change

Verify

Built-in rules

Use your own Alertmanager

Limits

Next steps

Table of Contents

Setup Alerts

How alerting works

Prerequisites

Step 1. Configure receivers

Step 2. Adjust or add rules (optional)

Step 3. Apply the change

Verify

Built-in rules

Use your own Alertmanager

Limits

Next steps

Table of Contents