Kubernetes Events

TensorFusion records Kubernetes events on the objects it manages: Pods, GPUPool, GPUNode, TensorFusionWorkload and TensorFusionConnection. This page lists every event reason, its type, and what it means, so you can tell at a glance why a Pod is pending or why a node was added or removed.

View events

Events are attached to the object they describe. Pass the object kind and name to kubectl:

# Events of one Pod
kubectl describe pod <pod-name> -n <namespace>

# Events of a GPUPool (cluster-scoped)
kubectl events --for gpupool/<pool-name>

# All warnings in a namespace, newest last
kubectl get events -n <namespace> --field-selector type=Warning --sort-by=.lastTimestamp

Kubernetes keeps events for one hour by default. For longer history, collect them with your logging stack.

Pod events

Scheduling and GPU allocation

Reason	Type	Meaning
`PreScheduleDone`	Normal	The scheduler found GPU nodes that fit the Pod and reserved resources. The message gives the number of candidate nodes.
`GPUDeviceAllocated`	Normal	The GPU device IDs were written to the Pod. The Pod is about to be bound to its node.
`PodIndexAllocationPending`	Normal	The Pod's device index is still held by another Pod on the same node. It is patched in once that Pod releases it.
`ScheduleWithNativeGPU`	Normal	The Pod is not managed by TensorFusion and was scheduled with native GPU resources. Appears only during progressive migration from the NVIDIA device plugin.
`GPUQuotaOrCapacityNotEnough`	Warning	No GPU has enough free TFLOPS or VRAM, or the namespace's `GPUResourceQuota` is exhausted. The message contains the detailed reason.
`PartitionTemplateMatchFailed`	Warning	In `partitioned` isolation mode, no partition template matches the requested resources.
`GPUDeviceAllocatedFailed`	Warning	The scheduler picked GPUs but could not write the device IDs to the Pod.
`ContainerGPUAllocationFailed`	Warning	The scheduler could not map the allocated GPUs to the Pod's containers.
`CommitFailed`	Warning	The GPU allocation could not be committed after retries. This scheduling attempt fails and the scheduler retries the Pod.

GPU topology

Reason	Type	Meaning
`GPUInterconnectTopologyFallback`	Normal	Topology data is incomplete on some nodes. The scheduler fell back to NUMA-level or unknown topology for them.
`GPUInterconnectTopologyUnsatisfied`	Warning	The Pod requires a GPU topology in `hard` mode and no node satisfies it.

Gang scheduling

Reason	Type	Meaning
`GangWaitingForMembers`	Normal	The Pod passed scheduling and is waiting for the rest of its group. The message shows `ready/required`.
`GangScheduled`	Normal	Enough members are ready and the group is scheduled together.
`GangSchedulingTimeout`	Warning	The group did not reach its required member count before the timeout. The message shows how many members were ready.
`GangStrictRejected`	Warning	One member could not be scheduled, so the whole group was rejected (strict mode).

Node expansion

These events appear on a pending Pod when the controller runs with -enable-auto-expander.

Reason	Type	Meaning
`NodeExpansionCheck`	Normal or Warning	Result of checking whether a new node would help. The message says which case applies: the Pod fits existing or in-flight capacity, expansion is delegated to Karpenter, the failure is not GPU-related, no GPU node can host the Pod, or the Pod is still unscheduled 10 minutes after expansion.
`NodeExpansionCompleted`	Normal	A `GPUNodeClaim` or Karpenter `NodeClaim` was created for the Pod.
`NodeExpansionFailed`	Warning	No node type can fit the Pod, or creating the node claim failed.
`NodeExpansionCandidateFailed`	Warning	A Karpenter `NodeClaim` failed for lack of cloud capacity. The expander tries the next preference.
`NodeExpansionCandidatesExhausted`	Warning	Every expansion preference failed. The Pod waits for the next scheduling attempt.

GPUPool events

Capacity

Reason	Type	Meaning
`ManagedNodeCreated`	Normal	A cloud instance was created for the pool. The message contains the instance ID and private IP.
`Compaction`	Normal	A node is empty and removing it keeps the pool's warm-up capacity, so the pool is terminating it.
`MaxResourceConstraintReached`	Warning	The pool reached `capacityConfig.maxResources` and cannot scale up.

Defragmentation

Recorded when node compaction has defragmentation enabled.

Reason	Type	Meaning
`DefragStarted`	Normal	A defragmentation round started.
`DefragPodEvicted`	Normal	A Pod was evicted from a source node so it can be rescheduled onto a fuller node.
`DefragFinished`	Normal	The round finished. The message summarizes candidate nodes, evicted Pods, failures and skips.
`DefragSkipUnschedulable`	Normal	A node was skipped because its Pods would not fit elsewhere.
`DefragSkipPDBBlocked`	Normal	A node was skipped because a PodDisruptionBudget does not allow the eviction now.
`DefragBlockedBySourceNode`	Normal	The round is paused: a node from an earlier round is still draining.
`DefragBlockedByEvictedPod`	Normal	The round is paused: a Pod evicted earlier has not been rescheduled yet.
`DefragSkipMissingPDB`	Warning	A node was skipped because one of its Pods has no PodDisruptionBudget. Add a PDB to let defragmentation move it.
`DefragPodEvictFailed`	Warning	Evicting a Pod, or marking the node as a defragmentation source, failed.
`DefragNodeEvictSkip`	Warning	After an eviction failure the node was added to the evict-skip list, so later rounds pass over it.
`DefragAbortNode`	Warning	Defragmentation of a node was aborted, for example because the scheduling simulation returned an error.
`DefragConfigInvalid`	Warning	The defragmentation settings are invalid. Defragmentation is disabled until they are fixed.

GPUNode events

Reason	Type	Object	Meaning
`HypervisorCrashLoopRecreate`	Warning	GPUNode	The hypervisor container restarted more times than `dynamicConfig.hypervisorMaxCrashCount` (default 5). The controller deleted the Pod to recreate it.
`DriverProbeFailed`	Warning	Job	The driver probe Job for a node failed all its retries. Check the GPU driver on that node.

TensorFusionWorkload events

Reason	Type	Meaning
`ResourceScaledUp`	Normal	The autoscaler raised the workers' TFLOPS or VRAM. The message shows the old and new values.
`ResourceScaledDown`	Normal	The autoscaler lowered the workers' TFLOPS or VRAM.

TensorFusionConnection events

Reason	Type	Meaning
`WorkerSelected`	Normal	A worker was chosen for the connection.
`ConnectionReady`	Normal	The worker is running. The message contains the connection URL.
`WorkerSelectionFailed`	Warning	No worker could be selected for the connection.

Next steps

Troubleshooting handbook maps common symptoms to fixes.
Metrics reference lists the counters behind these events, such as total_allocation_fail_cnt.

View events

Events are attached to the object they describe. Pass the object kind and name to kubectl:

# Events of one Pod
kubectl describe pod <pod-name> -n <namespace>

# Events of a GPUPool (cluster-scoped)
kubectl events --for gpupool/<pool-name>

# All warnings in a namespace, newest last
kubectl get events -n <namespace> --field-selector type=Warning --sort-by=.lastTimestamp

Kubernetes keeps events for one hour by default. For longer history, collect them with your logging stack.

Pod events

Scheduling and GPU allocation

Reason	Type	Meaning
`PreScheduleDone`	Normal	The scheduler found GPU nodes that fit the Pod and reserved resources. The message gives the number of candidate nodes.
`GPUDeviceAllocated`	Normal	The GPU device IDs were written to the Pod. The Pod is about to be bound to its node.
`PodIndexAllocationPending`	Normal	The Pod's device index is still held by another Pod on the same node. It is patched in once that Pod releases it.
`ScheduleWithNativeGPU`	Normal	The Pod is not managed by TensorFusion and was scheduled with native GPU resources. Appears only during progressive migration from the NVIDIA device plugin.
`GPUQuotaOrCapacityNotEnough`	Warning	No GPU has enough free TFLOPS or VRAM, or the namespace's `GPUResourceQuota` is exhausted. The message contains the detailed reason.
`PartitionTemplateMatchFailed`	Warning	In `partitioned` isolation mode, no partition template matches the requested resources.
`GPUDeviceAllocatedFailed`	Warning	The scheduler picked GPUs but could not write the device IDs to the Pod.
`ContainerGPUAllocationFailed`	Warning	The scheduler could not map the allocated GPUs to the Pod's containers.
`CommitFailed`	Warning	The GPU allocation could not be committed after retries. This scheduling attempt fails and the scheduler retries the Pod.

GPU topology

Reason	Type	Meaning
`GPUInterconnectTopologyFallback`	Normal	Topology data is incomplete on some nodes. The scheduler fell back to NUMA-level or unknown topology for them.
`GPUInterconnectTopologyUnsatisfied`	Warning	The Pod requires a GPU topology in `hard` mode and no node satisfies it.

Gang scheduling

Reason	Type	Meaning
`GangWaitingForMembers`	Normal	The Pod passed scheduling and is waiting for the rest of its group. The message shows `ready/required`.
`GangScheduled`	Normal	Enough members are ready and the group is scheduled together.
`GangSchedulingTimeout`	Warning	The group did not reach its required member count before the timeout. The message shows how many members were ready.
`GangStrictRejected`	Warning	One member could not be scheduled, so the whole group was rejected (strict mode).

Node expansion

These events appear on a pending Pod when the controller runs with -enable-auto-expander.

Reason	Type	Meaning
`NodeExpansionCheck`	Normal or Warning	Result of checking whether a new node would help. The message says which case applies: the Pod fits existing or in-flight capacity, expansion is delegated to Karpenter, the failure is not GPU-related, no GPU node can host the Pod, or the Pod is still unscheduled 10 minutes after expansion.
`NodeExpansionCompleted`	Normal	A `GPUNodeClaim` or Karpenter `NodeClaim` was created for the Pod.
`NodeExpansionFailed`	Warning	No node type can fit the Pod, or creating the node claim failed.
`NodeExpansionCandidateFailed`	Warning	A Karpenter `NodeClaim` failed for lack of cloud capacity. The expander tries the next preference.
`NodeExpansionCandidatesExhausted`	Warning	Every expansion preference failed. The Pod waits for the next scheduling attempt.

GPUPool events

Capacity

Reason	Type	Meaning
`ManagedNodeCreated`	Normal	A cloud instance was created for the pool. The message contains the instance ID and private IP.
`Compaction`	Normal	A node is empty and removing it keeps the pool's warm-up capacity, so the pool is terminating it.
`MaxResourceConstraintReached`	Warning	The pool reached `capacityConfig.maxResources` and cannot scale up.

Defragmentation

Recorded when node compaction has defragmentation enabled.

Reason	Type	Meaning
`DefragStarted`	Normal	A defragmentation round started.
`DefragPodEvicted`	Normal	A Pod was evicted from a source node so it can be rescheduled onto a fuller node.
`DefragFinished`	Normal	The round finished. The message summarizes candidate nodes, evicted Pods, failures and skips.
`DefragSkipUnschedulable`	Normal	A node was skipped because its Pods would not fit elsewhere.
`DefragSkipPDBBlocked`	Normal	A node was skipped because a PodDisruptionBudget does not allow the eviction now.
`DefragBlockedBySourceNode`	Normal	The round is paused: a node from an earlier round is still draining.
`DefragBlockedByEvictedPod`	Normal	The round is paused: a Pod evicted earlier has not been rescheduled yet.
`DefragSkipMissingPDB`	Warning	A node was skipped because one of its Pods has no PodDisruptionBudget. Add a PDB to let defragmentation move it.
`DefragPodEvictFailed`	Warning	Evicting a Pod, or marking the node as a defragmentation source, failed.
`DefragNodeEvictSkip`	Warning	After an eviction failure the node was added to the evict-skip list, so later rounds pass over it.
`DefragAbortNode`	Warning	Defragmentation of a node was aborted, for example because the scheduling simulation returned an error.
`DefragConfigInvalid`	Warning	The defragmentation settings are invalid. Defragmentation is disabled until they are fixed.

GPUNode events

Reason	Type	Object	Meaning
`HypervisorCrashLoopRecreate`	Warning	GPUNode	The hypervisor container restarted more times than `dynamicConfig.hypervisorMaxCrashCount` (default 5). The controller deleted the Pod to recreate it.
`DriverProbeFailed`	Warning	Job	The driver probe Job for a node failed all its retries. Check the GPU driver on that node.

TensorFusionWorkload events

Reason	Type	Meaning
`ResourceScaledUp`	Normal	The autoscaler raised the workers' TFLOPS or VRAM. The message shows the old and new values.
`ResourceScaledDown`	Normal	The autoscaler lowered the workers' TFLOPS or VRAM.

TensorFusionConnection events

Reason	Type	Meaning
`WorkerSelected`	Normal	A worker was chosen for the connection.
`ConnectionReady`	Normal	The worker is running. The message contains the connection URL.
`WorkerSelectionFailed`	Warning	No worker could be selected for the connection.

Next steps

Troubleshooting handbook maps common symptoms to fixes.
Metrics reference lists the counters behind these events, such as total_allocation_fail_cnt.

View events

Pod events

Scheduling and GPU allocation

GPU topology

Gang scheduling

Node expansion

GPUPool events

Capacity

Defragmentation

GPUNode events

TensorFusionWorkload events

TensorFusionConnection events

Next steps

Table of Contents

Kubernetes Events

View events

Pod events

Scheduling and GPU allocation

GPU topology

Gang scheduling

Node expansion

GPUPool events

Capacity

Defragmentation

GPUNode events

TensorFusionWorkload events

TensorFusionConnection events

Next steps

Table of Contents