Bring Your Own Cloud

A GPU pool can grow and shrink with demand in your own cloud account. This guide explains the three provisioning modes of a pool, which clouds each mode supports today, and how to configure them.

In short: on AWS EKS with Karpenter, keep the default AutoSelect mode and let a Karpenter NodePool own the GPU nodes. This is the path the Helm chart defaults to and the one covered by the end-to-end tests. The other two modes work from capacity targets you set on the pool. Treat them as preview and validate them in a test cluster first.

Provisioning modes

The mode is set per pool in nodeManagerConfig.provisioningMode.

A pool uses exactly one mode. Different pools in the same cluster can use different modes.

Prerequisites

• TensorFusion is installed. See Kubernetes Install.
• The controller runs with -enable-auto-expander, which is in the default command.
• For the two Karpenter options: Karpenter 1.6 or later with the nodeOverlay feature gate enabled, and an EC2NodeClass (or your provider's NodeClass) that can launch GPU nodes into the cluster.
• GPU nodes carry the label that initialGpuNodeLabelSelector selects, nvidia.com/gpu.present=true by default.

Option 1: AutoSelect with a Karpenter NodePool (recommended)

Karpenter owns the GPU nodes. TensorFusion selects them into the pool, asks Karpenter for another node when GPUs run out, and marks empty nodes for removal.

Step 1. Install Karpenter before TensorFusion

Enable the nodeOverlay feature gate in the Karpenter Helm values:

settings:
  featureGates:
    nodeOverlay: true

TensorFusion Pods request extended resources named tensor-fusion.ai/index_*. Karpenter only launches a node for a Pod if it believes some instance type offers what the Pod requests. The TensorFusion chart therefore creates a NodeOverlay named tensor-fusion-overlay that declares these resources, but only when the nodeoverlays.karpenter.sh CRD already exists at install time.

If you installed TensorFusion first, run helm upgrade on the TensorFusion release once after installing Karpenter. Then check:

kubectl get nodeoverlay tensor-fusion-overlay

Step 2. Create a NodePool for GPU nodes

Give the nodes the label the default pool selects on, and limit the instance types to GPU instances:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: tf-gpu
spec:
  template:
    metadata:
      labels:
        nvidia.com/gpu.present: "true"
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: <your-ec2-node-class>
      expireAfter: Never
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.xlarge", "g5.12xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
  # upper bound for everything this NodePool launches
  limits:
    nvidia.com/gpu: 8
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 5m

The pool created by the Helm chart already uses AutoSelect and selects nodes with nvidia.com/gpu.present=true, so no change to the TensorFusionCluster is needed.

Step 3. Verify

Once the NodePool has launched a GPU node, it appears in TensorFusion:

kubectl get nodeclaim,gpunode,gpu

Fill the existing GPUs, create one more TensorFusion Pod, and watch a new node arrive:

kubectl get events -A --field-selector reason=NodeExpansionCompleted
kubectl get nodeclaim -w

The Pod goes from Pending to Running when the node has joined and its GPUs are registered.

How it behaves

• Scale-up. When the scheduler rejects a TensorFusion Pod, the controller checks whether GPU capacity is the only thing missing. If so, it picks an existing GPU node owned by a Karpenter NodeClaim as a template and creates a new NodeClaim in the same NodePool. The NodePool's requirements apply, narrowed by the Pod's nodeSelector and node affinity on instance type, zone and region.
• Concurrency. At most dynamicConfig.maxInFlightNodes nodes are provisioned at once, 15 by default.
• Scale-down. The pool's compaction loop (nodeManagerConfig.nodeCompaction.period) labels empty nodes tensor-fusion.ai/should-delete=true. TensorFusion does not delete them. Karpenter consolidation does. Nodes younger than 5 minutes are left alone, and so are nodes whose removal would leave less free capacity than warmResources.
• Limits. capacityConfig.minResources and maxResources do not drive provisioning in this mode. Bound the size with limits on the NodePool.

Known limitations:

• The expander needs a template: at least one GPU node that Karpenter created. GPU nodes from a managed node group are not used as templates, and the Pod stays Pending with a NodeExpansionFailed event.
• With zero GPU nodes, the controller leaves the decision to Karpenter. What Karpenter launches then depends on your NodePools, so keep one GPU node running or test the cold start in your environment.
• Karpenter consolidation does not know how much GPU capacity TensorFusion has allocated on a node. Use consolidationPolicy: WhenEmpty unless you have verified WhenEmptyOrUnderutilized with your workloads.

Option 2: Karpenter mode

TensorFusion decides when to add or remove a node from the capacity targets on the pool, and uses Karpenter only to launch and terminate it. No NodePool is involved. TensorFusion creates a GPUNodeClaim, then a Karpenter NodeClaim with the same name.

Set computingVendor on the cluster and the provisioner on the pool:

apiVersion: tensor-fusion.ai/v1
kind: TensorFusionCluster
metadata:
  name: tensor-fusion
spec:
  computingVendor:
    name: karpenter
    type: karpenter
    authType: serviceAccountRole
    enable: true
    params:
      defaultRegion: us-east-1
  gpuPools:
    - name: elastic
      isDefault: false
      specTemplate:
        # componentConfig and schedulingConfigTemplate: copy them from your existing pool
        capacityConfig:
          minResources:
            tflops: "65"
            vram: "16Gi"
          warmResources:
            tflops: "0"
            vram: "0"
          maxResources:
            tflops: "500"
            vram: "128Gi"
        nodeManagerConfig:
          provisioningMode: Karpenter  // [!code highlight]
          nodeProvisioner:
            karpenterNodeClassRef:
              group: karpenter.k8s.aws
              kind: EC2NodeClass
              version: v1
              name: <your-ec2-node-class>
            gpuRequirements:
              - key: node.kubernetes.io/instance-type
                operator: In
                values: ["g4dn.xlarge", "g5.xlarge"]
              - key: karpenter.sh/capacity-type
                operator: In
                values: ["OnDemand"]
            gpuNodeLabels:
              nvidia.com/gpu.present: "true"
            # gpuTaints:
            #   - key: group
            #     value: gpu
            #     effect: NoSchedule
          nodeCompaction:
            period: 5m

Points to get right:

• type: karpenter needs no cloud credentials. The controller only talks to the Kubernetes API. It checks that it can list NodeClaim objects and reports the result in the cluster condition CloudVendorConnectionReady.
• karpenterNodeClassRef must point to an existing NodeClass. All four fields are required.
• gpuNodeLabels must include the label that initialGpuNodeLabelSelector selects. Otherwise the controller does not watch the new node.
• Capacity type values are OnDemand, Spot and Reserved. TensorFusion converts them to Karpenter's on-demand, spot and reserved. If Spot is in the list, it is preferred.
• Each NodeClaim requests nvidia.com/gpu. Karpenter waits for the node to advertise that resource, so the NVIDIA device plugin must run on the node.
• NodeClaim objects created this way carry karpenter.sh/do-not-disrupt: "true". Karpenter does not consolidate them. TensorFusion removes them when they are empty and no longer needed for warmResources.

Optional settings in computingVendor.params.extraParams:

Watch the progress:

kubectl get tensorfusioncluster tensor-fusion -o jsonpath='{.status.conditions}'
kubectl get gpunodeclaim,nodeclaim,gpunode -w

A GPUNodeClaim moves from Pending to Creating when its NodeClaim exists, and to Bound when the hypervisor on the new node is running.

How TensorFusion chooses the instance type

In Karpenter and Provisioned modes, TensorFusion computes the capacity gap in TFLOPS and VRAM and picks one instance type from a built-in catalog: the one that closes the gap at the lowest total hourly price. Only these gpuRequirements keys narrow the choice:

The catalog is static. It lists AWS GPU instance types with on-demand prices that do not vary by region, and spot is priced as on-demand. List the instance types you want explicitly instead of relying on the price comparison.

Option 3: Provisioned mode

TensorFusion calls the cloud's compute API itself. The scale-up and scale-down logic is the same as in Karpenter mode. The difference is who launches the instance, and that you describe the instance in a GPUNodeClass.

Supported clouds

computingVendor.type accepts many values in the CRD. The controller implements these:

Configure Alibaba Cloud

1. Give TensorFusion an access key that may create, describe and delete ECS instances. The chart keeps the key pair in the Secret tf-cloud-vendor-credentials (keys ak and sk) and mounts it into the controller at /tmp/secret.

   On a first install, pass --set cloudVendorCredentials.accessKey=<access-key-id> --set cloudVendorCredentials.secretKey=<access-key-secret>. The chart writes the Secret only when it does not exist, so on an existing install update the Secret directly:

   kubectl -n tensor-fusion-sys patch secret tf-cloud-vendor-credentials --type merge \
     -p '{"stringData":{"ak":"<access-key-id>","sk":"<access-key-secret>"}}'

2. Create a GPUNodeClass that describes the instance:

   apiVersion: tensor-fusion.ai/v1
   kind: GPUNodeClass
   metadata:
     name: tf-node-class
   spec:
     osImageSelectorTerms:
       - id: <image-id>
     securityGroupSelectorTerms:
       - id: <security-group-id>
     subnetSelectorTerms:
       - id: <vswitch-id>
     blockDeviceMappings:
       - deviceName: data
         ebs:
           volumeSize: "200"
           volumeType: cloud_essd
     tags:
       team: ai-platform
     userData: |
       #!/bin/bash
       # Join this node to the cluster with your usual bootstrap command, and
       # register it with the two node labels below.
       #   tensor-fusion.ai/node-provisioner=__GPU_NODE_RESOURCE_NAME__
       #   nvidia.com/gpu.present=true

   Selector terms are matched by id only. Before launching, the controller replaces __GPU_NODE_RESOURCE_NAME__ in userData with the name of the GPUNodeClaim. The node must register with the label tensor-fusion.ai/node-provisioner set to that name. That label is how TensorFusion matches a node to its claim.

3. Point the cluster and the pool at them:

   apiVersion: tensor-fusion.ai/v1
   kind: TensorFusionCluster
   metadata:
     name: tensor-fusion
   spec:
     computingVendor:
       name: alibaba
       type: alibaba
       authType: accessKey
       enable: true
       params:
         defaultRegion: cn-hangzhou
         accessKeyPath: /tmp/secret/ak
         secretKeyPath: /tmp/secret/sk
         extraParams:
           systemDiskSize: "100"
     gpuPools:
       - name: elastic
         isDefault: false
         specTemplate:
           # componentConfig and schedulingConfigTemplate: copy them from your existing pool
           capacityConfig:
             minResources:
               tflops: "65"
               vram: "15Gi"
             maxResources:
               tflops: "260"
               vram: "60Gi"
           nodeManagerConfig:
             provisioningMode: Provisioned  // [!code highlight]
             nodeProvisioner:
               nodeClass: tf-node-class
               gpuRequirements:
                 - key: node.kubernetes.io/instance-type
                   operator: In
                   values: ["ecs.gn6i-c8g1.2xlarge"]
                 - key: topology.kubernetes.io/zone
                   operator: In
                   values: ["cn-hangzhou-h"]
             nodeCompaction:
               period: 5m

Extra parameters for Alibaba Cloud in computingVendor.params.extraParams:

Instances are created with a public IP address, billed by traffic, with up to 100 Mbps outbound bandwidth.

Capacity settings

Karpenter and Provisioned modes act on capacityConfig of the pool:

Other behavior to know:

• New nodes are protected from compaction for 5 minutes.
• A node labeled tensor-fusion.ai/do-not-disrupt=true is never removed by compaction.
• nodeProvisioner.budget exists in the CRD, but the current controller does not enforce it. Use maxResources to cap spend.

Troubleshooting

kubectl logs -n tensor-fusion-sys deploy/tensor-fusion-sys-controller -c controller --tail=200

Next steps

• Production-Grade Deployment
• GPUPool schema and GPUNodeClass schema
• TensorFusionCluster schema, including computingVendor
• Helm Install Values