Skip to main content
Version: Next

AWS Setup

This guide covers setting up auto-provisioning capabilities for KAITO on Amazon Elastic Kubernetes Service (EKS). Auto-provisioning allows KAITO to automatically create GPU nodes when needed for your AI workloads.

Prerequisites​

  • An EKS cluster with KAITO workspace controller installed (see Installation)
  • AWS CLI for managing AWS resources
  • eksctl for EKS cluster management
  • kubectl configured to access your EKS cluster

Understanding Auto-Provisioning on AWS​

KAITO can use Karpenter to automatically provision GPU nodes. This controller:

  • Creates new GPU nodes when workspaces require specific instance types
  • Supports various AWS GPU instances (g4, g5, p3, p4 series, etc.)
  • Manages node lifecycle based on workload demands
  • Integrates with AWS IAM for secure access
note

Alternative: If you already have GPU nodes or manage them separately, use the preferred nodes approach in the Quick Start instead.

Set Up Auto-Provisioning​

Create EKS Cluster and install Karpenter​

Follow the instructions here to create an EKS cluster and install Karpenter.

Then update the KAITO workspace controller Helm chart values for AWS:

helm upgrade kaito-workspace --namespace kaito-workspace --set cloudProviderName=aws

Using Auto-Provisioning​

Once Karpenter is set up, you can create workspaces that automatically provision GPU nodes:

phi-4-workspace.yaml
apiVersion: kaito.sh/v1beta1
kind: Workspace
metadata:
  name: workspace-phi-4-mini
resource:
  instanceType: "g5.4xlarge" # Will trigger node creation
  labelSelector:
    matchLabels:
      apps: phi-4-mini
inference:
  preset:
    name: phi-4-mini-instruct

Apply the workspace:

kubectl apply -f phi-4-workspace.yaml

Supported AWS GPU Instance Types​

The GPU provisioner supports various AWS GPU SKUs, see supported options here.

For the complete list and specifications, see the AWS GPU instance documentation.

note

KAITO only supports NVIDIA GPUs with CUDA compute capability >= 8.0 (Ampere and newer, such as A10G, A100, L4, H100, H200). Older architectures such as NVIDIA T4 (Turing, 7.5), NVIDIA V100 (Volta, 7.0), NVIDIA M60 (Maxwell, 5.2), and NVIDIA K80 (Kepler, 3.7) are not supported. This means instance families such as g4dn.*, g5g.* (T4), p3.*, p3dn.* (V100), g3s.* (M60), and p2.* (K80) cannot be used with KAITO.

Clean Up​

See the Karpenter documentation for instructions on how to clean up your EKS cluster and remove Karpenter here.