AWS Setup

This guide covers setting up auto-provisioning capabilities for KAITO on Amazon Elastic Kubernetes Service (EKS). Auto-provisioning allows KAITO to automatically create GPU nodes when needed for your AI workloads.

Prerequisites

• An EKS cluster with KAITO workspace controller installed (see Installation)
• AWS CLI for managing AWS resources
• eksctl for EKS cluster management
• kubectl configured to access your EKS cluster

Understanding Auto-Provisioning on AWS

KAITO can use Karpenter to automatically provision GPU nodes. This controller:

• Creates new GPU nodes when workspaces require specific instance types
• Supports various AWS GPU instances (g4, g5, p3, p4 series, etc.)
• Manages node lifecycle based on workload demands
• Integrates with AWS IAM for secure access

note

Alternative: If you already have GPU nodes or manage them separately, use the preferred nodes approach in the Quick Start instead.

Setup Auto-Provisioning

Create EKS Cluster and install Karpenter

Follow the instructions here to create an EKS cluster and install Karpenter.

Then update the KAITO workspace controller Helm chart values for AWS:

helm update kaito-workspace --namespace kaito-workspace --set cloudProviderName=aws

Using Auto-Provisioning

Once Karpenter is set up, you can create workspaces that automatically provision GPU nodes:

phi-4-workspace.yaml

apiVersion: kaito.sh/v1beta1
kind: Workspace
metadata:
  name: workspace-phi-4-mini
resource:
  instanceType: "g5.4xlarge" # Will trigger node creation
  labelSelector:
    matchLabels:
      apps: phi-4-mini
inference:
  preset:
    name: phi-4-mini-instruct

Apply the workspace:

kubectl apply -f phi-4-workspace.yaml

Supported AWS GPU Instance Types

The GPU provisioner supports various AWS GPU SKUs, see supported options here.

For the complete list and specifications, see the AWS GPU instance documentation.

Clean Up

See the Karpenter documentation for instructions on how to clean up your EKS cluster and remove Karpenter here.