InferenceSet

InferenceSet is the recommended Custom Resource Definition (CRD) for serving models in production. It supports

• Serving a model with more than one replica for higher throughput and availability.
• Autoscaling the number of replicas based on inference request load (e.g. with the KEDA autoscaler).
• Integrating with the Gateway API Inference Extension for KV-cache-aware request routing.

info

The InferenceSet controller is available starting from KAITO v0.8.0 (alpha) and was promoted to beta in KAITO v0.11.0, where it is enabled by default. For older versions, enable it explicitly with --set featureGates.enableInferenceSetController=true during installation.

How it works

An InferenceSet is a higher-level controller that creates and manages one inference replica per spec.replicas. Every replica is rendered from the same shared spec.template, so all replicas serve the same model with the same configuration. The controller continuously reconciles the actual number of replicas to match spec.replicas, creating or deleting them as needed. Each replica runs as an independent inference server pod exposing the same OpenAI-compatible API.

Usage

Deploying a Hugging Face model with InferenceSet is straightforward. You just need to specify a Hugging Face model card ID (or a KAITO preset name), the GPU SKU, the desired number of replicas, and a labelSelector that identifies the replicas, with matching template.metadata.labels. For example:

apiVersion: kaito.sh/v1beta1
kind: InferenceSet
metadata:
  name: gemma-4-31b
spec:
  replicas: 2
  labelSelector:
    matchLabels:
      apps: gemma-4-31b
  template:
    metadata:
      labels:
        apps: gemma-4-31b
    resource:
      instanceType: "Standard_NC24ads_A100_v4"
    inference:
      preset:
        name: "google/gemma-4-31B-it"

InferenceSet spec fields

Field	Required	Description
spec.replicas	No (default 1)	Desired number of replicas. Set to 0 to scale to zero.
spec.labelSelector	Yes	Non-empty label query over the replica Workspaces (v1beta1 rejects an empty selector on create). Its matchLabels must not conflict with spec.template.metadata.labels and are added to every replica. With node auto-provisioning, it is also used as the GPU node selector when spec.template.resource.labelSelector is unset or empty.
spec.template.metadata.labels	No	Labels applied to every replica Workspace. Must not conflict with spec.labelSelector.matchLabels.
spec.nodeCountLimit	No	Maximum number of GPU nodes that may be created across all replicas. Unlimited if unset.
spec.template.resource.instanceType	Yes, unless nodes are bring-your-own (BYO)	GPU node SKU used for every replica. With BYO nodes it is optional and restricts replicas to nodes of that SKU.
spec.template.resource.labelSelector	No	GPU node selector copied into each replica Workspace's resource.labelSelector. With BYO nodes, leave it unset or empty ({}) and spec.template.resource.instanceType unset to auto-select the ready GPU SKU with the most GPU memory. See Selecting BYO nodes.
spec.template.inference.preset.name	Yes	The model to serve — a Hugging Face model card ID or a KAITO preset name.
spec.template.inference.adapters	No	One or more LoRA adapters to merge at serving time.
spec.template.inference.config	No	Name of a ConfigMap holding custom vLLM runtime parameters.
spec.autoUpgrade	No	Configures automatic base image upgrades of replicas after a controller upgrade. See Automatic base image upgrades.

Selecting BYO nodes

With bring-your-own (BYO) nodes, spec.labelSelector does not select GPU nodes. To pin replicas to specific nodes, set spec.template.resource.labelSelector:

apiVersion: kaito.sh/v1beta1
kind: InferenceSet
metadata:
  name: phi4-mini
spec:
  replicas: 2
  labelSelector:
    matchLabels:
      apps: phi4-mini
  template:
    metadata:
      labels:
        apps: phi4-mini
    resource:
      labelSelector:
        matchLabels:
          pool: gpu-a100
    inference:
      preset:
        name: "microsoft/Phi-4-mini-instruct"

Leave spec.template.resource.labelSelector unset to auto-select the ready GPU SKU with the most GPU memory.

Upgrading from earlier releases

Earlier releases used spec.labelSelector as the GPU node selector for every replica. With node auto-provisioning, it is still inherited when spec.template.resource.labelSelector is unset or empty, so existing InferenceSets behave the same. With BYO nodes, it is no longer inherited: existing replicas keep their node selector, but replicas created later, including scale-up replicas and replacements created by a Surge automatic base image upgrade, use spec.template.resource.labelSelector or, if it is unset, auto-select GPU nodes. To keep pinning them, recreate the InferenceSet with the node selector in spec.template.resource.labelSelector.

Creating a kaito.sh/v1beta1 InferenceSet is also stricter: spec.labelSelector must not be empty ({}), and spec.template.metadata.labels must not conflict with spec.labelSelector.matchLabels. Existing InferenceSets are not rechecked on update.

Checking status

The InferenceSet status reports how many replicas exist and how many are ready:

$ kubectl get inferenceset gemma-4-31b
NAME         REPLICAS   READYREPLICAS   AGE
gemma-4-31b   2          2               5m

You can also list the individual replicas created by the InferenceSet:

$ kubectl get workspace -l kaito.sh/inferenceset=gemma-4-31b

Scaling

To change the number of replicas, update spec.replicas:

kubectl scale inferenceset gemma-4-31b --replicas=3

The controller adds or removes replicas to match the new count. When scaling down, replicas that are not yet ready are removed first. The InferenceSet exposes the Kubernetes scale subresource, so it can be targeted directly by autoscalers such as HPA or a KEDA ScaledObject. For load-based and schedule-based autoscaling, see Autoscaling Inference with KEDA.

Graceful scale-down

Single-node vLLM workloads drain gracefully on a best-effort basis when a replica is removed. Kubernetes first marks the pod as terminating, which removes it from routing, and vLLM then has up to 50 seconds to finish in-flight requests within the pod's 60-second termination grace period. Requests that take longer than 50 seconds can still be interrupted, and a small routing-event delay can allow a late request to reach a draining pod. Multi-node (Ray) inference does not drain and keeps the default termination behavior.

You can override vLLM's drain timeout through the referenced inference ConfigMap:

data:
  inference_config.yaml: |
    vllm:
      shutdown-timeout: 45

ConfigMap parameters are applied after KAITO's defaults, so this value wins. Values above 50 seconds are not guaranteed to finish within KAITO's fixed pod grace period.

Updating the template

The InferenceSet controller reconciles the replica count (scaling) and replica labels. It does not currently perform an in-place rolling update of existing replicas when you edit other spec.template fields (such as inference parameters or adapters) — the one exception is the base image, which can be rolled out automatically via Automatic base image upgrades. To apply other template changes today, recreate the InferenceSet (or delete individual replicas so the controller recreates them from the updated template).

InferenceSet updates are validated with the same rules as Workspace updates. Fields that are immutable on a Workspace are also immutable on an InferenceSet: spec.labelSelector, spec.template.resource.labelSelector, spec.template.resource.instanceType (once set), spec.template.resource.partition, spec.template.inference.preset (including presetOptions), setting or unsetting spec.template.inference.template, the kaito.sh/capacity-type annotation, the kaito.sh/model-streaming annotation (when the ModelStreaming feature gate is enabled), and spec.template.inference.config for the custom preset. To change any of them, create a new InferenceSet.

Serving with custom parameters

You can customize vLLM runtime parameters by creating a ConfigMap containing an inference_config.yaml file and referencing it from spec.template.inference.config. Every replica created by the InferenceSet uses the same parameters. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: myns
  name: my-inference-params
data:
  inference_config.yaml: |
    max_probe_steps: 6
    vllm:
      gpu-memory-utilization: 0.95  # Controls GPU memory usage (0.0-1.0)
      tensor-parallel-size: 2       # Number of GPUs for tensor parallelism
      max-model-len: 131072         # Maximum sequence length
      cpu-offload-gb: 0             # Amount of GPU memory to offload to CPU
---
apiVersion: kaito.sh/v1beta1
kind: InferenceSet
metadata:
  namespace: myns
  name: example
spec:
  replicas: 2
  labelSelector:
    matchLabels:
      apps: example
  template:
    metadata:
      labels:
        apps: example
    resource:
      instanceType: "Standard_NC24ads_A100_v4"
    inference:
      preset:
        name: "example-model"
      config: "my-inference-params"  # Reference to ConfigMap name

Key vLLM parameters include:

• gpu-memory-utilization: Controls fraction of GPU memory allocated (between 0.0 and 1.0)
• tensor-parallel-size: Number of GPUs to use for tensor parallelism
• max-model-len: Maximum sequence length the model can handle

For the complete list of vLLM parameters, refer to the vLLM documentation.

Serving with LoRA adapters

KAITO supports serving inference with LoRA adapters produced by model fine-tuning jobs. Specify one or more adapters in the adapters field of spec.template.inference. Each replica created by the InferenceSet loads the adapters alongside the raw model weights. For example:

apiVersion: kaito.sh/v1beta1
kind: InferenceSet
metadata:
  name: phi4-mini
spec:
  replicas: 2
  labelSelector:
    matchLabels:
      apps: phi4-mini
  template:
    metadata:
      labels:
        apps: phi4-mini
    resource:
      instanceType: "Standard_NC24ads_A100_v4"
    inference:
      preset:
        name: "microsoft/Phi-4-mini-instruct"
      adapters:
        - source:
            name: "phi4-mini-adapter"
            image: "<YOUR_IMAGE>"

Currently, only images are supported as adapter sources.

Note: When building a container image for an existing adapter, ensure all adapter files are copied to the /data directory inside the container.

Inference API

Every replica runs the vLLM runtime and exposes an OpenAI-compatible HTTP API on port 80, reachable through the Kubernetes Service of each replica. To distribute requests across all replicas behind a single endpoint, front the InferenceSet with the Gateway API Inference Extension.

Chat Completions

curl -X POST "http://<SERVICE>:80/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MODEL_NAME",
        "messages": [{"role": "user", "content": "YOUR_PROMPT_HERE"}],
        "max_tokens": 200,
        "temperature": 0.7
    }'

Completions

curl -X POST "http://<SERVICE>:80/v1/completions" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MODEL_NAME",
        "prompt": "YOUR_PROMPT_HERE",
        "max_tokens": 200
    }'

List Models

curl -X GET "http://<SERVICE>:80/v1/models"

vLLM supports the full set of OpenAI-compatible inference APIs. See the vLLM OpenAI-compatible server documentation for the complete list of endpoints and parameters.

Automatic base image upgrades

KAITO ships the inference server (vLLM) as a base image embedded in the controller. When you upgrade the KAITO controller to a release that bundles a newer base image, existing replicas keep running their old image until they are recreated. With automatic base image upgrades enabled, the controller detects this version drift and rolls the replicas onto the new image one at a time, waiting for each replica to become ready before moving to the next. This is the rolling-update mechanism KAITO provides for InferenceSet.

When to use

Enable auto-upgrade when you want long-running InferenceSet workloads to automatically pick up newer base images (e.g. vLLM bug fixes, performance improvements, or security patches) shipped with KAITO controller upgrades, without manually recreating replicas — while keeping the service available throughout the rollout. If you instead prefer to pin replicas to a fixed base image and control exactly when they change, leave auto-upgrade disabled.

Upgrade strategies

Auto-upgrade supports two strategies, selected via spec.autoUpgrade.strategy. Both upgrade one replica at a time; they differ in how each replica is replaced.

Strategy	How it works	Downtime	Extra capacity
InPlace (default)	Updates the existing replica's StatefulSet image in place. The pod is recreated on the new image and the model is reloaded before it becomes ready again.	Brief per-replica downtime while the pod restarts and reloads weights.	None.
Surge	Creates a new (surge) replica on the new base image, waits for it to become inference-ready, then deletes the old replica.	None — the old replica keeps serving until the new one is ready.	Temporarily runs one extra replica during each cutover, requiring additional GPU capacity.

Choose Surge for latency-sensitive or large models where a reload-induced gap is unacceptable and you have spare GPU capacity for the surge replica. Choose InPlace (the default) when minimizing capacity/cost matters more than avoiding a brief per-replica interruption.

note

With Surge, the surge replica downloads the model weights into its own storage before it can become ready, so a cutover for a large model can take a while — but this happens off the serving path, so the old replica continues serving with no downtime. When node auto-provisioning is enabled this may provision an additional GPU node for the surge replica; in bring-your-own-nodes setups, ensure there is spare labeled node capacity or the surge replica will stay Pending.

Enabling auto-upgrade

Auto-upgrade requires two things:

1. The enableBaseImageAutoUpgrade feature gate must be enabled on the KAITO controller (it is off by default). Enable it at install/upgrade time:

   helm upgrade --install kaito-workspace ./charts/kaito/workspace \
     --set featureGates.enableBaseImageAutoUpgrade=true \
     # ... other values

2. The InferenceSet must opt in via spec.autoUpgrade.enabled: true:

   apiVersion: kaito.sh/v1beta1
   kind: InferenceSet
   metadata:
     name: gemma-4-31b
   spec:
     replicas: 3
     labelSelector:
       matchLabels:
         apps: gemma-4-31b
     template:
       metadata:
         labels:
           apps: gemma-4-31b
       resource:
         instanceType: "Standard_NC24ads_A100_v4"
       inference:
         preset:
           name: "google/gemma-4-31B-it"
     autoUpgrade:
       enabled: true
       # strategy: InPlace   # default; use Surge for zero-downtime upgrades

Restricting upgrades to a maintenance window

By default, upgrades may start at any time. To limit when rollouts begin, set a maintenanceWindow with a 5-field cron schedule (UTC) and an optional duration (defaults to 4h). The controller only starts upgrading replicas while the current time is inside the window; an upgrade already in progress is allowed to finish even if the window closes, and the next replica waits for the following window.

spec:
  autoUpgrade:
    enabled: true
    maintenanceWindow:
      schedule: "0 2 * * 6"   # every Saturday at 02:00 UTC
      duration: "4h"           # window stays open for 4 hours

Observing upgrade progress

The upgrade state is reported under status.autoUpgrade:

• numDriftedWorkspaces — number of replicas still running an old base image (0 when fully up-to-date).
• lastSuccessfulUpgradeTime — timestamp of the most recently completed replica upgrade.

kubectl get inferenceset gemma-4-31b -o jsonpath='{.status.autoUpgrade}'

Related documentation

• Workspace - The underlying single-replica CRD and how it works internally.
• Autoscaling Inference with KEDA - Load- and schedule-based autoscaling of an InferenceSet.
• Gateway API Inference Extension - KV-cache-aware routing across InferenceSet replicas.
• Multi-Node Inference - Distributed inference for large models across multiple nodes.