Skip to main content

Creating Model Images

note

This section shows how to create a custom image with models of your choosing. If you want to use one of the pre-made models, skip to running models.

First, create a buildx buildkit instance.

docker buildx create --use --name aikit-builder

Quick Start​

You can easily build an image using the following ways:

Hugging Face​

🎬 Demo: YouTube

You can use Hugging Face models directly by providing the model URL. For example:

docker buildx build -t my-model --load \
    --build-arg="model=huggingface://TheBloke/Llama-2-7B-Chat-GGUF/llama-2-7b-chat.Q4_K_M.gguf" \
    "https://raw.githubusercontent.com/kaito-project/aikit/main/models/aikitfile.yaml"

Resulting model name will be the file name. In this case, llama-2-7b-chat.Q4_K_M.gguf.

tip

Syntax for Hugging Face source is huggingface://{organization}/{repository}/{branch}/{file}.

If the branch is main, it can be omitted (huggingface://{organization}/{repository}/{file}).

HTTP(S)​

You can use HTTP(S) URLs to download models. For example:

docker buildx build -t my-model --load \
    --build-arg="model=https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf" \
    "https://raw.githubusercontent.com/kaito-project/aikit/main/models/aikitfile.yaml"

Resulting model name will be the file name. In this case, llama-2-7b-chat.Q4_K_M.gguf.

OCI Artifacts​

🎬 Demo: YouTube

You can use OCI artifacts to download models. For example:

docker buildx build -t my-model --load \
    --build-arg="model=oci://registry.ollama.ai/library/llama3:8b" \
    "https://raw.githubusercontent.com/kaito-project/aikit/main/models/aikitfile.yaml"

Resulting model name will be the image name. In this case, llama3.

After building the image, you can proceed to running models to start the server.

Build Arguments​

Below are the build arguments you can use to customize the image:

model​

The model build argument is the model URL to download and use. You can use any Hugging Face (huggingface://), HTTP(S) (http:// or https://), or OCI (oci://). For example:

--build-arg="model=huggingface://TheBloke/Llama-2-7B-Chat-GGUF/llama-2-7b-chat.Q4_K_M.gguf"

runtime​

The runtime build argument requests a runtime from the frontend's embedded backend catalog. By default, AIKit selects the catalog's CPU runtime and default backend family. You can also set cpu explicitly.

You can use cuda to preserve AIKit's legacy family-specific NVIDIA CUDA mapping. For example:

--build-arg="runtime=cuda".

For a new explicit CUDA 12 request, use cuda-12. These are distinct catalog mappings: depending on the backend family and frontend release, cuda can select CUDA 12 or CUDA 13 and can resolve to a different LocalAI version or artifact digest. Use cuda-13 to request CUDA 13 exactly.

You can use rocm to request an AMD ROCm catalog plan. This guide's ROCm example uses llama-cpp on Linux AMD64. For example:

--build-arg="runtime=rocm".

Or use applesilicon to include Apple Silicon runtime libraries. For example:

--build-arg="runtime=applesilicon".

Multi-Platform Support​

AIKit can build AMD64 and ARM64 multi-platform images when the requested catalog tuple exists for every target. To request both platforms, add --platform linux/amd64,linux/arm64 to the build command. For example:

docker buildx build -t my-model --load \
    --platform linux/amd64,linux/arm64 \
    --build-arg="model=huggingface://TheBloke/Llama-2-7B-Chat-GGUF/llama-2-7b-chat.Q4_K_M.gguf" \
    "https://raw.githubusercontent.com/kaito-project/aikit/main/models/aikitfile.yaml"

Pre-made models are offered with multi-platform support. Docker runtime will automatically choose the correct platform to run the image. For more information, please see multi-platform images documentation.

note

The default CPU llama-cpp tuple supports both Linux AMD64 and ARM64. Other combinations are release-specific. On Linux ARM64, cuda-12 and cuda-13 select the corresponding L4T artifact, while cuda uses the backend family's legacy L4T mapping. AIKit preflights every requested platform and fails the whole request if any tuple is unavailable; it does not silently choose a different backend, CUDA major, runtime, or platform.

Advanced Usage​

🎬 Demo: YouTube

Create an aikitfile.yaml with the following structure:

#syntax=ghcr.io/kaito-project/aikit/aikit:latest
apiVersion: v1alpha1
models:
  - name: llama-2-7b-chat.Q4_K_M.gguf
    source: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf
tip

For full aikitfile inference specifications, see Inference API Specifications.

Then build your image with:

docker buildx build . -t my-model -f aikitfile.yaml --load

This will build a local container image with your model(s). You can see the image with:

docker images
REPOSITORY    TAG       IMAGE ID       CREATED             SIZE
my-model      latest    e7b7c5a4a2cb   About an hour ago   5.51GB

Running models​

You can start the inferencing server for your models with:

# for pre-made models, replace "my-model" with the image name
docker run -d --rm -p 8080:8080 my-model

You can then send requests to localhost:8080 to run inference from your models. For example:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
     "model": "llama-2-7b-chat.Q4_K_M.gguf",
     "messages": [{"role": "user", "content": "explain kubernetes in a sentence"}]
   }'

Output should be similar to:

{
    "created": 1701236489,
    "object": "chat.completion",
    "id": "dd1ff40b-31a7-4418-9e32-42151ab6875a",
    "model": "llama-2-7b-chat",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {
                "role": "assistant",
                "content": "\nKubernetes is a container orchestration system that automates the deployment, scaling, and management of containerized applications in a microservices architecture."
            }
        }
    ],
    "usage": {
        "prompt_tokens": 0,
        "completion_tokens": 0,
        "total_tokens": 0
    }
}