Scaling Open AI: Deploying Kimi K3 on Amazon SageMaker HyperPod and EKS

Introduction to Kimi K3

The field of open weight models has advanced significantly, enabling them to manage highly complex operations such as multi-step agentic workflows, advanced reasoning tasks, and extensive coding projects. As these sophisticated models increase in capability, they also grow dramatically in size. Hosting architectures with trillions of parameters requires specialized infrastructure, high-end GPU computing power, and finely tuned serving frameworks.

On July 27, 2026, Moonshot AI introduced Kimi K3. This model is a Mixture of Experts (MoE) system boasting 2.8 trillion parameters, marking it as the first open-weight system to reach the three trillion parameter class. Kimi K3 provides frontier-level intelligence while making its weights publicly accessible; consequently, organizations can host one of the most powerful available models entirely on their own infrastructure.

This guide details two methods for implementing Kimi K3 on AWS: utilizing Amazon SageMaker HyperPod and deploying it within an Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Kimi K3 Architecture and Specifications

Kimi K3 is built upon a unique architecture incorporating Kimi Delta Attention (KDA), Gated Multi Head Latent Attention (MLA), and a Stable LatentMoE framework. The model distributes its 2.8 trillion parameters across 896 specialist experts, activating only 16 of these per token. This mechanism ensures that roughly 104 billion parameters are active during any single forward pass, achieving a 2.5x improvement in scaling efficiency compared to its predecessor, Kimi K2.

Kimi K3 is highly proficient in complex reasoning, agentic workflows, and long-horizon coding tasks. Furthermore, it supports native tool calling, structured output generation, and an always-on thinking mode for multi-step problem resolution.

Model Accessibility

The open weights for Kimi K3 are available on Hugging Face under the identifier `moonshotai/Kimi-K3`. These weights are distributed in MXFP4 (Microscaling Floating Point 4-bit) format, which provides an efficient balance between high model quality and low memory usage necessary for large-scale inference.

Given its size and architecture, serving Kimi K3 requires a vLLM day-0 inference container. At the time of writing, the dedicated vllm commits are located at `vllm/vllm-openai:kimi-k3`, though these are expected to be merged into the main vllm container in future releases. vLLM is recommended as the serving engine because it natively supports MoE architectures, tensor parallelism, and the MXFP4 quantization format.

Infrastructure Requirements

Deploying a model of Kimi K3’s magnitude necessitates substantial GPU computing resources. Specifically, the `ml.p6-b300.48xlarge` instance type is required, providing 8 NVIDIA B300 Blackwell Ultra GPUs and high-bandwidth interconnects crucial for efficient tensor-parallel inference across the entire expert pool.

AWS offers two primary methods to secure this capacity:

• Flexible Training Plans: These provide committed reservations that can be allocated to a HyperPod cluster, ensuring GPU resources are available for sustained workloads.
• Capacity Blocks: This option allows users to reserve EC2 GPU instances for a defined duration without committing to long-term agreements. Amazon EKS workloads consume these reservations by targeting the reserved capacity.

Deployment Option 1: Using Amazon SageMaker HyperPod

For developers seeking the simplest deployment path, Amazon SageMaker HyperPod with its Inference Operator is recommended. This operator automates container orchestration, model loading, and endpoint management, abstracting away much of the underlying complexity.

Prerequisites for Deployment

1. Step 1: Provision a HyperPod Cluster with EKS Orchestration

   Users must first provision a HyperPod cluster by selecting the “Orchestrated by Amazon EKS” option in the SageMaker AI console. During setup, it is critical to select either Quick Setup or Custom Setup (depending on whether existing networking resources are used). Ensure that the configuration selects the `ml.p6-b300.48xlarge` instance type within the worker group and confirms that the Use default Helm charts and add-ons option is active so that the necessary Inference Operator is installed.

2. Step 2: Secure Capacity with a Flexible Training Plan

   Since the `ml.p6-b300.48xlarge` instance type requires reserved capacity, users must secure a Flexible Training Plan (FTP) block. This plan guarantees access to p6-b300 instances without contention from general on-demand pools. The FTP can be attached during the instance group configuration by specifying an existing or new reservation for the required `ml.p6-b300.48xlarge` capacity and setting the Target Availability Zone accordingly.

Applying the Model Configuration

To finalize the deployment, apply the following manifest using the Inference Operator:

apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
name: kimik3
spec:
modelName: Kimi-K3
instanceType: ml.p6-b300.48xlarge
invocationEndpoint: v1/chat/completions
replicas: 1
modelSourceConfig:
huggingFaceModel:
modelId: moonshotai/Kimi-K3
modelSourceType: huggingface
worker:
image: vllm/vllm-openai:kimi-k3
modelInvocationPort:
containerPort: 8000
name: http
modelVolumeMount:
mountPath: /opt/ml/model
name: model-weights
resources:
limits:
nvidia.com/gpu: 8
requests:
nvidia.com/gpu: 8
args:
– “–model”
– “moonshotai/Kimi-K3”
– “–trust-remote-code”
– “–load-format”
– “fastsafetensors”
– “–enable-prefix-caching”
– “–enable-auto-tool-choice”
– “–tool-call-parser”
– “kimi_k3”
– “–reasoning-parser”
– “kimi_k3”
– “–served-model-name”
– “Kimi-K3”
– “–moe-backend”
– “auto”
– “–tensor-parallel-size”
– “8”
– “–no-enable-flashinfer-autotune”
environmentVariables:
– name: “VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION”
value: “1”}

This YAML configuration can be applied using the command: `kubectl apply -f kimi-k3.yaml`. The Inference Operator manages model download from Hugging Face, handles container scheduling and health checks, and exposes an OpenAI compatible API endpoint once the service reaches a ready state.

Deployment Option 2: Standalone Amazon EKS Cluster

For teams requiring full control over their Kubernetes infrastructure, deploying Kimi K3 on a standalone Amazon EKS cluster is possible. This method requires procuring GPU capacity using EC2 Capacity Blocks, which reserve p6-b300 instances for specified periods without long-term commitments.

Key Deployment Stages

1. Provision the EKS Cluster: Utilize provided Terraform modules to create a cluster optimized for GPUs. This module handles necessary components like VPC networking, managed node groups, and required IAM roles for GPU workloads.
2. Reserve Capacity with Capacity Blocks: A Capacity Block reservation must be created for `p6-b300.48xlarge` instances within the desired Availability Zone. Once active, these reserved nodes join the EKS cluster.
3. Install Drivers and Device Plugin: The deployment recipe installs the necessary NVIDIA device plugin and GPU drivers on the node group to allow Kubernetes to detect and schedule tasks against the available GPUs.
4. Deploy the vLLM Inference Server: A Helm chart or Kubernetes manifest deploys the vLLM container, configured with Kimi K3-specific parameters (e.g., tensor-parallel size of 8, MXFP4 load format, MoE backend). The model can be referenced by its Hugging Face ID or synchronized to Amazon Simple Storage Service (Amazon S3) for faster loading.
5. Expose the Inference Endpoint: A Kubernetes Service (using LoadBalancer type or an Ingress controller) exposes the vLLM server on port 8000, providing the OpenAI compatible `/v1/chat/completions` endpoint to external applications.
6. Validation: Confirm successful deployment by submitting a test request and verifying the model’s response.

Invoking the Model Endpoint

Once deployed, Kimi K3 exposes an API compatible with OpenAI’s chat completions standard. It can be called using either the OpenAI Python SDK or a `curl` command.

Using the OpenAI Python SDK

from openai import OpenAI
client = OpenAI(
base_url=”http://:8000/v1″,
api_key=”not-needed”
)

Example usage:

response = client.chat.completions.create(
model=”Kimi-K3″,
messages=[
{“role”: “system”, “content”: “You are a helpful assistant.”},
{“role”: “user”, “content”: “Explain the benefits of mixture of experts architectures.”}
],
temperature=0.7,
max_tokens=1024
)
print(response.choices[0].message.content)

Using cURL

curl -X POST http://:8000/v1/chat/completions
-H “Content-Type: application/json”
-d ‘{
“model”: “Kimi-K3”,
“messages”: [
{“role”: “system”, “content”: “You are a helpful assistant.”},
{“role”: “user”, “content”: “Explain the benefits of mixture of experts architectures.”}
],
“temperature”: 0.7,
“max_tokens”: 1024
}’

Note: When using cURL, replace “ with the service endpoint provided by the HyperPod Inference Operator or EKS ingress configuration.

Cleanup Procedures

To prevent accruing unnecessary charges, all created resources must be deleted when no longer needed.

• For SageMaker HyperPod:
  • Delete the `InferenceEndpointConfig` by executing `kubectl delete -f kimi-k3.yaml`.
  • In the SageMaker AI console, select and then delete the cluster.
  • Cancel or release the Flexible Training Plan reservation.
• For Amazon EKS:
  • Remove the vLLM deployment and associated Kubernetes services.
  • Terminate the GPU node group or destroy the entire EKS cluster using Terraform (`terraform destroy`).
  • Release the Capacity Block reservation if it has not yet expired.

Conclusion

Kimi K3 represents a major advancement in open-weight model capabilities, and AWS provides robust infrastructure options for scaling its deployment. Whether an organization chooses the streamlined path of the HyperPod Inference Operator or prefers the control offered by a self-managed EKS cluster, deploying Kimi K3 involves leveraging p6-b300 GPU instances, vLLM serving, and MXFP4 quantized weights, resulting in a robust system with built-in health checks, auto-recovery, and endpoint readiness verification for one of the world’s largest open models.