Running Kimi K3 on AWS: A Step‑by‑Step Guide

Key Takeaways

• Kimi K3 is a 2.8‑trillion‑parameter Mixture‑of‑Experts (MoE) model released by Moonshot AI on July 27 2026, representing the first open‑weight system to reach the 3‑trillion‑parameter class.
• The model activates only 16 of its 896 experts per token, yielding ~104 billion active parameters and a 2.5× scaling‑efficiency gain over its predecessor.
• It supports native multimodal (text + vision) input, a 1‑million‑token context window, tool calling, structured output, and an always‑on thinking mode for multi‑step reasoning.
• Deployment on AWS can be done via Amazon SageMaker HyperPod (with the Inference Operator) or a self‑managed Amazon EKS cluster using EC2 Capacity Blocks.
• Both approaches require the p6‑b300.48xlarge instance (8 × NVIDIA B300 Blackwell Ultra GPUs) and the vLLM day‑0 inference container configured for MXFP4 quantization, tensor‑parallel size = 8, and MoE‑specific arguments.
• Proper cleanup—deleting the InferenceEndpointConfig, releasing Flexible Training Plans or Capacity Blocks, and tearing down clusters—avoids unnecessary charges.

Overview of Kimi K3’s Capabilities

Kimi K3 “delivers frontier‑level intelligence while making its weights publicly available, so that organizations can self‑host one of the most capable models in existence on their own infrastructure.” Built on a differentiated architecture that combines Kimi Delta Attention (KDA), Gated Multi‑Head Latent Attention (MLA), and a Stable LatentMoE framework, the model spreads its 2.8 trillion parameters across 896 specialist experts, activating only 16 per token. This design results in roughly 104 billion active parameters per forward pass, providing a 2.5× improvement in scaling efficiency over Kimi K2. The model also boasts a 1‑million‑token context window, native multimodal support for text and vision, and features such as tool calling, structured output, and an always‑on thinking mode that enable multi‑step agentic workflows and complex reasoning.

Model Availability and Quantization

The open weights for Kimi K3 are hosted on Hugging Face under the identifier moonshotai/Kimi‑K3. They are distributed in MXFP4 (Microscaling Floating Point 4‑bit) format, which strikes a balance between model quality and memory efficiency for large‑scale inference. To serve the model, the vLLM day‑0 inference container is required; at the time of writing the relevant commits reside in vllm/vllm‑openai:kimi‑k3 and are expected to be merged into the main vLLM container soon. vLLM provides native support for MoE architectures, tensor parallelism, and the MXFP4 quantization format, making it the recommended serving engine.

Infrastructure Requirements

Serving Kimi K3 demands substantial GPU compute. The model requires a p6‑b300.48xlarge instance, which supplies 8 NVIDIA B300 Blackwell Ultra GPUs with high‑bandwidth interconnects essential for efficient tensor‑parallel inference across the full expert pool. AWS offers two primary mechanisms to procure this capacity:

• Flexible Training Plans (for SageMaker HyperPod) – committed capacity reservations that guarantee GPU availability for sustained workloads.
• Capacity Blocks – reservations of EC2 GPU instances for a defined period, providing guaranteed access without long‑term commitments.

Both options ensure that the needed p6‑b300 nodes are available when the deployment is launched.

Deploying with Amazon SageMaker HyperPod

Creating a SageMaker HyperPod cluster with EKS orchestration
The simplest path to deploying Kimi K3 on AWS uses the SageMaker HyperPod Inference Operator. First, provision a HyperPod cluster orchestrated by Amazon EKS: open the SageMaker AI console, select HyperPod Clusters → Cluster Management → Create HyperPod cluster, choose Orchestrated by Amazon EKS, and either use Quick setup or Custom setup for networking. Under Instance groups, add a worker group configured with the ml.p6‑b300.48xlarge instance type. Ensure the Use default Helm charts and add-ons option is selected so that the Inference Operator and required operators install automatically.

Procuring capacity via a Flexible Training Plan
Because the ml.p6‑b300.48xlarge instance type requires reserved capacity, create or attach a Flexible Training Plan that covers the needed instance count and duration. In the instance group configuration, select Training plan as the capacity source, pick an existing plan or create a new reservation, and set the Target Availability Zone to match where the plan’s capacity is allocated. Once the cluster reaches an Active state with healthy p6‑b300 nodes, you are ready to deploy the model.

Deploying the model
Apply the following InferenceEndpointConfig manifest to the cluster:

yaml
apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
name: kimik3
spec:
modelName: Kimi-K3
instanceType: ml.p6-b300.48xlarge
invocationEndpoint: v1/chat/completions
replicas: 1
modelSourceConfig:
huggingFaceModel:
modelId: moonshotai/Kimi-K3
modelSourceType: huggingface
worker:
image: vllm/vllm-openai:kimi-k3
modelInvocationPort:
containerPort: 8000
name: http
modelVolumeMount:
mountPath: /opt/ml/model
name: model-weights
resources:
limits:
nvidia.com/gpu: 8
requests:
nvidia.com/gpu: 8
args:

• "–model"
• "moonshotai/Kimi-K3"
• "–trust-remote-code"
• "–load-format"
• "fastsafetensors"
• "–enable-prefix-caching"
• "–enable-auto-tool-choice"
• "–tool-call-parser"
• "kimi_k3"
• "–reasoning-parser"
• "kimi_k3"
• "–served-model-name"
• "Kimi-K3"
• "–moe-backend"
• "auto"
• "–tensor-parallel-size"
• "8"
• "–no-enable-flashinfer-autotune"
  environmentVariables:
• name: "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION"
  value: "1"

Running kubectl apply -f kimi-k3.yaml triggers the Inference Operator to download the model from Hugging Face, schedule the container, perform health checks, and expose an OpenAI‑compatible API at /v1/chat/completions. As the documentation notes, “The Inference Operator handles model download from Hugging Face, container scheduling, health checks, and endpoint readiness.”

Deploying on a Self‑Managed Amazon EKS Cluster

Provision the EKS cluster
Teams that prefer full control can use the AI on EKS project’s inference‑ready cluster recipe. Terraform modules create a GPU‑optimized EKS cluster, including VPC networking, managed node groups, and the IAM roles/policies needed for GPU workloads.

Reserve GPU capacity with Capacity Blocks
Create a Capacity Block reservation for p6‑b300.48xlarge instances in the target Availability Zone. This guarantees that the requested GPU nodes will be available for the reserved time window; once active, the instances join the EKS cluster as worker nodes.

Install GPU drivers and device plugin
The recipe installs the NVIDIA device plugin and GPU drivers on the node group, enabling Kubernetes to discover and schedule against the available GPUs.

Deploy the vLLM inference server
A Helm chart or Kubernetes manifest deploys the vLLM container with Kimi K3‑specific arguments: tensor‑parallel size = 8, MXFP4 load format, MoE backend configuration, and the model identifier pointing to the Hugging Face repository (or optionally to weights synced to Amazon S3 for faster loading).

Expose the inference endpoint
A Kubernetes Service (type LoadBalancer or via an Ingress controller) exposes the vLLM server on port 8000, providing the OpenAI‑compatible /v1/chat/completions endpoint to applications.

Validate
Confirm the deployment by sending a test request and verifying a successful model response. Detailed Terraform modules, Helm values, and step‑by‑step instructions are available in the AI on EKS Kimi K3 recipe.

Interacting with the Deployed Model

Once the endpoint is ready, it can be invoked via the OpenAI Python SDK or a simple curl command. Example SDK usage:

python
from openai import OpenAI
client = OpenAI(
base_url="http://:8000/v1",
api_key="not-needed"
)

response = client.chat.completions.create(
model="Kimi-K3",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
],
temperature=0.7,
max_tokens=1024
)
print(response.choices[0].message.content)

The equivalent curl command is:

bash
curl -X POST http://:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d ‘{
"model": "Kimi-K3",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
],
"temperature": 0.7,
"max_tokens": 1024
}’

Replace <service-endpoint> with the hostname exposed by the HyperPod Inference Operator or the EKS ingress configuration.

Cleaning Up Resources

To avoid ongoing charges, delete resources when they are no longer needed.

• SageMaker HyperPod: kubectl delete -f kimi-k3.yaml, then delete the HyperPod cluster in the SageMaker AI console, and release or cancel the Flexible Training Plan reservation.
• Amazon EKS: Delete the vLLM deployment and associated services, tear down the GPU node group or destroy the EKS cluster via Terraform (terraform destroy), and release any active Capacity Block reservation.

Pricing details for p6‑b300 instances and Capacity Blocks can be found on the Amazon EC2 pricing page.

Conclusion

Kimi K3 marks a new frontier in open‑weight model capabilities, combining massive scale with efficient MoE architecture. AWS provides the necessary infrastructure—whether through the streamlined SageMaker HyperPod Inference Operator or the flexible EKS‑with‑Capacity‑Blocks approach—to deploy and serve this model at scale. By leveraging p6‑b300 GPU instances, the vLLM serving stack, and MXFP4‑quantized weights, organizations can obtain a production‑ready endpoint with built‑in health checks, auto‑recovery, and readiness verification, enabling advanced agentic workflows, long‑horizon coding, and complex reasoning tasks on their own infrastructure.

Further reading

• SageMaker HyperPod Kimi K3 Example
• AI on EKS Kimi K3 Recipe
• Creating a SageMaker HyperPod Cluster (AWS Documentation)
• Kimi K3 on Hugging Face
• vLLM Documentation

About the authors
Andrew Smith is a Sr. Cloud Support Engineer in the SageMaker, Vision & Other team at AWS (Sydney). Erez Zarum is a Senior Startups Solutions Architect at AWS, focused on Containers and AI/ML on Amazon EKS. Vivek Gangasani is a Worldwide Leader for Solutions Architecture, SageMaker Inference at AWS, specializing in GenAI model deployment and AI workflow optimization.

https://aws.amazon.com/blogs/machine-learning/deploying-kimi-k3-on-aws/