AI Question-Answering Service on AWS EKS

Architecture

flowchart TB
    User([Internet User])

    subgraph AWS["AWS Cloud"]
        subgraph VPC["VPC (10.0.0.0/16)"]
            subgraph Public["Public Subnets"]
                IGW[Internet Gateway]
                NAT[NAT Gateway]
                Ingress[NGINX Ingress Controller]
            end
            subgraph Private["Private Subnets"]
                subgraph EKS["EKS Cluster"]
                    FastAPI[FastAPI Service\nClusterIP :80]
                    vLLM[vLLM Service\nClusterIP :8000\nSmolLM2-135M-Instruct]
                end
            end
        end
    end

    User -->|HTTP Request| IGW
    IGW --> Ingress
    Ingress -->|/ask| FastAPI
    FastAPI -->|POST /v1/completions| vLLM
    Private -->|Outbound via| NAT
    NAT --> IGW

Loading

Components:

• VPC — Private subnets for EKS nodes, public subnets for NAT/ALB.
• EKS Cluster — Managed Kubernetes control plane.
• Node Group — EC2 worker nodes in private subnets.
• FastAPI Pod — Receives user questions, forwards to vLLM.
• vLLM Pod — Serves the SmolLM2-135M-Instruct model with an OpenAI-compatible API.
• Ingress — NGINX Ingress Controller exposes the FastAPI service to the internet.

Prerequisites

• AWS CLI configured with appropriate credentials
• Terraform >= 1.5
• kubectl
• Helm 3
• Docker

Setup Instructions

1. Provision Infrastructure

cd terraform
terraform init
terraform plan
terraform apply

2. Configure kubectl

aws eks update-kubeconfig --name llm-eks-cluster --region <your-region>

3. Install NGINX Ingress Controller

Note: The NGINX Ingress Controller is deployed automatically using the Terraform Helm provider. No manual installation is required.

4. Build and Push the FastAPI Image

Note: The FastAPI Docker image is built and pushed to ECR automatically via Terraform (using a null_resource and local-exec). No manual Docker commands are required.

5. Deploy vLLM

Note: The vLLM service is deployed automatically using the Terraform Helm provider. No manual helm install is required.

6. Deploy FastAPI App

Note: The FastAPI application is deployed automatically using the Terraform Helm provider. You do not need to run a manual helm install for FastAPI. The deployment and configuration are managed in your Terraform code.

7. Test the Service

Before testing, ensure all pods are running:

kubectl get pods -A

Wait until the STATUS column for all relevant pods (FastAPI, vLLM, ingress-nginx) shows "Running" or "Completed".

To find the external endpoint for your FastAPI application, run:

kubectl get ingress -A

Look for the ADDRESS column in the output. For example:

NAMESPACE   NAME                 CLASS   HOSTS   ADDRESS
default     qa-service-ingress   nginx   *       ab151ad96701e487d83998689992032a-1807300597.us-east-2.elb.amazonaws.com

Your FastAPI application is exposed at:

http://ab151ad96701e487d83998689992032a-1807300597.us-east-2.elb.amazonaws.com

You can test the /ask endpoint with:

curl -X POST "http://ab151ad96701e487d83998689992032a-1807300597.us-east-2.elb.amazonaws.com/ask" \
  -H "Content-Type: application/json" \
  -d '{"question": "What is Kubernetes?"}'

Scaling Strategies for LLM Workloads on EKS

Horizontal Pod Autoscaler (HPA)

Scale vLLM replicas based on CPU/memory utilization or custom metrics (e.g., request queue depth). For the FastAPI service, scale based on request rate.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-vllm
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Cluster Autoscaler / Karpenter

Use Karpenter to automatically provision new nodes when pods are pending due to resource constraints. Define NodePools with GPU instance types (e.g., g5.xlarge) for LLM workloads and CPU instance types for the API layer.

GPU Node Groups

For production, create a separate node group with GPU instances. Use node selectors or taints/tolerations to schedule vLLM pods on GPU nodes only.

Resource Requests and Limits

Set accurate resource requests so the scheduler can pack pods efficiently. For vLLM, memory is the key constraint — the model must fit in memory.

Multi-Model Serving

vLLM supports serving multiple models. Consolidate smaller models onto shared instances to improve utilization.

Request Batching

vLLM handles continuous batching internally. Tune --max-num-batched-tokens and --max-num-seqs to balance throughput and latency.

Spot Instances

Use Spot instances for non-critical or stateless workloads (FastAPI layer). For vLLM, prefer On-Demand to avoid model reload times on interruption.