Skip to content

fix(prod): raise pipelines memory request 1Gi→3Gi (stop eviction loop) - #32

Merged
sumit-badsara merged 1 commit into
mainfrom
feat/pipelines-mem-raise
Jul 14, 2026
Merged

fix(prod): raise pipelines memory request 1Gi→3Gi (stop eviction loop)#32
sumit-badsara merged 1 commit into
mainfrom
feat/pipelines-mem-raise

Conversation

@sumit-badsara

@sumit-badsara sumit-badsara commented Jul 14, 2026

Copy link
Copy Markdown

What

pipelines-prod mem request 1Gi → 3Gi (limit kept 4Gi), prod-only override.

Why

pipelines uses ~1978Mi = 193% of its 1Gi request. As Burstable QoS exceeding its request, the kubelet evicts it first under node memory pressure — repeatedly → 27 evicted zombie pods piled up + KubePodNotReady alert noise, and its memory-based HPA read ~193%/75% so it stayed pinned at maxReplicas. 3Gi request stops the evictions and lets the HPA scale on real load. Verified with helm template (renders memory: 3Gi).

🤖 Generated with Claude Code


Note

Low Risk
Single prod Helm resource tuning with documented rationale; no app code or security surface changes, though it increases guaranteed memory per pod on nodes.

Overview
Prod-only override in k8s/prod/values.yaml sets explicit resources for pipelines-prod: memory request goes from the chart default 1Gi to 3Gi, while limits stay 4Gi CPU/2 (same as before).

This targets a Burstable pod running ~1978Mi (~193% of the old request), which was getting kubelet-evicted under node memory pressure (many evicted pods, KubePodNotReady noise) and kept the memory HPA saturated at maxReplicas (~193%/75%). Raising the request should reduce evictions and let autoscaling reflect real load.

Reviewed by Cursor Bugbot for commit 31f6ef6. Bugbot is set up for automated code reviews on this repo. Configure here.

Greptile Summary

This PR raises the prod memory request for the pipelines deployment.

  • Adds a prod-only resources override in k8s/prod/values.yaml.
  • Changes the memory request from 1Gi to 3Gi.
  • Keeps the existing CPU request, CPU limit, and memory limit values.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
k8s/prod/values.yaml Adds a prod-specific resources override that matches the chart's existing container resources shape.

Reviews (1): Last reviewed commit: "fix(prod): raise pipelines memory reques..." | Re-trigger Greptile

Context used:

  • Rule used - Ensure that the confidence score is always within ... (source)

Learned From
websentry-ai/ai-gateway-data#448

…oop)

pipelines-prod uses ~1978Mi = 193% of the old 1Gi request. As Burstable QoS
using far more than its request, the kubelet evicted it first under node memory
pressure — repeatedly — piling up 27 evicted zombie pods and firing KubePodNotReady
noise. Its memory-based HPA also read ~193%/75% and stayed pinned at maxReplicas.

Raise the request to 3Gi (limit kept 4Gi, covers the ~2Gi peak) so it stops being
evicted and the HPA scales on real load. Prod-only override; CPU unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@sumit-badsara
sumit-badsara merged commit 18f5919 into main Jul 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants