fix(prod): raise pipelines memory request 1Gi→3Gi (stop eviction loop) - #32
Merged
Conversation
…oop) pipelines-prod uses ~1978Mi = 193% of the old 1Gi request. As Burstable QoS using far more than its request, the kubelet evicted it first under node memory pressure — repeatedly — piling up 27 evicted zombie pods and firing KubePodNotReady noise. Its memory-based HPA also read ~193%/75% and stayed pinned at maxReplicas. Raise the request to 3Gi (limit kept 4Gi, covers the ~2Gi peak) so it stops being evicted and the HPA scales on real load. Prod-only override; CPU unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gowshik450526511
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
pipelines-prodmem request1Gi → 3Gi(limit kept4Gi), prod-only override.Why
pipelines uses ~1978Mi = 193% of its 1Gi request. As Burstable QoS exceeding its request, the kubelet evicts it first under node memory pressure — repeatedly → 27 evicted zombie pods piled up +
KubePodNotReadyalert noise, and its memory-based HPA read ~193%/75% so it stayed pinned at maxReplicas. 3Gi request stops the evictions and lets the HPA scale on real load. Verified withhelm template(rendersmemory: 3Gi).🤖 Generated with Claude Code
Note
Low Risk
Single prod Helm resource tuning with documented rationale; no app code or security surface changes, though it increases guaranteed memory per pod on nodes.
Overview
Prod-only override in
k8s/prod/values.yamlsets explicitresourcesforpipelines-prod: memory request goes from the chart default 1Gi to 3Gi, while limits stay 4Gi CPU/2 (same as before).This targets a Burstable pod running ~1978Mi (~193% of the old request), which was getting kubelet-evicted under node memory pressure (many evicted pods,
KubePodNotReadynoise) and kept the memory HPA saturated at maxReplicas (~193%/75%). Raising the request should reduce evictions and let autoscaling reflect real load.Reviewed by Cursor Bugbot for commit 31f6ef6. Bugbot is set up for automated code reviews on this repo. Configure here.
Greptile Summary
This PR raises the prod memory request for the pipelines deployment.
resourcesoverride ink8s/prod/values.yaml.1Gito3Gi.Confidence Score: 5/5
This looks safe to merge.
Important Files Changed
Reviews (1): Last reviewed commit: "fix(prod): raise pipelines memory reques..." | Re-trigger Greptile
Context used:
Learned From
websentry-ai/ai-gateway-data#448