Skip to content

upfluence/http/server: add dedicated admin server to prevent thread starvation#38

Open
AlexisMontagne wants to merge 1 commit into
masterfrom
am/admin-server
Open

upfluence/http/server: add dedicated admin server to prevent thread starvation#38
AlexisMontagne wants to merge 1 commit into
masterfrom
am/admin-server

Conversation

@AlexisMontagne

Copy link
Copy Markdown
Member

In our current setup, Puma has a limited thread pool shared between all incoming requests — unlike our Go servers. This creates two compounding problems during traffic spikes or slow request periods:

  • Healthcheck endpoints get starved, causing pods to fail their liveness probes and enter a crash loop spiral
  • The Prometheus scrape endpoint becomes unreachable during starvation, blinding alerting exactly when it is needed most

This PR introduces an optional admin server (env: ADMIN_PORT) that binds a separate Puma listener exclusively for control plane traffic. The production builder handles application requests; the admin builder handles /healthcheck, the Prometheus exporter, the Thrift base processor, and /debug. The two are fully isolated — a saturated production thread pool has no impact on admin traffic.

  • Add admin_port option (env: ADMIN_PORT); when set, spins a side thread binding the admin builder on that port
  • Mount /healthcheck, prometheus exporter, base thrift processor, and /debug on the admin builder; keep the production builder clean
  • Fix spec hang: replace require 'sinatra' with require 'sinatra/base' in both api_endpoint files — sinatra/main registers an at_exit hook that starts a WEBrick server and blocks process exit
  • Add server_spec.rb covering both builders, healthcheck, debug mount, and custom healthcheck endpoint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants