Skip to content

[codex] Add serverless container wake path#103

Merged
arjunkomath merged 27 commits into
mainfrom
feat/serverless-containers
Jul 6, 2026
Merged

[codex] Add serverless container wake path#103
arjunkomath merged 27 commits into
mainfrom
feat/serverless-containers

Conversation

@techulus-agent

@techulus-agent techulus-agent commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds proxy-local serverless container support:

  • Proxy-hosted deployments can sleep locally by stopping the Podman container and reporting sleep through the normal agent status channel.
  • The local wake gateway queues incoming HTTP requests, reports wake_started, starts the local container, waits for readiness, then proxies the held request.
  • Worker/non-proxy deployments stay always on, including for serverless-enabled services.
  • Owner-only routing sends serverless public HTTP routes to 127.0.0.1:18080 only on proxy nodes that host a local replica for that service.
  • Deployment state is split into explicit fields: runtimeDesiredState, trafficState, and observedPhase, so expected-state is a projection of intent instead of inferred lifecycle history.
  • Serverless settings are deploy-time config via deployedConfig, so changing enablement, ports, or related service config applies on the next rollout.

Manual alpha cutover note

This PR replaces the old overloaded deployment status/desired model with explicit state fields. On an existing alpha database, drizzle-kit push cannot infer which pre-existing deployments should receive active traffic. I can manually repair any broken deployments after deploy.

Use these rules for manual repair:

  • Live deployments that should receive traffic: set runtime_desired_state = 'running', traffic_state = 'active', and observed_phase = 'healthy' or 'running' based on the actual agent/container state.
  • Sleeping serverless deployments that should remain wakeable: set runtime_desired_state = 'stopped', traffic_state = 'active', observed_phase = 'sleeping', and container_id = NULL.
  • Old rollout/drained/deleted deployments: set runtime_desired_state = 'removed' and traffic_state = 'inactive'.

Because the product is alpha, the simpler alternative is to redeploy affected services after the PR lands, which creates fresh deployments under the new state model.

Validation

  • pnpm exec biome format --write lib/agent-status.ts lib/agent/expected-state.ts tests/agent-status.test.ts tests/expected-state.test.ts
  • pnpm exec tsc --noEmit
  • pnpm exec vitest run tests/agent-status.test.ts tests/expected-state.test.ts
  • pnpm test
  • git diff --check
  • Earlier on this branch: go test ./...
  • Earlier on this branch: go test -count=1 -race ./internal/serverless

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

React Doctor found no new issues. 🎉

Reviewed by React Doctor for commit 5fb584f.

@arjunkomath arjunkomath marked this pull request as ready for review July 4, 2026 23:12
@mintlify

mintlify Bot commented Jul 5, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
techulus-cloud 🟢 Ready View Preview Jul 5, 2026, 1:16 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@techulus-agent

Copy link
Copy Markdown
Collaborator Author

Review — Lean Explicit Intent Refactor (b795412, 7959b9d)

Reviewed the two refactor commits on top of the serverless branch. Overall this is a clean, faithful implementation: tsc --noEmit is green and the 8 touched test suites pass. Intent (runtimeDesiredState, trafficState) is now cleanly separated from observation (observedPhase), and containerId is no longer overloaded as runtime intent. Below are the per-question findings and a few things worth a second look before merge.

1. Intent vs. observation — ✅ separated; containerId no longer used as intent

  • Container desiredState is now derived purely from runtimeDesiredState (expected-state.ts:308dep.runtimeDesiredState === "stopped" ? "stopped" : "running"). The 7959b9d fix is the important one here — it removed the old serverIsProxy && isDeployedServerlessService && sleepableServiceIds.has(...) && runtimeDesiredState === "stopped" gate that previously projected an explicitly stopped non‑serverless deployment back to "running". Good catch.
  • Container existence is gated by runtimeDesiredState ∈ {running, stopped} (runtimeExpectedStates); removed is excluded from the container query entirely, so the agent tears it down. containerId is only used as an operational handle (which container to stop/attach), never to decide desired state.

2. desired boolean — ✅ fully gone

  • Column dropped from schema.ts; markDeploymentUndesired replaced by markDeploymentRemoved / markDeploymentFailedRemoved. No residual deployments.status or .desired references anywhere in web/ (grep clean, tsc clean). runtimeDesiredState = "removed" fully subsumes the old desired = false.

3. Rollout / rollback transitions & precedence — ✅ correct

  • Promotion, drain, and teardown are all keyed on the right axis:
    • start-dns-sync promotes candidate → active only for healthy/running deployments of this rollout, and demotes prior active → draining for other/null rollouts (rollout-workflow.ts:498–521).
    • handleRolloutFailure / restoreDrainingDeploymentsForRollback restore draining → active on rollback, and markDeploymentFailedRemoved flips runtime→removed + traffic→inactive (rollout-utils.ts).
  • Precedence holds: existence is decided by runtimeDesiredState, traffic by trafficState === "active" && observedReady. The two axes are independent and never cross‑contaminate.
  • Nit: shouldRollBackDeploymentState takes trafficState but voids it, and isActiveDeploymentForRollout voids service. Dead params — worth dropping to avoid implying they matter.

4. Serverless wake/sleep suppression window — ✅ preserved

  • getStoppedContainerReportUpdate (agent-status.ts:59) keeps a stopped-container report as observedPhase: "sleeping" (and containerId: null) when runtimeDesiredState === "stopped", vs. "stopped" otherwise — this is what preserves the sleep observation rather than flapping to a terminal state. shouldAttachReportedContainer still excludes sleeping, so a late "running" report can't re-attach a sleeping deployment. Wake-failure backoff (getServerlessWakeFailureUpdate) correctly reverts to stopped/sleeping under the limit and to failed+removed at the limit. The sleep/wake_started/wake_failed transition guards in getInvalidServerlessTransitionReason are all keyed on the new fields with stale-containerId protection intact.

5. candidate → active promotion — ✅ correct

  • Only trafficState === "candidate" + observedPhase ∈ {healthy, running} get promoted, and the workflow has already gated on all target deployments being healthy/running before this step, so nothing is promoted prematurely. Sleeping serverless deployments retain trafficState: "active" (they only change observedPhase), so they stay valid gateway targets — consistent with the serverless route builder.

6. Agent API — ✅ stable

  • Both refactor commits touch only web/lib/agent/expected-state.ts on the agent-facing path; no agent/** Go files. ExpectedContainer.desiredState is still the "running" | "stopped" string the agent already consumes (drift.go:desiredContainerState, gateway.go), and the StatusReport / ContainerStatus shape is unchanged. (The serverlessTransitions field in the status route predates this refactor.)

7. Gaps between plan and implementation

(a) No data backfill for the schema swap — the one real risk. The project deploys schema via drizzle-kit push with no migration/backfill. This change drops status/desired and adds three NOT‑NULL columns with defaults running / candidate / pending. On any DB with existing rows, every live deployment becomes trafficState = "candidate" and observedPhase = "pending" — which removes it from DNS and Traefik routes (both require trafficState = active + observedReady) until the next rollout re-promotes it. If this branch will ever be pushed against a populated database, it needs a one-time backfill (e.g. map old running/healthyactive + matching observedPhase, old desired=falseremoved). Acceptable only if every environment is drained/redeployed as part of the cutover — worth stating explicitly.

(b) stopService uses markDeploymentRemoved(), not "stopped" — faithful, but naming invites confusion. I traced the old path: markDeploymentUndesired("stopping") set desired=false and the orphan loop deleted the row once the container was gone, so stopService always tore deployments down. The new removed + delete-on-gone path is behaviorally equivalent, so this is not a regression. But given there's now a first-class runtimeDesiredState = "stopped", a reader reasonably expects "stop service" to use it; a one-line comment clarifying that stopped == serverless sleep and removed == user teardown would save future confusion.

(c) Minor behavior change in deletion/volume guards. blocksProjectDeletion and the volume-removal guard now block whenever runtimeDesiredState !== "removed", so stopped/sleeping/draining deployments now block where the old status allow-lists didn't. This is arguably more correct (those deployments still exist), but it is a user-visible change — e.g. a sleeping serverless service now blocks project deletion.

(d) Cosmetic looseness. The routableDeployments / serverless-route queries filter runtimeDesiredState ∈ {running, stopped}; stopped is redundant there since the accompanying observedPhase ∈ {healthy, running} filter already excludes sleeping deployments. Harmless, just slightly wider than necessary.

Verdict

Solid refactor — the intent/observation split is clean and the precedence model is coherent. Nothing blocking in the code itself. The item that genuinely needs a decision before merge is (a) the backfill/cutover story for existing deployments; the rest are nits/clarifications.

@arjunkomath arjunkomath merged commit 903ac53 into main Jul 6, 2026
11 checks passed
@arjunkomath arjunkomath deleted the feat/serverless-containers branch July 6, 2026 07:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants