feat(linode): cache cloud-init WG bootstrap — Phase A (FJB-99)#110
Merged
hstern merged 3 commits intoMay 28, 2026
Merged
Conversation
* feat(agent): proto + ConnectRPC skeleton with Health RPC (FJB-94) First slice of the fjbagent foundation. Lands the proto package, the binary skeleton, and the Health RPC — no deploy story yet (cloud-init, systemd, cache base-image come in subsequent slices), no consumer code in the orchestrator yet (agent.Dialer is its own slice). - proto/fjbellows/agent/v1/agent.proto: AgentService.Health - cmd/fjbagent: static binary, ConnectRPC server, bearer auth, /healthz shim. -listen and -token-file flags (operator sets these from the systemd unit when deployment lands). - internal/agent: Handler captures build version + start time + hostname at construction; bearer middleware matches the internal/control auth shape (constant-time compare, /healthz exempt). - Unit tests: Health response shape, bearer accept/reject, /healthz open without token, graceful shutdown, LoadToken validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(agent): split main into run() so defer cancel() runs on error (FJB-94) gocritic flagged exitAfterDefer: os.Exit bypasses deferred cancel(), so the signal-context goroutine leaks on the error path. Moving the work into run() that returns error lets the defer fire before main maps the error to an exit code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First slice of FJB-93. Lands the AgentService.Exec bidi-streaming RPC
on the agent and a pipe-mode handler (no PTY yet — that's Phase C).
The orchestrator-side ShellSession proxy and the fjbctl ssh CLI come
in Phase B; the wire envelope shape (ShellMsg / ShellEvent) is
already designed to support both phases without a re-do.
Wire shape:
- ShellMsg: oneof Open | Stdin | Resize | Signal | CloseStdin
- ShellEvent: oneof Opened | Stdout | Stderr | Exit
- ExecOpen: target (set by fjbctl, stripped by orchestrator), argv,
env, tty (rejected with CodeUnimplemented this phase), term,
winsize
- ExecExit: code on normal exit, signal name on signal kill
Handler behavior:
- First frame must be Open or returns CodeInvalidArgument.
- tty=true returns CodeUnimplemented.
- argv exec'd directly (no shell interpolation); env is exact (no
agent-env passthrough, matches the design doc).
- exec.Cmd.Stdout/Stderr go through a streamWriter that feeds a
buffered sendBuf channel; a single sender goroutine owns the
bidi stream's Send side so we never need to mutex Connect's
BidiStream.
- cmd.Wait blocks until both child exit AND exec's internal copy
goroutines have drained, so all output frames precede Exit by
construction (avoids the StdoutPipe-vs-Wait race).
- Child runs in a fresh process group (Setpgid); signals are
delivered to the whole group via kill(-pgid).
- Command-not-found surfaces as on-stream Exit{127} (sh
convention); signal kills surface as Exit{code: 128+signum,
signal: <name>}.
Tests: 11 cases covering echo, stdin pump + CloseStdin, non-zero
exit, stderr capture, env passthrough, tty/empty-argv rejection,
command-not-found, signal kill, first-frame-must-be-Open, bearer
auth gating. `go test -race -count=10` clean.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foundation for the per-deployment ephemeral cache flow that FJB-91
called out as missing: each cache provisioned by fj-bellows now comes
up with its own WG keypair, brings up wg-quick@wg0 referencing the
orchestrator's pubkey, and publishes its own pubkey to S3 so the
orchestrator can read it back in Phase B.
What lands (Phase A):
- main.go loads or generates the orchestrator's WG keypair BEFORE
prov.Configure runs (was deferred until wgboot.Boot). Pubkey is
pushed into the Linode provider via SetOrchestratorWGPubkey for the
cache cloud-init injector.
- managedCache carries orchestratorWGPubkey and renders four new
fields into cacheCloudInitParams (OrchestratorWGPubkey, Orchestrator-
WGAddr, CacheWGAddr, WGListenPort) when transport.mode is
cache-gateway.
- cache-cloud-init.yaml.tmpl conditionally installs wireguard +
awscli, drops /usr/local/sbin/fjb-wg-bootstrap.sh, and runs it from
runcmd. The script:
- generates a Curve25519 keypair on first boot (idempotent — re-
runs reuse the persisted private key);
- writes /etc/wireguard/wg0.conf with the orchestrator pubkey baked
in as [Peer].PublicKey and the cache's address pinned at
100.64.0.2/32;
- brings up wg-quick@wg0;
- publishes /etc/wireguard/publickey to s3://<bucket>/wg-pubkey.txt
using the same Object Storage credentials already shared with
zot — no new identity to provision.
- ssh-mode renders byte-identical to before (the WG bootstrap block
is gated on OrchestratorWGPubkey).
Out of Phase A (lands in Phase B/C):
- Orchestrator-side WaitForWGPubkey polling against the bucket +
feeding the discovered pubkey into wgboot.Boot as the peer config.
- bootWGStack ordering reorder (today still wgboot before cache.Ensure;
Phase B moves it after).
- e2e harness reactivation: drop the persistent-cache preflight, use
the ephemeral cache + bootstrap, and reactivate the worker readiness
+ job-complete checks.
- Drop the static transport.wg.peer.{public_key,endpoint} config
knobs.
Tests cover the cache-gateway and ssh-mode render branches; verified
end-to-end against live Linode via test/e2e-linode/run-local.sh
--transport=cache-gateway (stack-up scope from FJB-91 still
passes — the new code path is exercised but worker validation is
the Phase C deliverable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hstern
added a commit
that referenced
this pull request
May 28, 2026
…FJB-98) (#109) * feat(linode): bake fjb-iptables FORWARD chain into cache cloud-init (FJB-98) Cache nanodes provisioned by fj-bellows in cache-gateway mode now come up with the FJB-FORWARD chain installed at first boot, persisted via iptables-persistent. Previously the chain was only rendered into a log line (`wgboot: cache iptables re-rendered (TODO: push via SSH)`) and had to be hand-installed — the persistent test nanode for FJB-91 was the only place this happened, and a rebuild would lose the FORWARD policy entirely. What lands: - managedCache learns the worker VPC CIDR via setHardwareContext, then renders cachegateway.RenderCacheIPTables at cache-create time and embeds the script in cloud-init under /usr/local/sbin/fjb-iptables.sh. - cache-cloud-init.yaml.tmpl conditionally installs iptables-persistent, writes the script, runs it from runcmd, and saves via netfilter-persistent. ssh-mode renders byte-identical to before. - cachegateway Inputs.CacheVPCIP is now optional: the renderer never used it in the script body, and the bake-in happens before any VPC IP has been assigned. wgboot still passes it (eager Status lookup). ACL-derived per-prefix ACCEPT rules are intentionally omitted from the bake-in: the ACL registry isn't wired until wgboot starts (post-Configure), so the script that lands at create time is the skeleton (sysctl + ESTABLISHED,RELATED + FJB-92 reverse-direction + DROP). Runtime push of ACL-updated rules is the wgboot TODO already noted in code; it lands with FJB-99's ephemeral-cache bootstrap loop. Tests cover the cache-gateway and ssh-mode render branches and the renderIPTablesForCacheGateway transport-mode guard. Verified end-to-end against live Linode via test/e2e-linode/run-local.sh --transport=cache-gateway. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(linode): cache cloud-init WG bootstrap — Phase A (FJB-99) (#110) * feat(agent): proto + ConnectRPC skeleton with Health RPC (FJB-94) (#107) * feat(agent): proto + ConnectRPC skeleton with Health RPC (FJB-94) First slice of the fjbagent foundation. Lands the proto package, the binary skeleton, and the Health RPC — no deploy story yet (cloud-init, systemd, cache base-image come in subsequent slices), no consumer code in the orchestrator yet (agent.Dialer is its own slice). - proto/fjbellows/agent/v1/agent.proto: AgentService.Health - cmd/fjbagent: static binary, ConnectRPC server, bearer auth, /healthz shim. -listen and -token-file flags (operator sets these from the systemd unit when deployment lands). - internal/agent: Handler captures build version + start time + hostname at construction; bearer middleware matches the internal/control auth shape (constant-time compare, /healthz exempt). - Unit tests: Health response shape, bearer accept/reject, /healthz open without token, graceful shutdown, LoadToken validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(agent): split main into run() so defer cancel() runs on error (FJB-94) gocritic flagged exitAfterDefer: os.Exit bypasses deferred cancel(), so the signal-context goroutine leaks on the error path. Moving the work into run() that returns error lets the defer fire before main maps the error to an exit code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(agent): Exec bidi RPC, pipe-mode handler (FJB-93 Phase A) (#108) First slice of FJB-93. Lands the AgentService.Exec bidi-streaming RPC on the agent and a pipe-mode handler (no PTY yet — that's Phase C). The orchestrator-side ShellSession proxy and the fjbctl ssh CLI come in Phase B; the wire envelope shape (ShellMsg / ShellEvent) is already designed to support both phases without a re-do. Wire shape: - ShellMsg: oneof Open | Stdin | Resize | Signal | CloseStdin - ShellEvent: oneof Opened | Stdout | Stderr | Exit - ExecOpen: target (set by fjbctl, stripped by orchestrator), argv, env, tty (rejected with CodeUnimplemented this phase), term, winsize - ExecExit: code on normal exit, signal name on signal kill Handler behavior: - First frame must be Open or returns CodeInvalidArgument. - tty=true returns CodeUnimplemented. - argv exec'd directly (no shell interpolation); env is exact (no agent-env passthrough, matches the design doc). - exec.Cmd.Stdout/Stderr go through a streamWriter that feeds a buffered sendBuf channel; a single sender goroutine owns the bidi stream's Send side so we never need to mutex Connect's BidiStream. - cmd.Wait blocks until both child exit AND exec's internal copy goroutines have drained, so all output frames precede Exit by construction (avoids the StdoutPipe-vs-Wait race). - Child runs in a fresh process group (Setpgid); signals are delivered to the whole group via kill(-pgid). - Command-not-found surfaces as on-stream Exit{127} (sh convention); signal kills surface as Exit{code: 128+signum, signal: <name>}. Tests: 11 cases covering echo, stdin pump + CloseStdin, non-zero exit, stderr capture, env passthrough, tty/empty-argv rejection, command-not-found, signal kill, first-frame-must-be-Open, bearer auth gating. `go test -race -count=10` clean. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(linode): cache cloud-init WG bootstrap — Phase A (FJB-99) Foundation for the per-deployment ephemeral cache flow that FJB-91 called out as missing: each cache provisioned by fj-bellows now comes up with its own WG keypair, brings up wg-quick@wg0 referencing the orchestrator's pubkey, and publishes its own pubkey to S3 so the orchestrator can read it back in Phase B. What lands (Phase A): - main.go loads or generates the orchestrator's WG keypair BEFORE prov.Configure runs (was deferred until wgboot.Boot). Pubkey is pushed into the Linode provider via SetOrchestratorWGPubkey for the cache cloud-init injector. - managedCache carries orchestratorWGPubkey and renders four new fields into cacheCloudInitParams (OrchestratorWGPubkey, Orchestrator- WGAddr, CacheWGAddr, WGListenPort) when transport.mode is cache-gateway. - cache-cloud-init.yaml.tmpl conditionally installs wireguard + awscli, drops /usr/local/sbin/fjb-wg-bootstrap.sh, and runs it from runcmd. The script: - generates a Curve25519 keypair on first boot (idempotent — re- runs reuse the persisted private key); - writes /etc/wireguard/wg0.conf with the orchestrator pubkey baked in as [Peer].PublicKey and the cache's address pinned at 100.64.0.2/32; - brings up wg-quick@wg0; - publishes /etc/wireguard/publickey to s3://<bucket>/wg-pubkey.txt using the same Object Storage credentials already shared with zot — no new identity to provision. - ssh-mode renders byte-identical to before (the WG bootstrap block is gated on OrchestratorWGPubkey). Out of Phase A (lands in Phase B/C): - Orchestrator-side WaitForWGPubkey polling against the bucket + feeding the discovered pubkey into wgboot.Boot as the peer config. - bootWGStack ordering reorder (today still wgboot before cache.Ensure; Phase B moves it after). - e2e harness reactivation: drop the persistent-cache preflight, use the ephemeral cache + bootstrap, and reactivate the worker readiness + job-complete checks. - Drop the static transport.wg.peer.{public_key,endpoint} config knobs. Tests cover the cache-gateway and ssh-mode render branches; verified end-to-end against live Linode via test/e2e-linode/run-local.sh --transport=cache-gateway (stack-up scope from FJB-91 still passes — the new code path is exercised but worker validation is the Phase C deliverable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on PR #109 (FJB-98). GitHub will retarget to
mainonce that merges.Summary
Foundation slice for the per-deployment ephemeral cache flow FJB-91 called out as missing. Each cache provisioned by fj-bellows now comes up with its own WG keypair, brings up
wg-quick@wg0referencing the orchestrator pubkey, and publishes its own pubkey to S3 for the orchestrator to read back in Phase B.prov.Configure, push pubkey into the Linode provider viaSetOrchestratorWGPubkey.orchestratorWGPubkey+ renders four new fields intocacheCloudInitParams(pubkey/orch addr/cache addr/listen port) when transport.mode is cache-gateway.wireguard+awscli, drops/usr/local/sbin/fjb-wg-bootstrap.sh, runs it from runcmd. Script generates the keypair (idempotent), writeswg0.confwith the orchestrator baked in as[Peer].PublicKey, brings upwg-quick@wg0, andaws s3 cps the cache's pubkey tos3://<bucket>/wg-pubkey.txtusing the same Object Storage creds zot already uses.Not in this PR (Phase B/C)
WaitForWGPubkeypolling against the bucket + feeding the discovered pubkey intowgboot.Boot.bootWGStackto fire AFTERcache.Ensure.transport.wg.peer.{public_key,endpoint}config knobs.Test plan
go build ./...go test ./...golangci-lint runbash test/e2e-linode/run-local.sh --transport=cache-gateway→ ALL OK (live Linode; Phase A code path exercised, worker validation lands with Phase C harness reactivation)🤖 Generated with Claude Code