refactor(daemon): run the orphan reconciler as a one-shot CronJob

[zfy0701]

Summary

The pool's orphan reconciler (#1074) moves out of the daemon process and becomes a one-shot
subcommand a Kubernetes CronJob runs: agentconnect-daemon reconcile --once. It boots only what a
sweep needs — the sandbox API surface and a control-plane connection for the agent/exists read —
runs exactly one sweep, prints the summary line, and exits 0 (non-zero when the sweep could not
complete). The in-process timer, the jitter, the interval knob, and the sweep_leases table are
gone.

The connection the job opens registers as an observer, a new optional register.observer flag,
so a process whose only job is to sweep can never be handed work to serve.

Why a CronJob

Everything the in-process sweep built by hand is something the cluster already owns:

• Schedule — a CronJob schedule instead of a jittered setTimeout on every member.
• Mutual exclusion — concurrencyPolicy: Forbid instead of a single-holder lease in the shared
  store. One run at a time, enforced by the thing that starts the runs.
• Failure reporting — a failed Job is visible in the cluster. A sweep that could not read the
  control plane now exits non-zero instead of logging a warning inside a daemon that keeps running.
• Blast radius — a sweep is housekeeping; running it in the process that also serves agent
  turns bought nothing and cost a scheduler, a lease table, and a feature gate.

One behavioural consequence, stated deliberately: the grace period is now the object's own age
alone. The old code also required the agent to have been missing across the sweeping member's own
sweeps, and a one-shot run has no memory of an earlier run to carry that in. The object-age gate is
the clock that actually guards the race the grace exists for — an in-flight creation still racing
the control plane's write — and it survives; the reconciler also still ships dry-run by default.

Observer registration

RegisterReq gains an optional observer: true. On an install-wide (pool identity) connection the
control plane admits it exactly as it admits a member — same TokenReview path, same projected
ac-control-plane token, same auth → register handshake — and then:

• withdraws the membership upsertOnAuth mints for every org-less row. Duty eligibility is a
  member_set_member lookup (claimVacant's eligibleAgent gate), so a daemon in no set can never
  be granted a set-placed group, and a machine-placed agent never names it.
• backdates lastSeenAt, so the existing pool-member reaper retires the row on its next sweep
  rather than after the ordinary 15-minute silence window. Marking the row is the smaller change:
  skipping the row entirely is not available, because the epoch mint, the connection registry, and
  every reply are keyed by a daemon id that has to exist. An observer sends no heartbeat, so nothing
  moves the stamp forward again.

The flag is refused on an org-scoped (API-key) connection with SCOPE_DENIED: that credential
is an operator's daemon key, not a job identity, and there is no reason to let it opt out of
membership.

agent/exists already served install-wide connections and now serves observers unchanged.

What was removed

• OrphanReconciler's scheduler: start()/stop(), the ±25% jitter, AC_K8S_ORPHAN_SWEEP_INTERVAL_MS,
  and the in-flight single-flight guard. AC_K8S_ORPHAN_DELETE (default off = dry-run) and
  AC_K8S_ORPHAN_GRACE_MS are unchanged.
• The sweep_leases table and LocalStore.acquireSweepLease. No migration step: the table only
  ever existed in the CREATE TABLE IF NOT EXISTS block, was introduced in a single prerelease, and
  is never read again — an existing store simply keeps an unused table until it is recreated.
• The daemon-side feature gate ("skip the sweep if the control plane does not advertise
  agent-exists-v1"). It only existed to keep an in-process sweep quiet against an older control
  plane; the job now fails loudly instead, which is what a Job status is for. The control plane
  still advertises the feature.
• CpClient.agentsExist and the daemon's liveAgentsFor, with the reconciler's wiring in
  Daemon/startK8sRuntimePlane. Members run no sweep at all now.

The CronJob the deployment side should add

Image = the daemon image, args reconcile --once, every 10 minutes, the pool members'
ServiceAccount, the same namespace/warm-pool/labels environment they read, and
AC_K8S_ORPHAN_DELETE left unset so it starts dry-run:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: agentconnect-orphan-reconciler
spec:
  schedule: '*/10 * * * *'
  # This is the mutual exclusion the in-process lease used to provide.
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 300
      template:
        spec:
          restartPolicy: Never
          # The pool members' ServiceAccount: same TokenReview identity, same sandbox-namespace RBAC.
          serviceAccountName: ac-cloud-daemon
          containers:
            - name: reconcile
              image: <the same daemon image the pool runs>
              args: ['reconcile', '--once']
              # The same environment source the pool members read, so namespace, warm pool and
              # labels cannot drift: AC_CP_URL and AC_K8S_SANDBOX_NAMESPACE come from it, and the
              # member-only variables it also carries are simply ignored by this command.
              # AC_K8S_ORPHAN_DELETE is deliberately UNSET here — dry run until the summary lines
              # have been observed; set it to "true" to enable collection. AC_K8S_ORPHAN_GRACE_MS
              # defaults to 10 minutes.
              envFrom:
                - configMapRef:
                    name: <the pool members' env ConfigMap>
              volumeMounts:
                - name: cp-identity
                  mountPath: /var/run/ac-cp-identity
                  readOnly: true
          volumes:
            # The projected control-plane audience token, exactly as the pool members mount it.
            - name: cp-identity
              projected:
                sources:
                  - serviceAccountToken:
                      path: token
                      audience: ac-control-plane
                      expirationSeconds: 3600

The command needs AC_CP_URL and AC_K8S_SANDBOX_NAMESPACE, plus the projected identity token at
/var/run/ac-cp-identity/token. Its RBAC is the members' existing sandbox-namespace Role (list and
delete sandboxclaims, list and delete sandboxes); a Role without the Sandbox list verb simply
narrows the sweep to claims, as before.

Test plan

• packages/daemon/test/cli-reconcile.test.ts (new) — the subcommand against the fake API server
  plus a fake control plane: one sweep, one existence read, the summary line, exit 0; exit 1 when
  the control plane cannot be reached (deleting nothing) and when the sandbox namespace is unset.
  Plus the observer handshake itself over a fake transport: auth then register with
  observer: true, the agent/exists chunking, and a refused registration failing the connection.
• packages/control-plane/test/protocol/observer-register.handler.test.ts (new, real Postgres) —
  an observer register leaves no member_set_member row and is granted nothing from a vacant duty
  group over a set-placed agent, while an ordinary member of the same pool claims that same group;
  agent/exists is answered on the observer connection; the row is selected by
  findRetiredPoolMembers immediately; the flag is refused with SCOPE_DENIED on an API-key
  connection.
• packages/daemon/test/k8s-orphan-reconciler.test.ts — the feat(daemon): pool orphan reconciler for leaked sandbox objects #1074 safety rules kept, minus the
  lease and scheduler cases, with the grace now asserted on the object's own age.
• Ran: pnpm lint, pnpm format:check, typecheck for protocol / daemon / control-plane, the daemon
  test/cp, test/k8s-*, test/local-store.test.ts and test/cli-reconcile.test.ts suites, and the
  control-plane unit + integration projects.

[agentconnect-md-test]

I found one blocking reliability issue in the new one-shot Job contract.

runReconcileOnce currently exits successfully whenever OrphanReconciler.sweep() returns a summary. Individual Kubernetes delete errors are caught inside the sweep and recorded in summary.failed, so a run that fails to remove one or every orphan still exits 0. Kubernetes therefore records a successful Job and the stated failure-reporting/alerting behavior is lost. Please make any nonzero failed count produce a nonzero process exit.

I verified the exact trusted base/head revision and inspected the complete daemon, protocol, control-plane, documentation, and test diff. Static whitespace/conflict-marker checks passed. I could not execute the Vitest/typecheck suites because this checkout has no installed dependencies.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

[P2] Return nonzero when object deletion failed

sweep() returns a summary even when one or more Kubernetes deletes failed; those failures are recorded in summary.failed. Since every summary object is truthy here, a run that fails to delete all candidates still exits 0 and Kubernetes records a successful Job, defeating the new failure-reporting contract. Please return nonzero when summary.failed > 0 (while continuing to treat undefined as failure).

[zfy0701]

Good catch — fixed in 22d5c90.

A failed delete is counted in summary.failed rather than thrown, so the run reports the whole
picture in one line; but returning 0 on that summary recorded a successful Job for a run that left
an orphan behind, and the leak would just be back on the next run with nothing in the cluster's
history saying so. runReconcileOnce now exits non-zero when summary.failed > 0, after the
summary line the sweep already logged. An unreadable sweep (undefined) stays a failure as before.

Test added: a sweep whose delete is refused 403 still logs
orphaned=1 deleted=0 skipped-live=1 skipped-grace=0 failed=1 and exits 1.

Re-ran cli-reconcile + orphan-reconciler suites (17 passed), daemon typecheck, pnpm lint,
pnpm format:check; rebased on origin/main before pushing.

[agentconnect-md-test]

The prior blocking finding is resolved in 22d5c90513ecf5e35cf7b05bea0b228e82ba3cdc: the one-shot reconciler now returns a nonzero exit when its completed sweep reports any failed Kubernetes deletion, while an unreadable sweep remains a failure. The added regression test covers the partial-delete failure and confirms the full summary is still logged before exit.

I verified the exact trusted head, confirmed the fix delta is limited to the exit-status correction and its test, and rechecked the complete base-to-head daemon/protocol/control-plane behavior. Static whitespace and conflict-marker checks passed. I could not execute Vitest or typecheck because this checkout has no installed dependencies. No blocking findings remain.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Confirmed the fix in 22d5c90; the failed-delete path now exits 1 and has regression coverage. I’ve approved the updated revision. Thanks!

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}