Skip to content

Flip shard signing to fail-closed (zeitghost_require_signing: 1) once cycles confirm signed>0 #6

Description

@aaronmarkham

Flip shard signing to fail-closed on us-ny1

Follow-up to PR #5 (shard integrity & lineage). The signing mechanism is fully landed and verified working in prod — this issue tracks the final rollout step: making ingest fail-closed when no valid signing key is configured, so an accidentally-cleared or mangled ZEITGHOST_SIGNING_KEY fails loudly instead of silently reverting to unsigned shards.

Deliberately deferred so the change lands after a few ingest cycles have produced organically-signed shards and we've confirmed signed > 0.

Current state (verified 2026-05-31)

  • ZEITGHOST_SIGNING_KEY provisioned via GitHub repo secret → deploy.ymlenv.j2 → container.
  • In-container probe: signs_ok: True, key_len 64, resolve_signing_seed() → OK.
  • Signer thumbprint: 7c1d34f0720bab10245ebca2f3eb8d540888d1b50e8f0cde73eaf846bde3a49b.
  • zeitghost_require_signing defaults to 0 (opt-in) — unsigned writes are currently allowed.
  • At verification time signed = 0 / 16863 only because no new articles had been ingested since the key was fixed (dedup skips the existing corpus).

Precondition before flipping

Confirm signing is taking effect on real ingest writes (not just the probe):

ssh us-ny1 "docker exec zeitghost-builder python -c 'from zeitghost.shards import init_store, SCOPE_INTERNAL; ss=list(init_store().by_scope(SCOPE_INTERNAL)); signed=[s for s in ss if s.signature]; print(\"total\", len(ss), \"signed\", len(signed))'"

Expect signed > 0, and the newest signed shard's created_by == the thumbprint above.

The change (one line)

In infra/ansible/inventories/us-ny1/hosts.yml, under the us-ny1 host:

zeitghost_require_signing: 1

Then redeploy (push to main / run the Deploy workflow). env.j2 already templates ZEITGHOST_REQUIRE_SIGNING from this var; the CLI's signing_required() + ingest fail-fast (exit 1) do the rest.

Acceptance criteria

  • signed > 0 confirmed organically, newest created_by matches the recorded thumbprint.
  • zeitghost_require_signing: 1 set in the us-ny1 inventory.
  • Post-deploy: zeitghost ingest still succeeds (key present), shards continue signing.
  • Sanity: with a temporarily-unset/garbled key, ingest exits non-zero (fail-closed) rather than writing unsigned — verify in a throwaway/manual run, not on the live builder.

Notes / gotchas

  • Secrets mechanism is the GitHub repo secret ZEITGHOST_SIGNING_KEY, NOT ansible-vault (this repo has no vault file).
  • gen-signing-key wraps the 64-char seed across terminal lines — the first prod paste was truncated (key_len 57). Fast tell if it recurs: key_len reads anything other than 64.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions