docs(deploy): document secret/API-key rotation procedure#269
Merged
Conversation
A key rotation on 2026-06-22 took an hour because `docker restart` was used to pick up a swapped OPENAI_API_KEY. `docker restart` re-reads nothing — it bounces the process with the environment baked in at container creation, so the old (revoked) key kept being served. Only `docker compose up` (via `up.sh --remote`) re-reads env_file. Document the correct procedure in HOWTO-deploy.md, plus two traps that bit during the same incident: a key pasted as a bare value with no `VAR=` prefix, and a repeated query passing on a cached embedding while the new key was actually broken. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
During the 2026-06-22 outage recovery, rotating
OPENAI_API_KEYtook ~1h becausedocker restartwas used to reload the key.docker restartre-reads nothing — it bounces the process with the environment baked in at container creation, so the old (revoked) key kept being served and every query 500'd with a 401. Onlydocker compose up(viaup.sh --remote) re-readsenv_file.Change
Adds a Rotating secrets and API keys section to
deploy/HOWTO-deploy.mddocumenting the correct procedure and three traps from the incident:docker restartdoes not reloadenv_file— useup.sh --remote.VAR=valueassignment (a bare value leaves it undefined).Doc-only; no code paths touched.
Relates to #266.
🤖 Generated with Claude Code