diff --git a/.claude/skills/handoff-report/SKILL.md b/.claude/skills/handoff-report/SKILL.md new file mode 100644 index 0000000..046dee4 --- /dev/null +++ b/.claude/skills/handoff-report/SKILL.md @@ -0,0 +1,325 @@ +--- +name: handoff-report +description: >- + Transfer working knowledge of an existing part of The Commons to someone who lacks it — + what it does and who depends on it, Mermaid diagrams of real behaviour, data-model/interface + tables, and the sharp edges that bite newcomers. Triggers: "how does X work", "document this + service", "onboard someone onto Y", "I'm handing this off", "explain this subsystem", + "write a handoff doc". +--- + +# Handoff Report + +## Overview + +A handoff report transfers working knowledge of a system that already exists to someone who +has to reason about it without you — a newcomer, an inheriting owner, a future maintainer of +this repo who wasn't around when it was built. + +The format is deliberately narrow: prose for why it exists, diagrams for how it behaves, +tables for surface area, and an explicit list of what will surprise you. Each carries what the +others can't. + +Two principles do most of the work: + +- **Describe behaviour, not intent.** Read the code and say what it does. A report that + documents what the system was supposed to do is worse than no report — the reader trusts it + and debugs against a fiction. +- **The report stands completely alone.** Every reference pointing outside the document is a + place the reader stops and doesn't come back. + +## When to use + +- Onboarding someone onto a service, subsystem, or domain in this repo (`ingestion/`, + `broadcast/`, the auth bridge, the devtools monitor, etc.). +- Handing off ownership — leaving the project, rotating, going on leave. (This is the sole-dev + → future-team case this skill exists for.) +- A neighbouring workstream needs to integrate and keeps asking how something works. +- Knowledge exists only in one session's context, one PR description, or one person's head. +- Post-hoc documentation of something built fast (a suite pushed in one sitting) and never + written down properly. + +**Not for:** point-in-time session status updates — this repo's `docs/handoff-suite-*.md` files +and `notion-sync/STATE.md` are that genre (what shipped, what's uncommitted, what to pick up +next). A handoff-report is durable subsystem knowledge, not a snapshot of where a session left +off. Also not for planning work that hasn't been built yet — use `/write-tickets` for that. + +## Where the finished report lands + +`docs/` is the system of record agents read on every task (per `CLAUDE.md`) — keep it lean and +agent-oriented, don't add human-onboarding material to it. A handoff report is written *for a +person*, so it lands in **`human-docs/`** instead: + +1. Write it to `human-docs/.md` (e.g. `human-docs/ingestion-pipeline.md` — descriptive + name, no suite number or date unless the report is itself dated point-in-time knowledge). +2. Add a row to the table in `human-docs/README.md` — Doc / Purpose / Written — so it's + discoverable. An undiscoverable handoff report is as good as not written. +3. If `docs/*.md` or a per-directory `AGENTS.md` already covers this ground for agents, don't + fork a duplicate — the handoff report can reference and go beyond it (more narrative context, + the sharp edges an agent doc wouldn't include), but state at the top which `docs/` file it + complements so the two don't drift into contradicting each other. + +If the report is a one-off session status note instead (not durable subsystem knowledge), it +follows the `docs/handoff-suite-N-*.md` pattern instead of this skill's structure — that's +agent/continuation-facing state, not human onboarding, and it stays in `docs/`. + +## Grounding — the part that makes it worth reading + +Everything in the report is a claim about a system that exists, so everything is checkable. +Check it. + +- **Read the source before writing the section.** Models in `models.py` for the data model, + `urls.py`/views for interfaces, the actual task/handler for a flow. Never reconstruct + behaviour from a ticket, a PR description, a commit message, or another doc — per + `CLAUDE.md`, if a doc contradicts the code, trust the code and flag the drift. +- **Point at the module, never the line.** Name the file, function, or component — + `ingestion/services.py`, "the `before_save` hook on `StagedEvent`" — so the reader knows + where to look. Never `services.py:88`. Line numbers are wrong within a week of anyone + touching the file, and a citation that's confidently wrong is worse than none: the reader + lands somewhere unrelated and stops trusting the rest. +- **Explain the idea, don't index the code.** The value is in what the module does and why, at + a level that survives refactoring. If a claim can only be supported by pointing at an exact + line, it's too fine-grained for a handoff report — the reader needs the shape, the code shows + the detail. +- **Say what you didn't verify.** "I couldn't find where this is cleaned up" is useful. A + confident guess in the same sentence style as verified facts is not. +- **Date it and name what it reflects** — a commit hash (`git rev-parse HEAD`), a release, or + just the date. A context-transfer doc is read long after it's written; the reader needs to + know how much drift to expect. +- **Prefer the system's own vocabulary.** If the code calls it a `StagedEvent`, don't call it a + "pending record" because it reads better. If it's a "source", don't call it a "feed" unless + the code does. + +## Structure + +Use the sections that carry weight; drop the rest and renumber contiguously. + +1. **What this is and who depends on it** — what the system does in plain terms, who the + callers are, what breaks for them if it's down. Two or three paragraphs, no diagrams. A + reader should be able to stop here and know whether they care. +2. **How it works** — a Mermaid diagram for every flow that earns one, not a representative + sample. Lead with the common case, then the paths people get wrong. Read path before write + path. See "Which flows get a diagram" below. +3. **Data model** — what the tables/collections are, what the non-obvious columns mean, what + nullable actually signifies. +4. **Interfaces** — endpoints, Celery tasks, management commands, webhooks. Who calls each one. +5. **Sharp edges** — the non-obvious behaviour that will bite. See below. +6. **Known gaps** — what's unresolved, undocumented, or actively suspicious, and what you'd + look at first. + +After each diagram, add a short "N things worth calling out" list for what the diagram can't +say: why it's shaped this way, what it costs, what it rules out, and where the obvious-looking +change is the wrong one. + +## Sharp edges + +This is the section that justifies the report, and the one only a current owner can write. It +is not a list of bugs — it's the knowledge that would otherwise be transferred by someone +losing a day. This repo's memory already holds candidates worth mining when writing about a +given subsystem (e.g. the `Event` PK being `uuid` not `id` breaking `Count("id")`, or +`INGEST_SHARD_COUNT` silently limiting a plain `ingest_events` run) — check for standing +gotchas before assuming you've found something new. + +Good entries: a field whose name lies about its contents; a sentinel value overloaded into a +normal column; ordering that looks incidental but isn't; a retry that's safe only because a +downstream call happens to be idempotent; a config that behaves differently in one environment +(dev vs. prod `DJANGO_ENV`, sharded vs. unsharded ingest); a "temporary" workaround load-bearing +enough that removing it breaks something distant. + +Each entry: what it is, why it's that way, what happens if someone "fixes" it. + +## Table shapes + +Describe what exists. Columns adapt to the domain — layer, owner, and consumer columns all +earn their place in different systems. + +| Table | Column | Meaning | Notes | +|---|---|---|---| +| `account` | `locked_until` | Null = not locked | Cleared on successful sign-in alongside the counter, in the sign-in handler (`auth.ts`). | + +| Endpoint | Auth | Called by | Description | +|---|---|---|---| +| `GET /v2/staff/thing` | `view_thing` | Admin settings screen | Returns the caller's groups only; 404s rather than 403s on a foreign id. | + +Name concrete values. "An expired sentinel" is not knowledge — `-infinity` is. If a column is +nullable and carries a magic value, state exactly what distinguishes null from the sentinel; +that ambiguity is a classic sharp edge. + +## If the report covers proposed changes instead + +Same structure, two swaps: add a New/Mod column to the tables so the reader can see what's +changing versus what's already there, and replace Sharp edges with a delivery sequence table +mapping each PR to the diagrams and rows it covers. Everything else — grounding, standalone +rule, diagram discipline — applies unchanged. + +## The standalone rule + +- No "companion to `.md`", no "per §3 of the plan", no local file paths as pointers. The + reader will not open them. Restate what they need inline, in a clause. (Naming a module is + different — that's a signpost for verifying, not a document the reader has to go read.) +- Number sections contiguously. Don't leave a §2→§4 gap advertising a missing piece. +- Every § reference must resolve inside the report. +- If the report is derived from a working doc (a design doc, a ticket thread), treat it as a + published copy. Regenerate when the source changes rather than patching, and expect the two + to diverge in structure. + +## Diagrams + +### Which flows get a diagram + +Diagram every flow that qualifies. Not one showcase diagram and prose for the rest — a +newcomer hits the uncommon paths too, and the retry, the failure, and the migration are exactly +the ones prose handles worst. + +A flow earns a diagram when any of these is true: + +- It crosses two or more components (view → DB → Celery task → worker). +- It branches in a way that changes the outcome (authorized vs. not, cache hit vs. miss, first + run vs. subsequent). +- Ordering matters and isn't obvious from the names. +- State persists between steps, or a step is only safe because an earlier one ran. +- You catch yourself writing "then… then… but if…". That sentence is a diagram you haven't + drawn yet. + +Skip it when the flow is one call and a return, when there's no branch and fewer than three +steps, or when the diagram would restate the sentence above it. Ten diagrams that each earn +their place is a good report; ten that include four trivial ones trains the reader to skip all +of them. + +Diagrams aren't confined to "How it works." A status lifecycle (`StagedEvent` states, a +`SourceRun` health level) belongs beside the data model; a decision tree explaining a sharp edge +belongs next to that sharp edge. Put the diagram where the question gets asked. + +### Don't draw the same diagram twice + +A redundant diagram costs more than a missing one. It pads the report, and it makes the reader +stop and work out whether two near-identical pictures differ in some way that matters. + +Before adding one, check it against what's already drawn: + +- **Same shape, different noun.** If "poll an ICS source" and "poll a scraper source" are the + same six steps with a different fetcher, draw it once and note what varies. +- **Happy path plus one branch.** A failure that diverges for a step or two is an `alt` block + inside the existing diagram, not a second diagram. +- **A subset.** If the read path is the write path with the last three steps removed, the write + diagram already showed it. +- **A table redrawn.** An `erDiagram` that restates the data-model table adds nothing. Keep + whichever one the reader will actually use, not both. + +Split when the shape diverges; merge when only the values do. A branch that changes which +participants are involved, or most of the steps, earns its own diagram. A branch that changes +one call or a label is an `alt`. + +### Picking the type + +Reaching for `sequenceDiagram` every time is the most common way these reports get harder to +read than they need to be. + +| What you're explaining | Type | +|---|---| +| Ordered interaction between components | `sequenceDiagram` | +| One entity moving through statuses over its lifetime | `stateDiagram-v2` | +| Branching logic, a decision tree, a dispatch table | `flowchart TD` | +| How tables/collections relate | `erDiagram` | +| What runs in parallel and what blocks | `flowchart LR` with subgraphs | + +If a "flow" is really "this record can be in one of six states and only some transitions are +legal" (e.g. `StagedEvent.status`), a sequence diagram will fight you the whole way. + +### Keeping them readable + +- One flow per diagram — when the flows genuinely differ in shape. A diagram covering both a + substantial read and a substantial write is two diagrams; one covering a happy path and its + one-step failure is still one. See "Don't draw the same diagram twice." +- Five participants is a lot. Middleware, context objects, and validators are not participants + — fold them into self-calls (`API->>API: check perms`). +- Watch nesting. `alt` inside `alt` inside `loop` renders wide and overflows a tracker's + comment column. Two levels is usually the limit. +- Autonumber so readers can say "step 7" instead of describing a line. + +### Mermaid syntax landmines + +Mermaid renders labels as HTML — which is why `
` works, and why these two break with +errors that don't point at the cause: + +- `;` is a statement separator. A semicolon in message or Note text truncates the statement; the + remainder parses as a new one and throws `Expecting 'SPACE', 'NEWLINE', 'create', 'box'...`. + Use a comma or dash. +- `<...>` is swallowed as an HTML tag. `col = ` silently loses the ``. Use + `(sentinel)`. + +Both are silent — one throws an error pointing nowhere near the cause, the other doesn't error +at all. Don't try to spot them by reading; run the two greps in "Before you publish." + +### Where it renders + +GitHub renders ```` ```mermaid ```` blocks natively — this matters here because handoff reports +land in `human-docs/` and get read on GitHub. VS Code's built-in Markdown preview does not — it +needs an extension, so don't promise it renders "everywhere." + +Don't build an HTML page or separate artifact just to make diagrams render; GitHub already +renders them in `human-docs/`. Build a standalone artifact only when the content is genuinely +interactive, and deliver it through its own channel rather than linking it from a report the +reader can't follow the link from. + +You usually can't verify rendering locally — `mmdc` is rarely installed. Say so and preview on +GitHub (or in the destination surface) before publishing. + +## Process + +1. **Establish who reads it and where it lands.** A newcomer needs the business context; an + integrating workstream needs the interfaces. It's landing in `human-docs/` per "Where the + finished report lands" above, not `docs/` — that decides emphasis, length, and structure. +2. **Read the system-of-record docs first**, per `CLAUDE.md`: root `AGENTS.md` → + `ARCHITECTURE.md` → `CODING_STYLE.md`, plus whichever per-directory `AGENTS.md` or + `docs/*.md` already touches this subsystem. Note where they're stale — flag drift rather + than propagating it. +3. **Read the code and note which modules matter as you go.** Models, urls/views, tasks. This + is most of the work, and skipping it is how reports become fiction. +4. **List every flow before drawing any of them.** Enumerate the paths through the system — + happy path, each failure, each branch, each background job — mark which qualify per "Which + flows get a diagram," then collapse the ones that share a shape. Deciding coverage up front + is what stops the report from diagramming the first flow and prosing the rest, and the + collapse pass is what stops the list from becoming twelve variations on one picture. +5. **Draft context → diagrams → tables → sharp edges → gaps.** Sharp edges accumulate while you + read; keep a running list from step 3. +6. **Reread as the newcomer.** Every unexplained noun is a gap. Every "obviously" is a sharp + edge you forgot to write down. +7. **Run the checks below**, then reread for the standalone rule — that one is invisible to + grep. +8. **Add the row to `human-docs/README.md`** before calling it done — an unlisted doc doesn't + count as written, per "Where the finished report lands." +9. **Have the current owner review before publishing.** They'll catch the confident-but-wrong + sentence, which is the one that does real damage. Don't commit or push it yourself unless + explicitly asked — surface it for review first. + +### Before you publish + +Two mechanical checks worth running literally, because both failures are silent and neither is +reliably caught by reading. Set `F` to the file. + +```bash +# 1. Semicolons inside a diagram — ';' ends the statement, and the rest of the +# line parses as a new one. Error message points nowhere near the cause. +awk '/^```mermaid/{b=1;next} /^```/{b=0} b{print NR": "$0}' "$F" | grep ';' + +# 2. Stray angle brackets — labels render as HTML, so is swallowed +# with no error at all. Strips
and arrows first; any hit is real. +awk '/^```mermaid/{b=1;next} /^```/{b=0} b{print NR": "$0}' "$F" \ + | sed -e 's|
||g' -e 's|[-=][-.=]*>>*||g' | grep '[<>]' +``` + +Both should print nothing. Then check by eye, in rough order of how much damage each does: + +1. **Does it stand alone?** Any pointer to another doc, any § that resolves nowhere, any + section-number gap. This is the failure that wastes the most reader time and no tool sees + it. +2. **Any line numbers?** `grep -nE '\.[a-z]+:[0-9]+' "$F"` — name the module instead. +3. **Do the diagrams cover the flows that matter, without duplicating each other?** Both + directions are failures. +4. **Are the sharp edges the real ones?** If they read like a changelog rather than hard-won + knowledge, the section isn't done. +5. **Is it in `human-docs/README.md`?** + +None of this tells you the report is correct — that's what owner review in step 9 is for, and +it's the check that matters most. diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..ac40d4b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,95 @@ +# Root .dockerignore — applies to any build whose context is the repo root +# (currently: Dockerfile.frontend, context `.`). backendServer/ has its own +# .dockerignore for its own context (backendServer/); this file doesn't +# replace that, it covers the root-context builds. +# +# Keep in sync with .gitignore where the two overlap. The point of this file +# is as much about SECRETS as it is about build speed: the repo root holds a +# real private key (oraclevps.key) and broadcastWeb/ holds a committed cert +# (commons-broadcast.pem) — neither may ever enter a build context. +# +# IMPORTANT — verified empirically against this Docker Desktop/buildkit +# version: a pattern WITHOUT a leading `**/` only matches at the exact +# context-root path, even with no slash in the pattern at all (e.g. bare +# `*.pem` did NOT exclude a nested broadcastWeb/commons-broadcast.pem in a +# throwaway test build — only `**/*.pem` did). This is stricter than plain +# .gitignore semantics, where a no-slash pattern matches at any depth. So +# every pattern below that must also apply inside a subdirectory (which is +# nearly all of them, since theCommonsWeb/, broadcastWeb/, backendServer/ +# are subdirectories of this context) carries an explicit `**/` form — +# don't "simplify" these back down to bare patterns. + +# ── Secrets — must never enter a build context ──────────────────────────── +oraclevps.key +*.key +**/*.key +*.pem +**/*.pem +.env +**/.env +.env.* +**/.env.* +!.env.example +!**/.env.example + +# ── VCS ──────────────────────────────────────────────────────────────────── +.git/ +.gitignore + +# Claude Code worktrees — full duplicate checkouts (own node_modules/.venv) +# used for parallel agent sessions, not application source. Never relevant +# to a build and, left in, they alone can add several hundred MB of stale +# installs to the context. +.claude/worktrees/ + +# ── Node ─────────────────────────────────────────────────────────────────── +# node_modules is reinstalled fresh inside each stage via `pnpm install +# --frozen-lockfile` — a host-built node_modules (wrong OS/arch: dev is +# Apple Silicon, prod is Oracle ARM64; also pnpm's node_modules is a symlink +# farm into a content-addressed store that doesn't travel with the folder) +# must never be copied in. +node_modules/ +**/node_modules/ + +# Build output — produced fresh inside the image, not copied from host. +.next/ +**/.next/ +dist/ +**/dist/ + +# TypeScript incremental build cache (generated, host-specific, diverges on +# every build — see root .gitignore). +*.tsbuildinfo +**/*.tsbuildinfo + +# ── Python / backend cruft (irrelevant to the frontend builds, but a root- +# context build still walks the whole tree unless excluded here) ────────── +backendServer/.venv/ +backendServer/staticfiles/ +**/.venv/ +**/staticfiles/ +__pycache__/ +**/__pycache__/ +*.pyc +**/*.pyc +.mypy_cache/ +**/.mypy_cache/ +.ruff_cache/ +**/.ruff_cache/ +.pytest_cache/ +**/.pytest_cache/ + +# ── Runtime/generated data ────────────────────────────────────────────────── +dump.rdb +broadcast-extension.zip + +# ── Editor/OS cruft ───────────────────────────────────────────────────────── +.DS_Store +**/.DS_Store +.idea/ +**/.idea/ +.vscode/ +**/.vscode/ + +# Local docker-compose bind-mount targets +**/.local-dev/ diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2b11105..c27891d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -208,80 +208,171 @@ jobs: username: ${{ secrets.ORACLE_USER }} key: ${{ secrets.ORACLE_SSH_KEY }} fingerprint: ${{ steps.fp.outputs.fingerprint }} - # Mirrors DEPLOY.md §Deploying Updates (validated dry-run in 16-12-vm-prep). - # uv is at /snap/bin (not on the non-interactive PATH); pnpm is a global - # npm bin already on PATH. set -e aborts on the first failure. + # Mirrors DEPLOY.md §Deploying Updates. Containers via + # docker-compose.yml (T6/T7 of the Dockerization suite) replace the + # old uv-sync + pnpm-build + systemctl-restart flow. Requires the VM + # to have Docker Engine + the `docker compose` v2 plugin, and the + # deploy user in the `docker` group — one-time setup, documented in + # a separate ticket. No more `sudo`: group membership is enough. + # `-f docker-compose.yml` is REQUIRED on every invocation below — a + # bare `docker compose` also auto-loads docker-compose.override.yml + # (local-dev-only: plain HTTP, no cert, repo-relative bind mounts), + # which would deploy the dev config to prod. set -e aborts on the + # first failure. script: | set -e - # CI=true lets pnpm purge/rebuild node_modules non-interactively (no TTY - # over SSH); without it pnpm aborts with ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY. - export CI=true cd /home/ubuntu/thecommons - # pnpm build regenerates tsconfig.tsbuildinfo locally each run, which used - # to be tracked in git and would then block the fast-forward pull below. - # It's a generated TS incremental-build cache, now gitignored — discard any - # local copy so the pull can proceed even before that lands on this VM. + # The real frontend build now happens inside `docker compose + # build` (Dockerfile.frontend), not via a host-side pnpm, but a + # stray tracked/leftover tsconfig.tsbuildinfo from a pre-Docker + # deploy could still block the fast-forward below — cheap + # insurance to discard it before pulling. git checkout -- theCommonsWeb/tsconfig.tsbuildinfo broadcastWeb/tsconfig.tsbuildinfo 2>/dev/null || true git pull origin main - cd backendServer - /snap/bin/uv sync + # Build every image on the VM — arm64-native (the VM is Oracle + # Ubuntu 24.04 ARM64) and needs no registry. nextjs and + # broadcast-spa-build bake NEXT_PUBLIC_*/VITE_* values in as + # build args with safe placeholder defaults (see + # docker-compose.yml's header) — a bare `build` here would + # silently ship those placeholders (verified failure mode: a + # broadcast bundle with no thecommons.town API origin baked in, + # misrouting every call at runtime). Source the real env files + # and re-export under the *_BUILD_* names docker-compose.yml's + # build args actually read. Scoped to this subshell so these + # values (DATABASE_URL, BETTER_AUTH_SECRET, ...) don't linger in + # the wider script env. + ( + set -a + . theCommonsWeb/.env.local + . broadcastWeb/.env + set +a + export NEXTJS_BUILD_DATABASE_URL="$DATABASE_URL" + export NEXTJS_BUILD_BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" + export NEXTJS_BUILD_BETTER_AUTH_URL="$BETTER_AUTH_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_BETTER_AUTH_URL="$NEXT_PUBLIC_BETTER_AUTH_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_API_BASE_URL="$NEXT_PUBLIC_API_BASE_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_THE_COMMONS_API_KEY="$NEXT_PUBLIC_THE_COMMONS_API_KEY" + export BROADCAST_BUILD_VITE_BROADCAST_API_BASE_URL="$VITE_BROADCAST_API_BASE_URL" + export BROADCAST_BUILD_VITE_BETTER_AUTH_URL="$VITE_BETTER_AUTH_URL" + export BROADCAST_BUILD_VITE_BROADCAST_EXTENSION_ID="$VITE_BROADCAST_EXTENSION_ID" + + # Guard: `. env_file` is shell sourcing, NOT a dotenv parser, so a + # BARE value containing a shell metacharacter is mis-parsed rather + # than rejected. The real case this caught: theCommonsWeb/.env.local + # had an unquoted DATABASE_URL whose Neon query string contains `&` + # (`...?sslmode=require&channel_binding=require`). The shell reads + # `VAR=x & y=z` as "assign x in a BACKGROUND subshell, then assign + # y", so DATABASE_URL never reached this shell at all. Without this + # guard the export below sets it to the empty string, compose's + # `${NEXTJS_BUILD_DATABASE_URL:-}` treats empty as + # unset, and the build "succeeds" with a placeholder DB URL baked + # into the image — green pipeline, wrong artifact. + # + # Fix the env file (quote the value); this only makes the failure + # loud. Every var here is required, so empty is always a bug. + missing="" + for v in NEXTJS_BUILD_DATABASE_URL NEXTJS_BUILD_BETTER_AUTH_SECRET \ + NEXTJS_BUILD_BETTER_AUTH_URL NEXTJS_BUILD_NEXT_PUBLIC_BETTER_AUTH_URL \ + NEXTJS_BUILD_NEXT_PUBLIC_API_BASE_URL NEXTJS_BUILD_NEXT_PUBLIC_THE_COMMONS_API_KEY \ + BROADCAST_BUILD_VITE_BROADCAST_API_BASE_URL BROADCAST_BUILD_VITE_BETTER_AUTH_URL; do + eval "val=\${$v:-}" + [ -n "$val" ] || missing="$missing $v" + done + if [ -n "$missing" ]; then + echo "::error::empty build arg(s):$missing — a value in theCommonsWeb/.env.local or broadcastWeb/.env failed to survive shell sourcing (usually a bare value containing & or a space; quote it)" + exit 1 + fi + + docker compose -f docker-compose.yml build + ) + + # Bug-class check, now run against the built image instead of a + # host dist/ dir: a malformed VITE_BROADCAST_API_BASE_URL (e.g. + # missing "//") builds "successfully" but silently misroutes + # every API call. Fail here, before the broken image goes live, + # rather than discovering it later via a confusing 404/405 in + # production. + if ! docker compose -f docker-compose.yml run --rm -T --no-deps --entrypoint sh broadcast-spa-build \ + -c 'grep -rq "https\?://[a-zA-Z0-9.-]*thecommons\.town" /app/dist/assets/*.js'; then + echo "::error::broadcastWeb build does not reference a thecommons.town API origin — check broadcastWeb/.env" + exit 1 + fi # Guarded migrate: `migrate --check` exits non-zero when unapplied # migrations exist and applies nothing (Django 6: "Exits with a # non-zero status if unapplied migrations exist and does not # actually apply migrations"). So prod only writes schema when - # there is real work — and never without a fresh pre-migrate dump. - if /snap/bin/uv run python manage.py migrate --check; then + # there is real work — and never without a fresh pre-migrate + # dump. Django commands run via one-shot containers off the + # image just built above; --no-deps skips starting redis/etc. + # for a plain management command. + if docker compose -f docker-compose.yml run --rm -T --no-deps migrate python manage.py migrate --check; then echo "No unapplied migrations — skipping migrate." else echo "Unapplied migrations detected — plan tail:" - /snap/bin/uv run python manage.py showmigrations --plan | tail -20 - # A silent skip here would defeat the guard: no pg_dump = no - # backup = hard failure, never a warning. - if ! command -v pg_dump >/dev/null 2>&1; then - echo "::error::pg_dump not found on the VM — run: sudo apt install postgresql-client. Refusing to migrate without a pre-migrate backup." - exit 1 - fi + docker compose -f docker-compose.yml run --rm -T --no-deps migrate python manage.py showmigrations --plan | tail -20 + mkdir -p /home/ubuntu/backups - # Subshell so backendServer/.env (DATABASE_URL etc.) never leaks - # into the rest of this script's env or the log (set -x stays - # off). The app's DATABASE_URL already carries Neon's sslmode, - # so pg_dump consumes it as-is. pipefail: a failed pg_dump must - # not leave a truncated-but-"green" gzip behind. This dump is + # pg_dump now runs in a postgres:18-alpine container: the VM no + # longer ships postgresql-client, and the backend image (slim + # Python) never did. Pinned to 18 deliberately — pg_dump can + # dump servers older than itself but refuses newer ones, so the + # newest client stays unconditionally safe against whatever + # version Neon runs. Subshell so backendServer/.env + # (DATABASE_URL etc.) never leaks into the rest of this + # script's env or the log (set -x stays off); `-e + # DATABASE_URL` (no `=value`) passes it into the container by + # reference so the value never appears in argv either. + # `/home/ubuntu/backups` is bind-mounted so the dump lands on + # the host and survives the container being removed (--rm). + # set -o pipefail (verified supported by postgres:18-alpine's + # /bin/sh): a failed pg_dump must not leave a + # truncated-but-"green" gzip behind. A silent skip here would + # defeat the guard: no dump = no backup = hard failure, never a + # warning — set -e (outer) + pipefail (inner) already enforce + # that without a separate tool-existence check. This dump is # belt-and-suspenders — Neon PITR/branching is the real restore # mechanism (see DEPLOY.md). ( - set -o pipefail - set -a; . ./.env; set +a - pg_dump "$DATABASE_URL" | gzip > "/home/ubuntu/backups/pre-migrate-$(date +%Y%m%d-%H%M%S).sql.gz" + set -a; . backendServer/.env; set +a + docker run --rm \ + -e DATABASE_URL \ + -v /home/ubuntu/backups:/backups \ + postgres:18-alpine \ + sh -c 'set -o pipefail; pg_dump "$DATABASE_URL" | gzip > "/backups/pre-migrate-$(date +%Y%m%d-%H%M%S).sql.gz"' ) # Keep only the 5 newest dumps. ls -1t /home/ubuntu/backups/pre-migrate-*.sql.gz | tail -n +6 | xargs -r rm -f -- - /snap/bin/uv run python manage.py migrate --noinput + docker compose -f docker-compose.yml run --rm -T --no-deps migrate python manage.py migrate --noinput fi - /snap/bin/uv run python manage.py collectstatic --noinput - - cd ../theCommonsWeb - pnpm install --frozen-lockfile - pnpm run build - - cd ../broadcastWeb - pnpm install --frozen-lockfile - pnpm run build + # collectstatic is no longer a deploy step — it now runs at image + # build time (backendServer/Dockerfile bakes it into + # /app/staticfiles_build/static, and deploy/nginx/Dockerfile + # COPY --from='s it into the nginx image above). + # + # pnpm install/build are no longer deploy steps either — + # Dockerfile.frontend does both frontend builds as part of the + # `docker compose build` above; there is no host-side + # node_modules or build step left to run. - # Bug-class check: a malformed VITE_BROADCAST_API_BASE_URL (e.g. missing - # "//") builds "successfully" but silently misroutes every API call. - # Fail here, before the broken build goes live, rather than discovering - # it later via a confusing 404/405 in production. - if ! grep -rq 'https\?://[a-zA-Z0-9.-]*thecommons\.town' dist/assets/*.js; then - echo "::error::broadcastWeb build does not reference a thecommons.town API origin — check broadcastWeb/.env" - exit 1 - fi + # Recreate every service from the images just built. No more + # `sudo systemctl restart` — the deploy user is in the `docker` + # group, so this runs unprivileged. + docker compose -f docker-compose.yml up -d - sudo -n systemctl restart gunicorn nextjs celery celerybeat broadcast-worker scrape-worker - systemctl is-active gunicorn nextjs celery celerybeat broadcast-worker scrape-worker + # Health assertion replacing `systemctl is-active`. migrate and + # broadcast-spa-build are one-shot (`restart: "no"`) and exit 0 + # by design, so they're excluded from the "still running" check. + docker compose -f docker-compose.yml ps + running_services=$(docker compose -f docker-compose.yml ps --status running --services) + for svc in redis backend celery celerybeat broadcast-worker scrape-worker nextjs nginx; do + if ! printf '%s\n' "$running_services" | grep -qx "$svc"; then + echo "::error::service '$svc' is not running after deploy — see docker compose ps output above" + exit 1 + fi + done - name: Post-deploy smoke test uses: appleboy/ssh-action@v1 with: @@ -312,6 +403,16 @@ jobs: check "https://thecommons.town/" 200 check "https://broadcast.thecommons.town/" 200 check "https://api.thecommons.town/events/" 200 + # Fourth origin, easy to forget because nothing user-facing points + # at it directly: auth.thecommons.town is what makes the + # .thecommons.town cookie domain work across subdomains (suite 37), + # and backendServer/.env's BETTER_AUTH_JWKS_URL points here, so every + # broadcast JWT verification fetches this exact URL. Probing the JWKS + # endpoint rather than `/` checks the thing the backend depends on. + # Dropping this server block from deploy/nginx/thecommons.conf makes + # the hostname fall through to the www->apex redirect, which the + # garbage-JWT probe below would only report indirectly as a 500. + check "https://auth.thecommons.town/api/auth/jwks" 200 # Regression check for the Unix-socket REMOTE_ADDR gap: nginx proxies # to gunicorn over a Unix socket, which used to leave REMOTE_ADDR diff --git a/.gitignore b/.gitignore index 51fb006..1854c45 100644 --- a/.gitignore +++ b/.gitignore @@ -45,3 +45,6 @@ dump.rdb # Chrome Web Store package (generated by extensionzipper.sh) broadcast-extension.zip + +# Local docker-compose bind-mount targets (docker-compose.override.yml) +.local-dev/ diff --git a/AGENTS.md b/AGENTS.md index 06036de..abe5bbe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,8 +9,11 @@ thecommons/ ├── backendServer/ # Django 6 + DRF — API, ingestion pipeline, broadcast, async │ ├── backend/ # Project config: settings/ (base/dev/prod/test), urls, celery, │ │ # jwt_auth (Better Auth JWKS), permissions, test_runner -│ ├── events/ # Public app: Event/Town/Tag/Category/UserProfile/Business + -│ │ # neon_auth mirrors, digests (Brevo), Redis cache, Celery tasks +│ ├── accounts/ # Identity/auth-bridge app: neon_auth mirrors (BetterAuth*), +│ │ # UserProfile + BusinessProfile, /auth/me, /businesses +│ ├── events/ # Public app: Event/Town/Tag/Category + genuine event views +│ ├── newsletter/ # NewsletterSubscriber, subscribe/manage views, digest engine +│ │ # (email_service, tasks, templates, digest commands) │ ├── ingestion/ # Pipeline: EventSource → RawEvent → StagedEvent → published Event │ ├── broadcast/ # Event syndication: Playwright adapters, DB-queue worker, routing │ ├── templates/ # HTML for admin docs pages + email digests @@ -71,17 +74,17 @@ Run backend + theCommonsWeb together for end-to-end auth (Django validates JWTs | Concern | Key files | Deep dive | |---------|-----------|-----------| -| Auth bridge | `backend/jwt_auth.py`, `backend/permissions.py`, `src/lib/auth.ts` | [ARCHITECTURE.md §Authentication](ARCHITECTURE.md#authentication) | -| Data models | `events/models.py`, `ingestion/models.py`, `broadcast/models.py` | [ARCHITECTURE.md §Data Models](ARCHITECTURE.md#data-models) | -| API endpoints | `backend/urls.py`, `events/urls.py`, `broadcast/urls.py` | [ARCHITECTURE.md §API Endpoints](ARCHITECTURE.md#api-endpoints) | +| Auth bridge | `backend/jwt_auth.py`, `backend/permissions.py`, `accounts/models.py`, `src/lib/auth.ts` | [ARCHITECTURE.md §Authentication](ARCHITECTURE.md#authentication) | +| Data models | `accounts/models.py`, `events/models.py`, `newsletter/models.py`, `ingestion/models.py`, `broadcast/models.py` | [ARCHITECTURE.md §Data Models](ARCHITECTURE.md#data-models) | +| API endpoints | `backend/urls.py` (include()-only), `accounts/urls.py`, `events/urls.py`, `newsletter/urls.py`, `ingestion/urls.py`, `broadcast/urls.py` | [ARCHITECTURE.md §API Endpoints](ARCHITECTURE.md#api-endpoints) | | Ingestion pipeline | `ingestion/services.py`, `ingestion/standardizer.py`, `ingestion/importers/`, `ingestion/safety_scorer.py` | [docs/ingestion-pipeline.md](docs/ingestion-pipeline.md) | | Safety scoring | `ingestion/safety_scorer.py` | [docs/safety-scoring.md](docs/safety-scoring.md) | | Broadcast | `broadcast/services.py`, `broadcast/worker.py`, `broadcast/runner.py`, `broadcast/adapters/` | [docs/broadcast.md](docs/broadcast.md) | -| Redis + Celery | `backend/celery.py`, `events/tasks.py`, `ingestion/tasks.py`, `events/cache.py` | [docs/redis-celery-handoff.md](docs/redis-celery-handoff.md) | +| Redis + Celery | `backend/celery.py`, `newsletter/tasks.py`, `ingestion/tasks.py`, `events/tasks.py`, `events/cache.py` | [docs/redis-celery-handoff.md](docs/redis-celery-handoff.md) | | Frontend data layer | `src/lib/queryClient.ts`, `src/hooks/useEvents.ts`, `src/services/` | [ARCHITECTURE.md §Frontend](ARCHITECTURE.md#frontend-architecture) | -| Email digests | `events/email_service.py`, `events/tasks.py` | [ARCHITECTURE.md §Async](ARCHITECTURE.md#async-redis--celery) | +| Email digests | `newsletter/email_service.py`, `newsletter/tasks.py` (generic transport: `events/email_service.py::send_email`) | [ARCHITECTURE.md §Async](ARCHITECTURE.md#async-redis--celery) | | Design system | `src/app/globals.css`, `src/components/ui/` | [CODING_STYLE.md](CODING_STYLE.md) | -| Admin UI | `events/admin.py`, `ingestion/admin.py` | [docs/admin-backend.md](docs/admin-backend.md) | +| Admin UI | `accounts/admin.py`, `events/admin.py`, `newsletter/admin.py`, `ingestion/admin.py` | [docs/admin-backend.md](docs/admin-backend.md) | | Testing & CI | `backend/settings/test.py`, `.github/workflows/ci.yml`, `vitest.config.ts` | [ARCHITECTURE.md §Testing](ARCHITECTURE.md#testing--ci) | | Dev DB isolation | Neon dev branch + `settings/dev.py` | [docs/dev-db-isolation.md](docs/dev-db-isolation.md) | | Deployment | systemd units, nginx, env vars on VM | [DEPLOY.md](DEPLOY.md) | @@ -91,7 +94,7 @@ Run backend + theCommonsWeb together for end-to-end auth (Django validates JWTs - **Never migrate `neon_auth` tables.** Better Auth (Next.js) owns them. Django mirrors are `managed = False`. - **`Town` and `Category` are SQL tables** — don't hardcode. Pipeline skips events with unknown town slugs. - **Auth lives in Next.js**, not Django. Don't add Django login/signup views or use `django.contrib.auth.User` for app users. -- **`broadcast/` is isolated from `events/`.** `broadcast/routing.py` must not import from `events` (enforced by tests). +- **`broadcast/` and `ingestion/` are isolated from the rest.** No app imports from `ingestion`/`broadcast`; `broadcast/routing.py` must not import from `events` (enforced by tests). `accounts`, `newsletter`, and `events` may read each other where the domain genuinely overlaps — e.g. `accounts.me` writes a `NewsletterSubscriber` row (email-preference sync) and `newsletter._build_recipients` reads `accounts.UserProfile` (tag-filtered digests). Both directions are intentional; don't "fix" the coupling. - **No Django ORM inside `sync_playwright`** — fetch all data into plain objects first, then drive the browser. - **Redis layout is fixed:** DB 0 = Celery broker + results, DB 1 = Django cache. Don't mix them. - **pnpm only for frontends** (pinned to pnpm 11). `npm install` breaks the symlinked store / peer pinning. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b3558da..2dfb3ee 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -19,35 +19,67 @@ Data lives in Postgres on Neon (`public` schema owned by Django, `neon_auth` sch ## Data Models -**Key files:** `events/models.py`, `ingestion/models.py`, `broadcast/models.py` +**Key files:** `accounts/models.py`, `events/models.py`, `newsletter/models.py`, `ingestion/models.py`, `broadcast/models.py` -### `events` app — `public` schema (managed) +### `accounts` app — identity / auth-bridge + +`accounts` owns the Better Auth mirrors and every profile that hangs off a user — a +"business" is modeled as a kind of user profile, not a separate app. | Model | Key fields | Relationships | |-------|-----------|---------------| -| `Tag` | `name` (unique) | M2M from users/businesses/events | -| `Town` | `slug` (unique), `name` | FK target of `Event.town` | -| `Category` | `slug` (unique), `display_name` | M2M with `Event` | -| `UserProfile` | `uuid`, `user_type` (LOCAL/BUSINESS/VENUE), `primary_city`, `address`, `email_preference` (WEEKLY/MONTHLY/NEVER) | OneToOne→`BetterAuthUser` (`db_constraint=False`); M2M→`Tag` | -| `BusinessProfile` | `uuid`, `business_name`, `description`, `contact_email/phone`, `is_published`, timestamps | OneToOne→`BetterAuthUser`; M2M→`Tag`; M2M→`Town` (`service_area`) | -| `NewsletterSubscriber` | `email` (unique), `frequency`, `is_active`, `manage_token` (UUID, unique — unguessable credential for the manage link), `subscribed_at` | — | -| `Event` | `uuid` (PK), `title`, `date` (indexed), `venue`, `description`, `price`, `photo`, `link`, `is_verified`, `source_name` | FK→`Town` (SET_NULL); M2M→`Tag`, `Category`; FK→`BetterAuthUser` (`created_by`) | +| `UserProfile` | `uuid`, `user_type` (LOCAL/BUSINESS/VENUE), `primary_city`, `address`, `email_preference` (WEEKLY/MONTHLY/NEVER) | OneToOne→`BetterAuthUser` (`db_constraint=False`); M2M→`events.Tag`. `db_table="events_userprofile"` (unchanged — model moved app, table didn't) | +| `BusinessProfile` | `uuid`, `business_name`, `description`, `contact_email/phone`, `is_published`, timestamps | OneToOne→`BetterAuthUser`; M2M→`events.Tag`; M2M→`events.Town` (`service_area`). `db_table="events_businessprofile"` | -### Better Auth mirrors — `neon_auth` schema (`managed = False`) +#### Better Auth mirrors — `neon_auth` schema (`managed = False`) -Better Auth (Next.js) owns these tables; Django maps them **read-only** for joins. **Never create migrations for them.** Models: `BetterAuthUser`, `BetterAuthSession`, `BetterAuthAccount`, `BetterAuthVerification`, `BetterAuthJwks`. +Better Auth (Next.js) owns these tables; Django maps them **read-only** for joins. **Never create migrations for them.** Models: `BetterAuthUser`, `BetterAuthSession`, `BetterAuthAccount`, `BetterAuthVerification`, `BetterAuthJwks` — all in `accounts.models`. - The `db_table` values use a double-quote trick (e.g. `'neon_auth"."user'`) so Django emits a valid cross-schema reference `FROM "neon_auth"."user"`. - `BetterAuthUser` hardcodes `is_authenticated=True` / `is_anonymous=False` so DRF permission classes treat it as a real user. - FKs into these mirrors use `db_constraint=False` (no DB-level FK against unmanaged tables). +### `events` app — `public` schema (managed, slimmed) + +`events` now owns only the genuine event/taxonomy models; profile and newsletter models +moved to `accounts`/`newsletter` respectively (state-only migrations — see below). + +| Model | Key fields | Relationships | +|-------|-----------|---------------| +| `Tag` | `name` (unique) | M2M from users/businesses/events | +| `Town` | `slug` (unique), `name` | FK target of `Event.town` | +| `Category` | `slug` (unique), `display_name` | M2M with `Event` | +| `Event` | `uuid` (PK), `title`, `date` (indexed), `venue`, `description`, `price`, `photo`, `link`, `is_verified`, `source_name` | FK→`Town` (SET_NULL); M2M→`Tag`, `Category`; FK→`accounts.BetterAuthUser` (`created_by`) | + +### `newsletter` app — `public` schema (managed) + +| Model | Key fields | Relationships | +|-------|-----------|---------------| +| `NewsletterSubscriber` | `email` (unique), `frequency`, `is_active`, `manage_token` (UUID, unique — unguessable credential for the manage link), `subscribed_at` | — . `db_table="events_newslettersubscriber"` (unchanged) | + +There is a deliberate `accounts ↔ newsletter` coupling, not a boundary bug: `accounts.me` +writes a `NewsletterSubscriber` row (email-preference sync) and +`newsletter._build_recipients` reads `accounts.UserProfile` (tag-filtered digests). Both +directions are intentional and covered by each app's `test_isolation_fast.py` (which forbid +reaching into `ingestion`/`broadcast`, not into each other or `events`). + +### Migration mechanics for the model moves + +All three moves (`UserProfile`/`BusinessProfile` → `accounts`, `NewsletterSubscriber` → +`newsletter`) used `migrations.SeparateDatabaseAndState` with `db_table` preserved +(`events_userprofile`, `events_businessprofile`, `events_newslettersubscriber`) — state-only, +zero physical DDL. The `neon_auth.*` mirrors were never migrated (still `managed=False`). +A companion data migration (`newsletter/migrations/0002_repoint_digest_beat.py`) repoints the +existing `django_celery_beat` `PeriodicTask` rows from `events.tasks.fan_out_*_digest` to +`newsletter.tasks.fan_out_*_digest`. + ### `ingestion` app — `public` schema (managed) | Model | Key fields | Relationships | |-------|-----------|---------------| | `EventSource` | `name`, `source_type` (ics/scraper/email/**direct**), `url`, `active`, `last_polled`, `poll_interval_hours` | reverse `raw_events` | | `RawEvent` | raw title/description/location, raw start/end, `source_url`, `source_uid`, `processed` | FK→`EventSource`; `unique_together=(source, source_uid)` | -| `StagedEvent` | LLM fields (title, description, location, town, datetimes, tags JSON, category, price, link), `status` (pending/approved/rejected/duplicate), `safety_score/notes`, `reviewer_notes` | OneToOne→`RawEvent`; self-FK `duplicate_of`; FK→`events.Event` (`published_event`); FK→`BetterAuthUser` (`submitted_by`) | +| `StagedEvent` | LLM fields (title, description, location, town, datetimes, tags JSON, category, price, link), `status` (pending/approved/rejected/duplicate), `safety_score/notes`, `reviewer_notes` | OneToOne→`RawEvent`; self-FK `duplicate_of`; FK→`events.Event` (`published_event`); FK→`accounts.BetterAuthUser` (`submitted_by`) | ### `broadcast` app — `public` schema (managed) @@ -70,27 +102,43 @@ Better Auth (Next.js) owns these tables; Django maps them **read-only** for join ## API Endpoints -**Key files:** `backend/urls.py`, `events/urls.py`, `broadcast/urls.py` +**Key files:** `backend/urls.py`, `accounts/urls.py`, `events/urls.py`, `newsletter/urls.py`, `ingestion/urls.py`, `broadcast/urls.py` Notes that apply throughout: - **`APPEND_SLASH=False`** — trailing slashes are matched exactly as written below. - **No global DRF config.** Each view sets its own `@authentication_classes` / `@permission_classes` (house pattern). - Auth column: `—` = public, `user` = Better Auth JWT, `API key` = `THE_COMMONS_API_KEY`, `tier≥N` = broadcast tier (Bearer JWT or `X-Broadcast-Access-Code`, resolved by `broadcast/access.py`). +- **`backend/urls.py` is `include()`-only** — it delegates to each app's own urlconf (`accounts.urls`, `events.urls`, `newsletter.urls`, `ingestion.urls`, `broadcast.urls`, plus `admin.site.urls` and, when `DEBUG`, `devtools.urls`). There is no `from .views import ...` in the kernel; the tables below are grouped by the app that actually owns the route. -### Root (`backend/urls.py`) +### accounts (`accounts/urls.py`) | Method | Path | Auth | Purpose | |--------|------|------|---------| -| GET | `/api/cron/ingest` | `CRON_SECRET` | Queue the ingestion pipeline (Celery) | -| POST | `/api/events/publish-approved` | API key | Queue bulk publish of approved staged events | -| POST | `/api/events/direct-submit` | JWT optional (anonymous allowed) | Direct host event submission — fire-and-forget from broadcast SPA; 10/m by IP; invalid token → 401 | | GET/PATCH | `/auth/me` | user | Read / update own profile | -| POST | `/newsletter/subscribe` | — | Newsletter signup (`{email, frequency}`); sends a welcome email with a manage link | -| GET/PATCH | `/newsletter/manage` | — (token) | Manage a subscription via `?token=` — GET returns `{email, frequency, is_active}`; PATCH body `{frequency: WEEKLY\|MONTHLY\|NEVER}` (`NEVER` sets `is_active=false`) | | GET/POST | `/businesses` | user | Browse published businesses / create a listing | | GET | `/businesses/me` | user | Own business listing | | GET/PATCH/DELETE | `/businesses/` | user | Business listing CRUD | + +### newsletter (`newsletter/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| +| POST | `/newsletter/subscribe` | — | Newsletter signup (`{email, frequency}`); sends a welcome email with a manage link | +| GET/PATCH | `/newsletter/manage` | — (token) | Manage a subscription via `?token=` — GET returns `{email, frequency, is_active}`; PATCH body `{frequency: WEEKLY\|MONTHLY\|NEVER}` (`NEVER` sets `is_active=false`) | + +### ingestion (`ingestion/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| +| GET | `/api/cron/ingest` | `CRON_SECRET` | Queue the ingestion pipeline (Celery) | +| POST | `/api/events/publish-approved` | API key | Queue bulk publish of approved staged events | +| POST | `/api/events/direct-submit` | JWT optional (anonymous allowed) | Direct host event submission — fire-and-forget from broadcast SPA; 10/m by IP; invalid token → 401 | | GET/POST | `/admin/docs/...` | staff | Pipeline/admin docs pages + publish-approved button | + +### admin + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| | — | `/admin/` | staff | Django admin (django-unfold) | ### Events (`/events/`) @@ -131,7 +179,7 @@ Auth via Bearer JWT or `X-Broadcast-Access-Code` header, resolved to a tier by ` ## Authentication -**Key files:** `backend/jwt_auth.py`, `backend/permissions.py`, `src/lib/auth.ts`, `src/lib/redirect-allowlist.ts`, `src/hooks/useAuth.tsx`, `src/app/(portal)/`, `src/components/layout/SiteChrome.tsx` +**Key files:** `backend/jwt_auth.py`, `backend/permissions.py`, `accounts/models.py`, `src/lib/auth.ts`, `src/lib/redirect-allowlist.ts`, `src/hooks/useAuth.tsx`, `src/app/(portal)/`, `src/components/layout/SiteChrome.tsx` Auth is owned by **Better Auth running inside Next.js**, fronted by a standalone **portal** — there are no Django login/signup endpoints, and no app renders its own embedded auth form anymore. Django only *verifies* tokens. @@ -149,12 +197,12 @@ Every service that needs a user to authenticate redirects into the portal with ` - Browser authenticates with Better Auth and holds a session cookie. - To call Django, the frontend fetches a short-lived **JWT** from `/api/auth/token` (Better Auth `jwt()` plugin) and sends it as `Authorization: Bearer `. JWT payload carries the `email` claim by default (`sub` = user id). - `BearerTokenAuthentication` accepts either: - 1. a **Better Auth JWT** verified statelessly against the JWKS endpoint (`BETTER_AUTH_JWKS_URL`); `sub` resolves to a `BetterAuthUser`. The JWKS client is cached in-process with a TTL and **stale-grace** fallback so brief Next.js outages don't cascade. In `broadcast/`, `verify_better_auth_jwt` is called directly — no `BetterAuthUser` ORM lookup (isolation preserved). + 1. a **Better Auth JWT** verified statelessly against the JWKS endpoint (`BETTER_AUTH_JWKS_URL`); `sub` resolves to a `BetterAuthUser` (now in `accounts.models` — `backend/permissions.py` imports it from there, not `events.models`). The JWKS client is cached in-process with a TTL and **stale-grace** fallback so brief Next.js outages don't cascade. In `broadcast/`, `verify_better_auth_jwt` is called directly — no `BetterAuthUser` ORM lookup (isolation preserved). 2. the shared **`THE_COMMONS_API_KEY`** (no user attached) — for app-level calls like event creation. - Permission classes live in `backend/permissions.py` and are applied per-view alongside DRF's `IsAuthenticated`. ### User-creation side effect -`src/lib/auth.ts` defines `databaseHooks.user.create.after`, which inserts a matching `public.events_userprofile` row whenever Better Auth creates a user — so every account has a Django profile. +`src/lib/auth.ts` defines `databaseHooks.user.create.after`, which inserts a matching `public.events_userprofile` row whenever Better Auth creates a user — so every account has a Django profile. The table name is a historical artifact: the `UserProfile` model now lives in `accounts/models.py`, but its `db_table` was pinned to `events_userprofile` by a state-only migration, so the physical table (and this INSERT) is unchanged. ### Account creation is password-required Signup collects **email + password + confirm** in one step on `/join`, via Better Auth's standard `emailAndPassword` flow (`autoSignIn: true` in `src/lib/auth.ts`) — `signUp.email` creates the Better Auth user + `credential` account and signs the user in immediately; the `databaseHook` fires as usual. There is no passwordless/email-only path and no separate set-password step. **No email verification for MVP.** @@ -198,21 +246,21 @@ Flow: tier-based auth (Bearer JWT or access code, resolved by `broadcast/access. ## Async: Redis + Celery -**Key files:** `backend/celery.py`, `backend/__init__.py`, `events/tasks.py`, `ingestion/tasks.py`, `events/cache.py`, `events/signals.py` +**Key files:** `backend/celery.py`, `backend/__init__.py`, `newsletter/tasks.py`, `events/tasks.py`, `ingestion/tasks.py`, `events/cache.py`, `events/signals.py` - **One Redis instance, two logical DBs:** DB 0 = Celery broker **and** result backend (`REDIS_URL`); DB 1 = Django cache (`RedisCache`, `REDIS_CACHE_URL`). - **Celery** app is built in `backend/celery.py`, loaded eagerly via `backend/__init__.py`, and autodiscovers tasks. `CELERY_TIMEZONE = UTC` (beat entries carry their own tz). -- **Beat** uses `django_celery_beat`'s `DatabaseScheduler` — schedules live in Postgres and are editable in admin. Seeded by migrations: - - `weekly-digest-sunday` → `events.tasks.fan_out_weekly_digest`, Sun 18:00 America/New_York (`events/migrations/0015_seed_digest_beat.py`). - - `monthly-digest` → `events.tasks.fan_out_monthly_digest`, 1st of month 18:00 America/New_York (`events/migrations/0020_seed_monthly_digest_beat.py`). +- **Beat** uses `django_celery_beat`'s `DatabaseScheduler` — schedules live in Postgres and are editable in admin. Seeded by migrations, then repointed by a data migration when the digest engine moved apps: + - `weekly-digest-sunday` → `newsletter.tasks.fan_out_weekly_digest`, Sun 18:00 America/New_York (seeded by `events/migrations/0015_seed_digest_beat.py`; repointed from `events.tasks.fan_out_weekly_digest` by `newsletter/migrations/0002_repoint_digest_beat.py`). + - `monthly-digest` → `newsletter.tasks.fan_out_monthly_digest`, 1st of month 18:00 America/New_York (seeded by `events/migrations/0020_seed_monthly_digest_beat.py`; repointed by the same `0002_repoint_digest_beat.py`). - `ingest-events-daily` → `ingestion.tasks.run_ingestion_pipeline`, 04:00 America/New_York (`ingestion/migrations/0007_seed_ingest_beat.py`). -- **Tasks:** `events.tasks` (`ping`, `send_one_digest`, `fan_out_weekly_digest`, `fan_out_monthly_digest`), `ingestion.tasks` (`run_ingestion_pipeline`, `publish_all_approved_task`). +- **Tasks:** `newsletter.tasks` (`send_one_digest`, `fan_out_weekly_digest`, `fan_out_monthly_digest`), `events.tasks` (`ping`), `ingestion.tasks` (`run_ingestion_pipeline`, `publish_all_approved_task`). - **Read-endpoint cache:** `events/cache.py` is a version-keyed Redis cache for the hot list endpoints; `events/signals.py` bumps the version on `Event`/`Town`/`Category` writes to invalidate. See [docs/redis-celery-handoff.md](docs/redis-celery-handoff.md). ### Email digests -`events/email_service.py` wraps **Brevo** transactional email and builds digest HTML from `templates/email/`. `NewsletterSubscriber` is the single source of truth for both weekly and monthly digests: `_build_recipients(frequency)` resolves the recipient list (deduped by email) from active subscriber rows — anonymous newsletter subscribers get all events, account holders (`UserProfile.email_preference`) are tag-filtered — and returns `{email, tags, manage_token}` per recipient. This one resolver backs both the Celery path (`fan_out_weekly_digest` / `fan_out_monthly_digest` queue one `send_one_digest` per recipient) and the synchronous `send_digest`/`send_weekly_digest` management commands (`send_test_digest` sends a one-off test). Every digest email carries a "Manage preferences / Unsubscribe" link built from the recipient's `manage_token` (`/newsletter/manage?token=`). +`newsletter/email_service.py` wraps **Brevo** (via the generic transport, `events/email_service.py::send_email`) and builds digest HTML from `newsletter/templates/email/`. `NewsletterSubscriber` (`newsletter` app) is the single source of truth for both weekly and monthly digests: `_build_recipients(frequency)` resolves the recipient list (deduped by email) from active subscriber rows — anonymous newsletter subscribers get all events, account holders (`accounts.UserProfile.email_preference`) are tag-filtered — and returns `{email, tags, manage_token}` per recipient. This one resolver backs both the Celery path (`newsletter.tasks.fan_out_weekly_digest` / `fan_out_monthly_digest` queue one `send_one_digest` per recipient) and the synchronous `send_digest`/`send_weekly_digest` management commands in `newsletter/management/commands/` (`send_test_digest` sends a one-off test). Every digest email carries a "Manage preferences / Unsubscribe" link built from the recipient's `manage_token` (`/newsletter/manage?token=`). `events/email_service.py::send_email` remains the generic Brevo wrapper used by non-digest commands — it did not move. --- @@ -262,7 +310,7 @@ A separate Vite + React 19 SPA (`broadcastWeb/`) for the broadcast operator cons **Key files:** `backend/settings/{base,dev,prod,test}.py` Settings are split by `DJANGO_SETTINGS_MODULE`: -- `base.py` — shared: installed apps (unfold, corsheaders, DRF, the 3 local apps, `django_celery_beat`), CORS allowlist (+ custom `x-broadcast-access-code` header), `APPEND_SLASH=False`, Celery/Redis config, unfold admin. +- `base.py` — shared: installed apps (unfold, corsheaders, DRF, the 5 local apps — `accounts`, `events`, `newsletter`, `ingestion`, `broadcast`; `devtools` is added by `dev.py` only — `django_celery_beat`), CORS allowlist (+ custom `x-broadcast-access-code` header), `APPEND_SLASH=False`, Celery/Redis config, unfold admin. - `dev.py` — `DEBUG=True`, parses `DATABASE_URL` (Neon dev branch), console email, `BROADCAST_AUTOSPAWN_WORKER=true` by default. - `prod.py` — `DEBUG=False`; requires `DJANGO_SECRET_KEY`, `DJANGO_ALLOWED_HOSTS`, `DATABASE_URL`. - `test.py` — inherits dev; strips `-pooler` from the DB host (Neon direct endpoint so the test DB can be created/dropped), eager Celery, locmem cache, stubbed external creds. See [§Testing](#testing--ci). diff --git a/DEPLOY.md b/DEPLOY.md index 3aef808..f5fdfcd 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -1,11 +1,20 @@ # Deployment Runbook — The Commons This is the single source of truth for deploying The Commons to the production VM. -Follow **Part 1** top-to-bottom for the **first deploy** of the new -Redis/Celery/broadcast/CI infrastructure (the VM does not have it yet). After that, +The stack is **containerized** (Docker Compose) — this replaces the old systemd-unit +runbook. Follow **Part 1** top-to-bottom for the **one-time VM prep** that gates the +first automated container deploy (the VM does not have Docker on it yet). After that, deploys are automatic — see **Part 2**. **Part 3** is reference (services, env vars, nginx, firewall, troubleshooting). +> ⚠️ **Every prod Compose command in this document passes `-f docker-compose.yml` +> explicitly, and so must you.** A bare `docker compose` (no `-f`) auto-loads +> `docker-compose.override.yml` from the same directory — that file is **local-dev +> only** (plain HTTP nginx with no cert, repo-relative bind-mount paths, `DJANGO_ENV=dev`). +> Running it against the VM would deploy the dev config to prod: no TLS, wrong +> hostnames, wrong Redis auth. If you ever type `docker compose` without `-f +> docker-compose.yml` on the VM, stop and re-type it. + --- ## Facts you need first @@ -16,534 +25,696 @@ nginx, firewall, troubleshooting). | VM IP | `129.80.229.41` | | SSH | `ssh -i oraclevps.key ubuntu@129.80.229.41` (key is in repo root — **never commit it**) | | Repo path on VM | `/home/ubuntu/thecommons` | -| Deploy user | `ubuntu` (all services run as `ubuntu`) | -| Python | **`uv`** at `/snap/bin/uv` — never `pip` — but **only for one-shot commands** (`uv sync`, `manage.py migrate`, etc.); long-lived systemd units exec `.venv/bin/` directly, never `uv run` (see §4) | -| Node | **`pnpm`** — never `npm install` (it breaks peer-dependency pinning) | -| DNS / TLS | Cloudflare, proxied (orange cloud), SSL mode **Full (strict)**; origin cert at `/etc/ssl/cloudflare/thecommons.town.{pem,key}` | - -> **What changed since `main`:** this release adds Redis, a Celery worker + beat -> scheduler, the broadcast worker (Playwright), a healthcheck, and a CI/CD -> auto-deploy. The ingestion and weekly-digest jobs **moved off OS cron** onto -> `django-celery-beat` (a DB-backed scheduler seeded by migrations). If you followed -> any older notes that set up `crontab`/`logrotate`/`/var/log/thecommons` for those -> two jobs, that approach is **dead** — Part 1 §9 retires it. +| Deploy user | `ubuntu` (member of the `docker` group; every container runs as `ubuntu`'s Docker daemon, unprivileged) | +| Container runtime | Docker Engine + the `docker compose` v2 plugin (arm64). **No `sudo` in the deploy path** — group membership is enough. | +| Python / Node on the VM | **Not used directly anymore.** `uv` and `pnpm` still matter for local dev and CI (they build the images), but the VM never runs `uv sync`, `manage.py migrate` bare, or `pnpm build` outside a container — see Part 2. | +| DNS / TLS | Cloudflare, proxied (orange cloud), SSL mode **Full (strict)**; origin cert at `/etc/ssl/cloudflare/thecommons.town.{pem,key}`, bind-mounted read-only into the `nginx` container | + +> **What changed since the last runbook:** the seven systemd units (`gunicorn`, +> `nextjs`, `redis-server`, `celery`, `celerybeat`, `broadcast-worker`, +> `scrape-worker`) plus the hand-edited nginx config are replaced by +> [`docker-compose.yml`](docker-compose.yml) — one file describing the whole +> service graph, built and run with `docker compose`. `git pull` + `uv sync` + +> `pnpm build` + `sudo systemctl restart …` is gone from the deploy path entirely; +> CI now runs `docker compose -f docker-compose.yml build` and +> `docker compose -f docker-compose.yml up -d`. See +> [`docs/adr/0001-containerization.md`](docs/adr/0001-containerization.md) for the +> full rationale behind each decision (nginx-in-a-container, Redis-in-a-container, +> Postgres staying external on Neon, what's baked vs. volumed). --- -# Part 1 — First deploy (manual, one time) - -Do these in order. The CI auto-deploy in Part 2 **cannot** succeed until this is -done, because it restarts `celery`/`celerybeat`/`broadcast-worker`/`scrape-worker`, -which don't exist on the box yet. - -## 1. Pull the new code onto the VM +# Part 1 — One-time VM prep (blocking) + +**This gates the first automated container deploy.** The `deploy` job in +[`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs `docker compose -f +docker-compose.yml build` and `up -d` on every push to `main` — that fails +immediately if the VM doesn't have Docker, the `ubuntu` user can't run it without +`sudo`, or the bind-mount directories/env files it depends on don't exist. Do all +eleven steps in order before the first push to `main` after this cutover. + +> ⚠️ **If you check a feature branch out on the VM to rehearse this before +> merging, expect an immediate 500 on `api.thecommons.town`.** The systemd +> gunicorn is still running and imports Python modules lazily *from the working +> tree*, so a `git checkout` swaps files underneath a process whose settings are +> already loaded in memory. A branch carrying an app-layout change (the suite-41 +> `accounts`/`newsletter` extraction is the live example) then produces +> `RuntimeError: Model class accounts.models.BetterAuthUser doesn't declare an +> explicit app_label and isn't in an application in INSTALLED_APPS` — new file, +> old `INSTALLED_APPS`. `sudo systemctl restart gunicorn` immediately after the +> checkout reloads both consistently and clears it. Restart the Celery units too +> if the branch moved any task modules. None of this applies once the stack is +> containerized: an image is a snapshot, so a checkout can't change code out from +> under a running container. + +## 1. Install Docker Engine + the Compose v2 plugin (arm64) ```bash ssh -i oraclevps.key ubuntu@129.80.229.41 -cd /home/ubuntu/thecommons -git fetch origin -git checkout testing+ci && git pull origin testing+ci +sudo apt-get update +sudo apt-get install -y ca-certificates curl +sudo install -m 0755 -d /etc/apt/keyrings +sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc +sudo chmod a+r /etc/apt/keyrings/docker.asc + +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update + +sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` -> You provision from the `testing+ci` branch now. In §10 you'll switch the VM to -> `main` so CI's `git pull` tracks the right branch going forward. +`dpkg --print-architecture` resolves to `arm64` automatically on this VM — no +manual platform pin needed; every image in this stack (`python:3.13-slim`, +`node:22-slim`, `redis:7-alpine`, `nginx:1.27-alpine`, `postgres:18-alpine`) ships +a maintained arm64 build (`docs/adr/0001-containerization.md` §Base images). -## 2. Provision Redis (one time) +Verify: ```bash -sudo apt update && sudo apt install -y redis-server -openssl rand -hex 32 # copy the output — this is -sudo nano /etc/redis/redis.conf # set the four lines below +docker --version +docker compose version # the `compose` SUBCOMMAND must work — that's the plugin, not the old standalone `docker-compose` binary. + # Don't assert a specific major here; what matters is that `docker compose` (space, not hyphen) + # resolves at all. Verified installed on this VM: Docker 29.7.1 / Compose v5.3.1. ``` -In `/etc/redis/redis.conf` set: +## 2. Add `ubuntu` to the `docker` group — and prove it non-interactively -``` -bind 127.0.0.1 -::1 -requirepass -maxmemory 512mb -maxmemory-policy allkeys-lru +```bash +sudo usermod -aG docker ubuntu ``` +**Group membership only applies to sessions started *after* this command runs.** +The SSH session you just ran `usermod` in still has the old group list — testing +`docker ps` in that same session (or papering over it with `newgrp docker`) proves +nothing about how CI will actually connect. CI's `appleboy/ssh-action` opens a +brand-new non-interactive SSH connection on every run (`ssh host 'command'`, not an +interactive login shell), so that's exactly what you must test: + ```bash -sudo systemctl enable --now redis-server && sudo systemctl restart redis-server -redis-cli -a '' PING # → PONG +exit # close the current SSH session completely +ssh -i oraclevps.key ubuntu@129.80.229.41 'id -nG; docker version --format "{{.Server.Version}}"' ``` -One Redis instance, two logical DBs: **DB 0** = Celery broker + results, **DB 1** = -read-endpoint cache. The password lives **only** in `backendServer/.env`, never in git. - -## 3. Backend — env, deps, migrate, static +`id -nG` must list `docker`, and the second command must print a server version +with **no `sudo` and no permission-denied error** on the Docker socket. If it +fails, the group add didn't take (check `getent group docker`) or the session +really is stale — reconnect fresh and retry. -Edit `backendServer/.env` and add the Redis URLs (note the **leading `:`** and no -username). Full env reference is in Part 3. +## 3. Create the persistent host directories (bind mounts) -``` -REDIS_URL=redis://:@127.0.0.1:6379/0 -REDIS_CACHE_URL=redis://:@127.0.0.1:6379/1 -INGEST_SHARD_COUNT=3 # optional — spreads source polling across 3 days; set 1 or omit to poll all daily +```bash +mkdir -p /home/ubuntu/broadcast/{media,screenshots,downloads} /home/ubuntu/backups ``` -**`INGEST_SHARD_COUNT=3` is a deliberate prod setting — keep it at 3.** `_resolve_env_shard` -(`backendServer/ingestion/tasks.py:22-39`) polls only the sources where -`id % INGEST_SHARD_COUNT == day_of_year % INGEST_SHARD_COUNT`, so with a shard count -of 3 any given source is actually polled roughly every **72 hours**, not the 24h its -own `poll_interval_hours` implies — sharding is disabled (all sources polled daily) -only when the var is unset, fails to parse as an int, or is `<= 1`. Two consequences -to keep in mind: +These back real state that must survive a container restart — client-uploaded +event images, Playwright debug artifacts, and pre-migrate `pg_dump`s +(`docs/adr/0001-containerization.md` §Decision 4). `docker-compose.yml` bind-mounts +them at the identical absolute path inside the `backend`, `broadcast-worker`, +`scrape-worker`, `migrate`, and `nginx` containers. -- A source that fails transiently gets **1/3 as many chances** to recover before the - next poll compared to unsharded daily polling. -- Monitoring/alerting thresholds must be shard-aware — a source that looks "3 days - stale" is expected and healthy under this setting, not a symptom of an outage. (A - companion ticket is making the ingestion monitor account for this.) +They must be **writable by uid 1000** — the image's non-root `app` user +(`backendServer/Dockerfile`: `useradd --uid 1000 --gid app …`) — **and** by the +host's `ubuntu` user, which is what the deploy pipeline runs as. -```bash -cd /home/ubuntu/thecommons/backendServer -/snap/bin/uv sync -/snap/bin/uv run python manage.py migrate # also seeds the beat schedules -/snap/bin/uv run python manage.py collectstatic --noinput -``` - -The `migrate` step runs `ingestion/migrations/0007_seed_ingest_beat.py` (ingest, -04:00 ET daily) and `events/migrations/0015_seed_digest_beat.py` (digest, Sun -18:00 ET), which create the scheduled tasks in the database. - -## 4. Celery worker + beat services (one time) - -**The `uv` rule for this project has two cases — don't collapse them:** - -- **One-shot commands** (`uv sync`, `uv run python manage.py migrate`, `playwright - install`, etc.) — always go through `/snap/bin/uv`. That part of the Facts table - above is unchanged. -- **Long-lived systemd units** (`celery`, `celerybeat`, `scrape-worker`, - `broadcast-worker`) — **never** `uv run`. They exec the venv binary directly, - `/home/ubuntu/thecommons/backendServer/.venv/bin/celery …`, the same pattern - `gunicorn` has always used (`.venv/bin/gunicorn`). This is not a style preference: - snap-packaged `uv run` spawns its child process inside a transient - `snap.astral-uv.uv-*.scope` under `user@1001.service` — the *user* manager, not the - unit's own cgroup. With `Linger=no`, systemd-logind tears down that user slice the - moment the deploying SSH session ends, and the child (celery/beat) receives SIGTERM - and performs a clean warm shutdown — **`status=0/SUCCESS`**, which - `Restart=on-failure` correctly declines to restart. That is exactly how the async - stack died silently on 2026-07-21 12:46 UTC and stayed down for 8 days while the - site looked healthy: every `uv run`-based unit died within a minute of the deploy - session ending; `gunicorn` (execs its venv binary) and `nextjs` (`/usr/bin/npm`) - never went through snap and stayed up 68 days straight. See - `docs/prod-incident-2026-07-21-scheduler-outage.md` for the full forensics. - -Two complementary mitigations are now in place, not alternatives — keep both: - -1. **`.venv/bin/celery` in `ExecStart`** (all four unit files) removes the mechanism - entirely — the process never lands in a user-manager slice, so there's nothing for - a lost SSH session to tear down. -2. **`Restart=always`** (all four units) + **linger enabled for `ubuntu`** are - defense-in-depth, in case a unit is ever reverted to `uv run` or linger is somehow - turned back off: - ```bash - sudo loginctl enable-linger ubuntu - loginctl show-user ubuntu --property=Linger # must print Linger=yes - ``` +> ⚠️ **On this VM `ubuntu` is uid 1001, not 1000.** The Oracle Ubuntu image ships +> an `opc` user that already holds uid 1000 (`getent passwd 1000 1001` to see it). +> So a plain `sudo chown -R 1000:1000` — the obvious reading of "writable by uid +> 1000" — hands these directories to `opc` and locks `ubuntu` *out* of them. That +> silently breaks the backup-pruning step in Part 2 §4 +> (`ls -1t /home/ubuntu/backups/… | xargs rm`, which runs as `ubuntu`), while the +> containers themselves stay perfectly happy — a failure that only shows up as +> backups quietly accumulating forever. Don't assume; check. + +Give the container the ownership and the host user the group, with setgid so +newly created files keep inheriting that group: ```bash -cd /home/ubuntu/thecommons -sudo cp deploy/celery.service deploy/celerybeat.service /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now celery celerybeat -sudo systemctl status celery celerybeat # both active (running) -sudo journalctl -u celery -n 30 # should show "celery@... ready" + broker connected +getent passwd 1000 1001 # confirm who actually holds each uid +sudo chown -R 1000:1001 /home/ubuntu/broadcast /home/ubuntu/backups +sudo chmod -R 2775 /home/ubuntu/broadcast /home/ubuntu/backups ``` -Run **exactly one** beat process. The worker drains Redis DB 0; beat uses -`django-celery-beat`'s DatabaseScheduler, so schedules are editable later in the -Django admin. - -**The real acceptance test is a full SSH login→logout cycle, not `systemctl -is-active` right after `enable --now`.** A unit that was just started by your current -session is active regardless of whether the underlying bug is fixed — that's the -exact false-positive that let this incident go unnoticed for 8 days. Log out -completely and reconnect fresh before you trust the result: +That leaves uid 1000 (the container's `app` user) as **owner**, and gid 1001 +(`ubuntu`) as a **group** with write — so both sides can write the same trees. Use +numeric ids rather than names: the container's `app` user is fixed at uid 1000 +regardless of what the host calls that uid. Verify both directions before moving +on: ```bash -exit # close the SSH session completely -ssh -i oraclevps.key ubuntu@129.80.229.41 'systemctl is-active celery celerybeat scrape-worker broadcast-worker' -# all four must print "active" from a session that did NOT start them +touch /home/ubuntu/broadcast/media/.wtest && rm /home/ubuntu/broadcast/media/.wtest # host `ubuntu` +docker run --rm -u 1000:1000 -v /home/ubuntu/broadcast/media:/m alpine \ + sh -c 'touch /m/.ctest && rm /m/.ctest' # container uid 1000 ``` -Confirm every queue has a consumer. A queue with depth but no consumer is -silent — jobs enqueue and are never dispatched: +## 4. Confirm the Cloudflare origin cert is in place ```bash -cd /home/ubuntu/thecommons/backendServer -uv run celery -A backend inspect active_queues | grep -o "'name': '[a-z]*'" # expect celery, scrape, broadcast +ls -l /etc/ssl/cloudflare/thecommons.town.pem /etc/ssl/cloudflare/thecommons.town.key ``` -## 5. Broadcast feature (one time) +The `nginx` container bind-mounts `/etc/ssl/cloudflare` read-only +(`docker-compose.yml`'s `nginx.volumes`) — it is **never** baked into the image +(a rebuild-and-repush on every cert rotation, and the private key ending up in a +pullable image layer, are exactly what a bind mount avoids — +`docs/adr/0001-containerization.md` §Decision 1). If this cert isn't already on the +box from the earlier broadcast-subdomain rollout, reissue a +`thecommons.town, *.thecommons.town` origin cert in the Cloudflare dashboard and +place it here before continuing — `nginx` refuses to boot with `listen 443 ssl` +blocks pointing at missing files. -The broadcast feature adds a third subdomain, a Playwright worker, and a static SPA. -Do all seven in order. - -> **Local dev note:** `BROADCAST_AUTOSPAWN_WORKER` no longer exists — Suite 25 moved -> broadcast dispatch onto on-demand Celery (`transaction.on_commit(process_broadcast_queue.delay)`), -> so there's nothing to autospawn. To drain broadcast jobs locally, either run -> `celery -A backend worker -Q broadcast -c 1 -l info` alongside `runserver`, or set -> `CELERY_TASK_ALWAYS_EAGER=True` (as `settings/test.py` does) to execute tasks -> synchronously with no worker at all. See [`docs/broadcast.md`](docs/broadcast.md). - -1. **TLS (do this first).** The existing origin cert covers only `thecommons.town` - and `api.thecommons.town`. In the Cloudflare dashboard, reissue an origin cert - for `thecommons.town, *.thecommons.town`, replace - `/etc/ssl/cloudflare/thecommons.town.{pem,key}`, then: - ```bash - sudo nginx -t && sudo systemctl reload nginx - ``` - Skipping this makes the subdomain fail with a Cloudflare 526 error. -2. **DNS.** Cloudflare → add an **A record** `broadcast` → `129.80.229.41`, proxied - (orange cloud). An A record, not a CNAME. -3. **Playwright Chromium** (bundled only — arm64 has no branded Chrome): - ```bash - cd /home/ubuntu/thecommons/backendServer - /snap/bin/uv run playwright install chromium - /snap/bin/uv run playwright install-deps chromium # apt system libs (uses sudo) - ``` -4. **Artifact dirs:** - ```bash - mkdir -p /home/ubuntu/broadcast/{screenshots,downloads} - ``` -5. **Env.** Add the `BROADCAST_*` block from `backendServer/.env.example` to - `backendServer/.env`, and append `https://broadcast.thecommons.town` to both - `CORS_EXTRA_ORIGINS` and `CSRF_TRUSTED_ORIGINS`. -6. **Worker service.** Broadcast dispatch is on-demand Celery, not a poll loop: - the app calls `transaction.on_commit(process_broadcast_queue.delay)` on - submit/retry, routed to a dedicated `broadcast` Celery queue - (`CELERY_TASK_ROUTES` in `backend/settings/base.py`). `deploy/broadcast-worker.service` - runs `celery -A backend worker -Q broadcast -c 1 -l info` — **`-c 1` is - mandatory, not a tuning default**: `recover_orphans()` assumes any `running` - submission at startup is orphaned, so a second concurrent worker could race - a live queue-drain. A `broadcast-orphan-recovery` beat task (seeded by - migration `0009_seed_orphan_recovery_beat.py`) sweeps orphaned submissions - every 6 hours as a crash-recovery net; normal stalled-target recovery is - client-driven from the SPA (`POST /broadcast/jobs//retry-stuck`) and - doesn't involve this worker restarting. - ```bash - cd /home/ubuntu/thecommons - sudo cp deploy/broadcast-worker.service /etc/systemd/system/ - sudo systemctl daemon-reload && sudo systemctl enable --now broadcast-worker - ``` -7. **nginx.** Add the server block from `deploy/nginx-broadcast.conf.snippet` into - the **existing** `/etc/nginx/sites-available/thecommons` (one file, many `server` - blocks — do not create a new sites-available file), then: - ```bash - sudo nginx -t && sudo systemctl reload nginx - ``` - -## 6. Scrape worker (one time) - -The ingestion scraper (`ingestion.tasks.scrape_all_sources_task`) renders headless -Chromium and is routed to a dedicated `scrape` Celery queue (`CELERY_TASK_ROUTES` in -`backend/settings/base.py`) so its memory never lands on the default `celery` -worker. `deploy/celery.service`'s `ExecStart` has no `-Q` flag (`celery -A backend -worker -l info --concurrency=2`), so the default worker only drains the `celery` -queue — it will **not** pick up `scrape` tasks. Without this dedicated worker, -scrape tasks queue forever. - -1. **Playwright Chromium** (skip if already installed for broadcast — same shared - cache): - ```bash - cd /home/ubuntu/thecommons/backendServer - /snap/bin/uv run playwright install chromium - /snap/bin/uv run playwright install-deps chromium # apt system libs (uses sudo) - ``` - Bundled Chromium only — **never** a branded "chrome" channel, unsupported on - arm64. It installs into `/home/ubuntu/.cache/ms-playwright/`, shared with - `broadcast-worker`, so there's no extra download if that's already in place. -2. **Env.** Add the three `INGEST_SCRAPER_*` vars to `backendServer/.env` if you - want non-default values (all optional — see Part 3 for defaults). -3. **Worker service:** - ```bash - cd /home/ubuntu/thecommons - sudo cp deploy/scrape-worker.service /etc/systemd/system/ - sudo systemctl daemon-reload && sudo systemctl enable --now scrape-worker - sudo systemctl status scrape-worker # active (running) - ``` - -> ⚠️ **Operator ordering matters.** From this point on, `.github/workflows/ci.yml`'s -> `deploy` job restarts `scrape-worker` on every push to `main` (see §10/Part 2). -> If you haven't installed and `systemctl enable`d the unit (and run `playwright -> install chromium`) **before** the first deploy that includes this change, CI's -> `sudo -n systemctl restart ... scrape-worker` step will fail and block the -> deploy. Do steps 1–3 above first. - -## 7. Frontend builds + restart +## 5. Confirm the three runtime env files exist on the box ```bash -cd /home/ubuntu/thecommons/theCommonsWeb -pnpm install && pnpm run build -sudo systemctl restart nextjs +ls -l /home/ubuntu/thecommons/backendServer/.env \ + /home/ubuntu/thecommons/theCommonsWeb/.env.local \ + /home/ubuntu/thecommons/broadcastWeb/.env +``` + +These are consumed two different ways, and it matters which: + +- **`env_file:`** in `docker-compose.yml` (all Django/Celery services, plus + `nextjs`) loads them into the container's runtime environment when it starts. +- **Build args** (`nextjs` and `broadcast-spa-build`'s `NEXT_PUBLIC_*`/`VITE_*` + values) get baked into the compiled JS at *build* time via Compose variable + interpolation, which only reads the shell environment or a `.env` file next to + `docker-compose.yml` — it **cannot** read `theCommonsWeb/.env.local` or + `broadcastWeb/.env` directly. Part 2 covers how the deploy pipeline bridges this + (`source` the real files, re-export under the `NEXTJS_BUILD_*`/`BROADCAST_BUILD_*` + names the build args read). + +If any of the three files is missing, create it from its `.env.example` and fill +in real values before proceeding — every other step assumes they're already +correct except step 6, which calls out the one change that's mandatory. + +> ⚠️ **Quote any value containing `&`, a space, or `#`.** The build path +> (`set -a; . theCommonsWeb/.env.local`) is **shell sourcing, not a dotenv +> parser** — it does not reject a malformed line, it *mis-parses* it. A bare +> Neon URL is the live example, because its query string contains `&`: +> +> ``` +> DATABASE_URL=postgres://…?sslmode=require&channel_binding=require # BROKEN +> DATABASE_URL='postgres://…?sslmode=require&channel_binding=require' # correct +> ``` +> +> The shell reads `VAR=x & y=z` as "assign `x` in a **background subshell**, +> then assign `y`", so `DATABASE_URL` never reaches the calling shell. Compose's +> `env_file:` parses the same file correctly, so the running containers look +> fine — only the *build args* are wrong, and compose's `${VAR:-placeholder}` +> defaults turn that into a build that succeeds with placeholder config baked +> in. `backendServer/.env` already quotes its `DATABASE_URL`; +> `theCommonsWeb/.env.local` did not, which is what surfaced this. +> +> Audit all three files at once — this compares each key's literal value against +> what sourcing actually yields, and prints no secrets: +> +> ```bash +> cd /home/ubuntu/thecommons +> for f in backendServer/.env theCommonsWeb/.env.local broadcastWeb/.env; do +> echo "=== $f ===" +> awk '/^[A-Za-z_][A-Za-z0-9_]*=/ {k=substr($0,1,index($0,"=")-1); v=substr($0,index($0,"=")+1); +> if ((substr(v,1,1)=="\"" && substr(v,length(v),1)=="\"") || (substr(v,1,1)=="'"'"'" && substr(v,length(v),1)=="'"'"'")) v=substr(v,2,length(v)-2); +> print k"\t"length(v)}' "$f" > /tmp/lit +> ( set -a; . "$f" >/dev/null 2>&1; set +a +> while IFS=$'\t' read -r k n; do eval "cur=\${$k-__UNSET__}" +> [ "$cur" = "__UNSET__" ] && { echo " $k BROKEN (unset after sourcing)"; continue; } +> [ "${#cur}" = "$n" ] || echo " $k TRUNCATED ($n -> ${#cur})" +> done < /tmp/lit ) +> done +> ``` +> +> The `deploy` job also fails loudly now if any required build arg comes out +> empty (`.github/workflows/ci.yml`), but fix the file rather than relying on +> that backstop. + +## 6. ⚠️ Blocking: point `backendServer/.env` at the container Redis, not localhost + +**This is the step this whole suite exists to get right — treat it as a release +blocker, not a nit.** `backendServer/.env` almost certainly still has: -cd ../broadcastWeb -pnpm install && pnpm run build # static → dist/, served directly by nginx (no service) - -sudo systemctl restart gunicorn +``` +REDIS_URL=redis://:@127.0.0.1:6379/0 +REDIS_CACHE_URL=redis://:@127.0.0.1:6379/1 ``` -## 8. Verify +Inside a container, `127.0.0.1`/`localhost` is *that container itself* — Redis now +runs in its own container reachable only by its Compose service name. Leaving +these as-is does **not** error at startup: Celery just silently connects to +nothing and no task ever runs, while gunicorn, nginx, and everything synchronous +stays green. That is the exact shape of the 2026-07-21 outage (async stack dead, +everything else looked healthy) — see §Historical incident below. Fix it: + +``` +REDIS_URL=redis://:@redis:6379/0 +REDIS_CACHE_URL=redis://:@redis:6379/1 +``` + +The `` embedded in both URLs must match a `REDIS_PASSWORD` entry +you add to the same file — the `redis` container reads `REDIS_PASSWORD` directly +(`docker-compose.yml`'s `redis.command`) and starts `--requirepass` with it; Django +and Celery never read `REDIS_PASSWORD` itself, only the URLs above, which already +carry the password in their `:@` segment. If it isn't already present: + +``` +REDIS_PASSWORD= # same value embedded in the two URLs above +``` + +While you're in this file, confirm two more things `prod.py` hard-requires — +missing either crashes every container on boot, not just Celery: + +- **`DJANGO_ALLOWED_HOSTS`** is set (`backend/settings/prod.py:16` does + `os.environ["DJANGO_ALLOWED_HOSTS"]` — a hard `KeyError`, not a default, if it's + absent). Expected value: `localhost,127.0.0.1,api.thecommons.town`. + `docker-compose.override.yml` (local dev only) sets `DJANGO_ENV=dev` specifically + to dodge this same crash on a laptop that has no reason to set this var — that + workaround does not apply on the VM. +- **`DJANGO_ENV=prod`** is set. `docker-compose.yml` also sets it via + `environment: DJANGO_ENV: prod` on every backend/Celery service, so this is + belt-and-suspenders — but a stray unset/wrong value in `.env` that later + overrides it (env_file loads before `environment:` in Compose precedence, + so `environment:` wins) is not worth relying on. Confirm it's `prod` in the file + too. + +`manage.py healthcheck` pings Redis DB 0 and round-trips DB 1, so the hourly +watchdog (Part 3 §Health check) will eventually catch a missed edit here — but the +point of calling this out as its own step is to not discover it that way. + +> ⚠️ **This edit breaks the still-running host stack the moment you save it.** +> Steps 7–9 haven't happened yet, so the *systemd* gunicorn/celery are still +> serving production out of this same `.env` — and on the host there is no such +> hostname as `redis`, so every cached read starts throwing +> `redis.exceptions.ConnectionError: Error -3 connecting to redis:6379. +> Temporary failure in name resolution` and `api.thecommons.town/events/` 500s. +> The step reads like inert preparation; it isn't. Bridge it before you edit, so +> the host stack keeps resolving `redis` to the local `redis-server` until the +> containers take over: +> +> ```bash +> grep -q '^127.0.0.1[[:space:]]\+redis\b' /etc/hosts || echo '127.0.0.1 redis' | sudo tee -a /etc/hosts +> sudo systemctl restart gunicorn celery celerybeat broadcast-worker scrape-worker +> ``` +> +> Containers are unaffected either way — they resolve `redis` through Docker's +> embedded DNS inside their own namespace and never consult the host's +> `/etc/hosts`. **Remove the line once step 9 has retired the host units** +> (`sudo sed -i '/^127\.0\.0\.1[[:space:]]\+redis$/d' /etc/hosts`); leaving it +> behind is harmless but will mislead whoever debugs this box next. + +## 7. First manual bring-up — everything except `nginx` + +Bring the non-ingress services up by hand once, before CI ever touches the box, +so you can see them start clean without racing the host-nginx cutover in the next +step: ```bash cd /home/ubuntu/thecommons -UV_BIN=/snap/bin/uv bash deploy/healthcheck.sh # everything should be ✓ +docker compose -f docker-compose.yml up -d --build redis migrate backend celery celerybeat broadcast-worker scrape-worker nextjs +docker compose -f docker-compose.yml ps ``` -Then smoke-test the two scheduled jobs by hand (they still exist as manual triggers): +`migrate` is a one-shot container (`restart: "no"`) — it applies migrations +(seeding the `django_celery_beat` schedule tables in the process) and exits 0; +`docker compose ps` will show it `Exited (0)`, which is correct, not a failure. +Every other service listed should show `running`. Then run the full report: ```bash -sudo -u ubuntu bash -lc 'cd /home/ubuntu/thecommons/backendServer && /snap/bin/uv run python manage.py ingest_events --shard 0/3 --skip-standardize --skip-dedup --skip-safety --skip-autopublish' -sudo -u ubuntu bash -lc 'cd /home/ubuntu/thecommons/backendServer && /snap/bin/uv run python manage.py send_test_digest --email aryav@unc.edu' -curl -I https://broadcast.thecommons.town/ # expect 200/3xx +bash deploy/healthcheck.sh ``` -If the broadcast `curl` hangs, it's the iptables REJECT-before-ACCEPT gotcha — see -Part 3 §Firewall. +Expect `redis`, `backend`, `celery`, `celerybeat`, `broadcast-worker`, and +`scrape-worker` all `✓`, plus the app-level Redis/Postgres/Celery-ping/beat-schedule +checks from `manage.py healthcheck` running *inside* the `backend` container. +`nextjs` and `nginx` will `✗` here — `nginx` hasn't been started yet (next step), +and that's expected at this point. -## 9. Retire the old OS cron (only if it exists) +## 8. Cutover: stop the host nginx, start the container nginx -Beat now owns the ingest + digest schedules. If the box still has the old cron -lines, remove them so the jobs don't run twice: +**This is the one step in the whole suite where a mistake produces an immediate, +visible outage** (`docs/adr/0001-containerization.md` §Decision 1) — two processes +cannot bind ports 80/443 at once, so the container nginx cannot come up cleanly +until the host one is out of the way. ```bash -crontab -l # look for ingest_events / send_weekly_digest lines -crontab -e # delete those two lines if present +sudo systemctl disable --now nginx +docker compose -f docker-compose.yml up -d --build nginx +docker compose -f docker-compose.yml ps ``` -`healthcheck.sh` also flags leftover cron lines for these jobs. - -## 10. Switch the VM to `main` and enable CI/CD - -**On your laptop** — merge and push: +Verify all three domains from outside the box: ```bash -git checkout main && git merge testing+ci && git push origin main +curl -I https://thecommons.town/ +curl -I https://api.thecommons.town/events/ +curl -I https://broadcast.thecommons.town/ ``` -**On the VM** — point it at `main` so CI's `git pull` works: +If `broadcast.thecommons.town` 526s, the origin cert doesn't cover the wildcard — +back to step 4. If any of the three hangs, it's the iptables +REJECT-before-ACCEPT gotcha — see Part 3 §Firewall (unchanged by containerization: +the container nginx still binds the host's 80/443 the same way the old one did). -```bash -cd /home/ubuntu/thecommons && git checkout main && git pull -``` +## 9. Retire the old systemd units and the sudoers drop-in -**Deploy SSH key (laptop):** +Now that the containers own gunicorn, Next.js, and all four Celery roles, the old +units would only cause confusion (or a port clash) if left enabled: ```bash -ssh-keygen -t ed25519 -C "github-actions-deploy@thecommons" -f ~/.ssh/thecommons_deploy -N "" -cat ~/.ssh/thecommons_deploy.pub | ssh -i oraclevps.key ubuntu@129.80.229.41 \ - 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys' -ssh -i ~/.ssh/thecommons_deploy ubuntu@129.80.229.41 'echo deploy-key-ok' +sudo systemctl disable --now celery celerybeat broadcast-worker scrape-worker gunicorn nextjs +sudo rm -f /etc/systemd/system/{celery,celerybeat,broadcast-worker,scrape-worker,gunicorn,nextjs}.service +sudo systemctl daemon-reload +sudo rm -f /etc/sudoers.d/deploy-restart ``` -**Sudoers drop-in (VM)** — passwordless restart for exactly the six units: +Leave `redis-server` alone if it's still `apt`-installed from before — stop and +disable it too, since the `redis` container now owns port 6379: ```bash -sudo visudo -f /etc/sudoers.d/deploy-restart +sudo systemctl disable --now redis-server 2>/dev/null || true ``` -``` -ubuntu ALL=(root) NOPASSWD: /usr/bin/systemctl restart gunicorn, \ - /usr/bin/systemctl restart nextjs, \ - /usr/bin/systemctl restart celery, \ - /usr/bin/systemctl restart celerybeat, \ - /usr/bin/systemctl restart broadcast-worker, \ - /usr/bin/systemctl restart scrape-worker -``` +`healthcheck.service`/`healthcheck.timer` are **not** part of this retirement — +they were rewritten to check containers (`docker compose ps` / `docker inspect`) +rather than `systemctl is-active`, and still run as a host-level systemd timer on +purpose (Part 3 §Health check explains why). Leave them installed, or install them +now if this is the first time — see that section. + +## 10. Confirm the GitHub Actions secrets -**GitHub repo secrets** (Settings → Secrets and variables → Actions) — all four: +The deploy job only needs the four secrets it already had — nothing new, since +Docker commands run unprivileged via the `docker` group instead of the old +`systemctl`-restart sudoers allowlist: | Secret | Value | |--------|-------| -| `ORACLE_SSH_KEY` | Full PEM **private** key (`~/.ssh/thecommons_deploy`, incl. BEGIN/END lines) | +| `ORACLE_SSH_KEY` | Full PEM **private** key for the deploy key (incl. BEGIN/END lines) | | `ORACLE_HOST` | `129.80.229.41` (raw IP — Cloudflare won't proxy port 22) | | `ORACLE_USER` | `ubuntu` | -| `ORACLE_KNOWN_HOSTS` | The **single** line from `ssh-keyscan -t ed25519 129.80.229.41` | +| `ORACLE_KNOWN_HOSTS` | Output of `ssh-keyscan 129.80.229.41` (all key types — `.github/workflows/ci.yml`'s fingerprint step needs ECDSA/RSA/ED25519 all present, since `appleboy/ssh-action`'s Go SSH client negotiates ECDSA first) | -**Confirm the non-interactive PATH** sees the tools CI will call: +Confirm the deploy key still authenticates and the non-interactive Docker check +from step 2 still passes through it specifically (not just `oraclevps.key`): ```bash -ssh -i ~/.ssh/thecommons_deploy ubuntu@129.80.229.41 'command -v uv; command -v pnpm; command -v node' +ssh -i ~/.ssh/thecommons_deploy ubuntu@129.80.229.41 'id -nG; docker version --format "{{.Server.Version}}"' ``` -If `uv` isn't found, CI uses `/snap/bin/uv` (the workflow already accounts for this). +## 11. Trigger and verify the first automated deploy -## 10. Trigger and verify the first automated deploy - -The `deploy` job runs only on **push to `main`**, after `backend`, -`frontend-commons`, and `frontend-broadcast` pass. +The `deploy` job runs only on **push to `main`**, gated on `backend`, +`frontend-commons`, and `frontend-broadcast` all passing. - Push to `main` (or re-run the Action). Watch **Actions → CI**: three green test jobs → `deploy` starts. -- The `deploy` log should show each step: `git pull`, `uv sync`, `migrate`, - `collectstatic`, both `pnpm` builds, the restart, and `is-active` lines all - printing `active`. -- Confirm the VM is on the pushed commit: +- The `deploy` log should show, in order: `git pull`, `docker compose … build` (all + services), the broadcast-bundle origin grep check, the guarded-migrate check, + `docker compose … up -d`, the running-services assertion, then a separate + post-deploy smoke-test job (three domain checks, the rate-limit regression probe, + the auth-bridge probes). See Part 2 for what each of those actually does. +- Confirm the VM is on the pushed commit and every long-running service is up: ```bash ssh -i ~/.ssh/thecommons_deploy ubuntu@129.80.229.41 \ - 'cd /home/ubuntu/thecommons && git log -1 --oneline' + 'cd /home/ubuntu/thecommons && git log -1 --oneline && docker compose -f docker-compose.yml ps' ``` > **Budget 30–60 min for the first automated run.** It usually trips on something -> environmental (a PATH gap, a sudoers path mismatch, `--frozen-lockfile` drift). -> Read the failing step's log, fix on the VM or in the workflow, push again. +> environmental — a bind-mount permission gap, a stale `.env`, `--frozen-lockfile` +> drift in `uv.lock`/`pnpm-lock.yaml`. Read the failing step's log +> (`docker compose -f docker-compose.yml logs ` on the VM reproduces most +> failures locally), fix it, push again. --- # Part 2 — Ongoing deploys (automatic) -Once Part 1 is done, **every push to `main`** runs CI (`.github/workflows/ci.yml`) -and, after all three test jobs pass, a gated `deploy` job SSHes into the VM and runs -the full sequence: `git pull` → `uv sync` → guarded `migrate` (below) → `collectstatic` -→ both frontend `pnpm install`/`build` → restart `gunicorn nextjs celery celerybeat -broadcast-worker scrape-worker`, then a post-deploy smoke test: all three domains, `/events/`, -auth probes (`/auth/me` with no/garbage token must 401/403 — never 500 — and the -API-key path must accept the real key), and a broadcast rate-limit regression check. -A failing test on `main` blocks the deploy. - -> **Guarded migrate:** the deploy runs `migrate --check` first and skips `migrate` -> entirely when nothing is pending. When migrations *are* pending it logs the plan, -> takes a `pg_dump` to `/home/ubuntu/backups/pre-migrate-.sql.gz` (keeps -> the 5 newest), and only then applies. It **fails the deploy** if `pg_dump` is -> missing — one-time VM prep: `sudo apt install -y postgresql-client`. The dump is -> belt-and-suspenders; Neon PITR/branching is the real restore mechanism. +Once Part 1 is done, **every push to `main`** runs CI +(`.github/workflows/ci.yml`) and, after `backend`, `frontend-commons`, and +`frontend-broadcast` all pass, a gated `deploy` job SSHes into the VM and runs the +full container build-and-release sequence, followed by a separate post-deploy +smoke test. A failing test on `main` blocks the deploy entirely — nothing below +runs until all three test jobs are green. + +**On the VM, in order:** + +1. **`git pull origin main`** (after discarding any stray tracked + `tsconfig.tsbuildinfo` that could block the fast-forward — a leftover from a + pre-Docker deploy). +2. **Build every image**, arm64-native, no registry needed — the VM builds what it + runs: + ```bash + ( + set -a + . theCommonsWeb/.env.local + . broadcastWeb/.env + set +a + export NEXTJS_BUILD_DATABASE_URL="$DATABASE_URL" + export NEXTJS_BUILD_BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" + export NEXTJS_BUILD_BETTER_AUTH_URL="$BETTER_AUTH_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_BETTER_AUTH_URL="$NEXT_PUBLIC_BETTER_AUTH_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_API_BASE_URL="$NEXT_PUBLIC_API_BASE_URL" + export NEXTJS_BUILD_NEXT_PUBLIC_THE_COMMONS_API_KEY="$NEXT_PUBLIC_THE_COMMONS_API_KEY" + export BROADCAST_BUILD_VITE_BROADCAST_API_BASE_URL="$VITE_BROADCAST_API_BASE_URL" + export BROADCAST_BUILD_VITE_BETTER_AUTH_URL="$VITE_BETTER_AUTH_URL" + export BROADCAST_BUILD_VITE_BROADCAST_EXTENSION_ID="$VITE_BROADCAST_EXTENSION_ID" + docker compose -f docker-compose.yml build + ) + ``` + This re-export is required because Compose's `${VAR:-default}` interpolation + only reads the shell env or a root-level `.env` — it cannot read + `theCommonsWeb/.env.local`/`broadcastWeb/.env` directly (see `docker-compose.yml`'s + header comment). Skipping it silently ships the safe *placeholder* build-arg + defaults instead of real config — a build that "succeeds" but is wrong. +3. **Guard against exactly that**: grep the built `broadcast-spa-build` bundle for + a real `thecommons.town` API origin. A malformed `VITE_BROADCAST_API_BASE_URL` + builds fine and misroutes every API call silently — this catches it before the + image goes live: + ```bash + docker compose -f docker-compose.yml run --rm -T --no-deps --entrypoint sh broadcast-spa-build \ + -c 'grep -rq "https\?://[a-zA-Z0-9.-]*thecommons\.town" /app/dist/assets/*.js' + ``` +4. **Guarded migrate.** `migrate --check` exits non-zero only when unapplied + migrations exist, and applies nothing itself — so prod only ever writes schema + when there's real work, and never without a fresh dump first: + ```bash + docker compose -f docker-compose.yml run --rm -T --no-deps migrate python manage.py migrate --check + ``` + If that fails (migrations pending): take a `pg_dump` via a throwaway + `postgres:18-alpine` container (the VM has no host-installed Postgres client — + see the doc-drift correction below), keep the 5 newest, then apply: + ```bash + mkdir -p /home/ubuntu/backups + ( + set -a; . backendServer/.env; set +a + docker run --rm \ + -e DATABASE_URL \ + -v /home/ubuntu/backups:/backups \ + postgres:18-alpine \ + sh -c 'set -o pipefail; pg_dump "$DATABASE_URL" | gzip > "/backups/pre-migrate-$(date +%Y%m%d-%H%M%S).sql.gz"' + ) + ls -1t /home/ubuntu/backups/pre-migrate-*.sql.gz | tail -n +6 | xargs -r rm -f -- + docker compose -f docker-compose.yml run --rm -T --no-deps migrate python manage.py migrate --noinput + ``` + `postgres:18-alpine` is pinned deliberately — `pg_dump` can dump a server + *older* than itself but refuses one *newer*, so the newest client image stays + safe regardless of what Postgres version Neon runs, with no version-matching + exercise to redo on every Neon upgrade. This dump is belt-and-suspenders; Neon + PITR/branching is the real restore mechanism. + + `collectstatic` and both frontend builds are **not** separate deploy steps + anymore — `collectstatic` runs at image build time (baked into + `/app/staticfiles_build/static`, then `COPY --from=`'d into the `nginx` image), + and both `pnpm build`s happened inside step 2 above. There is no host-side + `node_modules` or static directory left to manage. +5. **Recreate every service from the images just built** — no `sudo`, no + `systemctl restart`: + ```bash + docker compose -f docker-compose.yml up -d + ``` +6. **Health assertion.** `migrate` and `broadcast-spa-build` are one-shot + (`restart: "no"`) and correctly exit 0, so they're excluded from the + "still running" check; every long-running service must be: + ```bash + docker compose -f docker-compose.yml ps + running_services=$(docker compose -f docker-compose.yml ps --status running --services) + for svc in redis backend celery celerybeat broadcast-worker scrape-worker nextjs nginx; do + printf '%s\n' "$running_services" | grep -qx "$svc" || { echo "$svc not running"; exit 1; } + done + ``` +7. **Post-deploy smoke test** (separate SSH step, so a deploy that "succeeded" but + serves garbage still fails CI): the three domains return 200, the broadcast + rate-limit path returns 403 (not 500 — regression-checks the old + Unix-socket-`REMOTE_ADDR` bug now that gunicorn is on TCP), `/auth/me` 401/403s + on no/garbage credentials (never 500), and a real `THE_COMMONS_API_KEY` against + `/events/create` gets past auth to a 400 body-validation error. See the + `deploy`/`Post-deploy smoke test` steps in `.github/workflows/ci.yml` for the + literal script. ### Manual fallback (CI down, or a hand hotfix) ```bash -cd /home/ubuntu/thecommons && git pull +cd /home/ubuntu/thecommons && git pull origin main -# Backend (only the steps that apply) -cd backendServer -/snap/bin/uv sync # if pyproject.toml changed -/snap/bin/uv run python manage.py migrate # if models changed -/snap/bin/uv run python manage.py collectstatic --noinput # if static changed -sudo systemctl restart gunicorn -sudo systemctl restart celery celerybeat # if task code or deps changed +docker compose -f docker-compose.yml build # rebuild whatever changed +docker compose -f docker-compose.yml run --rm --no-deps migrate python manage.py migrate --noinput # if models changed +docker compose -f docker-compose.yml up -d # recreate from the new images +docker compose -f docker-compose.yml ps +``` -# Frontend -cd ../theCommonsWeb && pnpm install && pnpm run build && sudo systemctl restart nextjs +To restart a single service without touching the rest (e.g. picked up a `.env` +change with no code change): -# Broadcast -cd ../backendServer && sudo systemctl restart broadcast-worker -cd ../broadcastWeb && pnpm install && pnpm run build # static — no service to restart +```bash +docker compose -f docker-compose.yml up -d --no-deps +``` + +If you only edited nginx's config (`deploy/nginx/thecommons.conf`), the change +needs a rebuild — it's baked into the image at build time, not bind-mounted, in +prod: -# Scrape worker (if scraper/task code changed) -cd ../backendServer && sudo systemctl restart scrape-worker +```bash +docker compose -f docker-compose.yml build nginx +docker compose -f docker-compose.yml up -d --no-deps nginx ``` --- # Part 3 — Reference +## Compose services + +| Service | What it is | Image / build target | Command | Notes | +|---------|-----------|----------------------|---------|-------| +| `redis` | Celery broker + results (DB 0) + read-endpoint cache (DB 1) | `redis:7-alpine` | `redis-server --requirepass … --appendonly yes` | Named volume `redis-data`; password from `REDIS_PASSWORD` in `backendServer/.env` | +| `migrate` | One-shot: `manage.py migrate --noinput`, seeds `django_celery_beat` schedules | `backendServer/Dockerfile` target `app` | `python manage.py migrate --noinput` | `restart: "no"`; exits 0 by design — not a long-running service | +| `backend` | Django / gunicorn | `backendServer/Dockerfile` target `app` | `gunicorn --bind 0.0.0.0:8000 --workers 3 backend.wsgi:application` | TCP `:8000`, internal only (`expose`, not `ports`) — no more unix socket | +| `celery` | Default async worker (digest emails, etc. — everything not routed to `broadcast`/`scrape`) | same `app` image | `celery -A backend worker -n commons-default@%h -l info --concurrency=2` | `mem_limit: 1g` | +| `celerybeat` | Scheduler | same `app` image | `celery -A backend beat -l info` | DatabaseScheduler; **exactly one** process — do not scale this service | +| `broadcast-worker` | Playwright form-filler, drains the `broadcast` queue | `backendServer/Dockerfile` target `playwright` | `celery -A backend worker -Q broadcast -n commons-broadcast@%h -c 1 -l info` | `mem_limit: 2g`; **`-c 1` is mandatory**, not tuning — `recover_orphans()` assumes a single worker | +| `scrape-worker` | Ingestion scraper (headless Chromium), drains the `scrape` queue | `backendServer/Dockerfile` target `playwright` | `celery -A backend worker -Q scrape -n commons-scrape@%h -c 1 -l info` | `mem_limit: 2g`; keeps Chromium memory off the default worker | +| `nextjs` | theCommonsWeb (Next.js) | `Dockerfile.frontend` target `commons-runtime` | `node server.js` | Port 3000, internal only | +| `broadcast-spa-build` | Build-only: produces the compiled broadcastWeb SPA for `nginx` to `COPY --from=` | `Dockerfile.frontend` target `broadcast-build` | `true` (no-op) | `restart: "no"`; never actually runs as a service — exists so `docker compose up` doesn't crash-loop on a stray start | +| `nginx` | Single ingress for all three subdomains | `deploy/nginx/Dockerfile` | (stock `nginx:1.27-alpine` entrypoint) | **Only** service publishing host ports (`80:80`, `443:443`); bakes in both `backend`'s `collectstatic` output and `broadcast-spa-build`'s `dist/` via Buildx named contexts | + +All images build for **arm64**, matching the VM +(`docs/adr/0001-containerization.md` §Base images). Every long-running service's +`CMD`/`command` execs the real binary directly (no wrapper shell) so the +container's PID 1 receives `SIGTERM` on `docker stop`/`down` and shuts down +cleanly — the one piece of the old systemd `ExecStart=.venv/bin/celery …` +convention worth keeping, for an unrelated reason to why it existed originally +(see §Historical incident). + +The three Celery **workers** pass `-n commons-{default,scrape,broadcast}@%h`. +Without it they'd all default to `celery@`, and duplicate nodenames make +`celery -A backend inspect|control` ambiguous (`DuplicateNodenameWarning`, and a +`revoke` can misroute). Verify all three are distinct: + +```bash +docker compose -f docker-compose.yml exec backend celery -A backend inspect ping +# expect three distinct nodes: commons-default@, commons-scrape@, commons-broadcast@ +``` + +**Logs and status — the container replacements for `journalctl`/`systemctl`:** + +```bash +docker compose -f docker-compose.yml ps # replaces `systemctl status ` +docker compose -f docker-compose.yml logs # replaces `journalctl -u -n 50` +docker compose -f docker-compose.yml logs -f # add -f to follow, same as journalctl -f +docker compose -f docker-compose.yml restart # replaces `systemctl restart ` +``` + +Logging is plain `json-file` with rotation (`max-size: 10m`, `max-file: 3` — the +`x-logging` anchor at the top of `docker-compose.yml`), since there's no +`journald` inside a container. + ## Health check -One command prints a scannable report of the whole box — RAM/disk, every systemd -unit, Redis, Postgres, the Celery worker, and whether the beat schedule is firing: +Unchanged interface, container-aware internals. One command still prints a +scannable report of the whole box: ```bash cd /home/ubuntu/thecommons -UV_BIN=/snap/bin/uv bash deploy/healthcheck.sh -UV_BIN=/snap/bin/uv bash deploy/healthcheck.sh --no-color | tee /tmp/health.log -``` - -It checks (✓/!/✗): RAM/disk vs thresholds; `systemctl is-active` for `redis-server`, -`celery`, `celerybeat`, `gunicorn`, `nextjs`, `broadcast-worker`, `scrape-worker`; leftover OS-cron -lines; and via `manage.py healthcheck` — Postgres `SELECT 1`, Redis broker ping -(DB 0), Django cache round-trip (DB 1), a Celery worker `control.ping`, and each -seeded `PeriodicTask` (enabled + last-run freshness: daily within ~25h, weekly -within ~8d). A stale or never-run schedule is a **`FAIL`**, not a warning — it was -a warning until the 2026-07-21 outage, which is precisely why a month-dead beat -still exited 0. A seeded task with no window configured in `DEFAULT_STALENESS_HOURS` -reports `WARN` rather than silently passing. It exits non-zero on any critical -failure. Tunables: -`RAM_WARN`/`RAM_FAIL` (80/95), `DISK_WARN`/`DISK_FAIL` (80/95), `CELERY_TIMEOUT` -(1.0s), `UV_BIN` (default `uv`; the VM has it at `/snap/bin/uv`). The Django command -also runs standalone: `/snap/bin/uv run python manage.py healthcheck [--json]`. - -### Scheduled health check (systemd timer) - -`deploy/healthcheck.sh` catches the failure modes that let the 2026-07-21 scheduler -outage run silent for five weeks — but only if something actually runs it. A systemd -timer does that hourly, independent of Celery beat (beat is one of the things being -checked, so it can't be the thing doing the checking). +bash deploy/healthcheck.sh +bash deploy/healthcheck.sh --no-color | tee /tmp/health.log +``` + +It checks (✓/!/✗): RAM/disk vs. thresholds; `docker compose ps`/`docker inspect` +liveness (running vs. restarting vs. exited, health status, restart count) for +every long-running service in the table above (`migrate` and +`broadcast-spa-build` are deliberately excluded — both are one-shot and exiting 0 +is correct, not a failure); leftover legacy OS-cron lines; and, via +`docker compose exec -T backend python manage.py healthcheck`, Postgres +`SELECT 1`, Redis broker ping (DB 0), Django cache round-trip (DB 1), a Celery +worker `control.ping`, and each seeded `PeriodicTask`'s freshness (daily within +~25h, weekly within ~8d). A stale or never-run beat schedule is a **`FAIL`**, not a +`WARN` — that's precisely the bug class that let the 2026-07-21 outage run silent +for weeks (§Historical incident below). Exits non-zero on any critical failure. + +Tunables (env vars): `RAM_WARN`/`RAM_FAIL` (80/95), `DISK_WARN`/`DISK_FAIL` +(80/95), `CELERY_TIMEOUT` (1.0s), `RESTART_WARN` (3 — a container that's +auto-restarted this many times trips a `WARN` as a possible crash-loop), +`COMPOSE_FILE` (default `docker-compose.yml` — override only for local testing, +never on the VM). The Django command also runs standalone inside the container: -1. Copy the unit files and enable the **timer**, not the service: +```bash +docker compose -f docker-compose.yml exec backend python manage.py healthcheck [--json] +``` + +### Scheduled health check (host systemd timer) +Deliberately still a **host** systemd timer, not a Compose `healthcheck:` block or +a Celery beat task — both of those go silent in exactly the "everything is down" +case this exists to catch (a container health check only watches its own +container; beat watching itself is circular). A host-level timer fires even if +every container in `docker-compose.yml`, including `nginx`, is down. + +1. Copy the unit files and enable the **timer**, not the service: ```bash sudo cp deploy/healthcheck.service deploy/healthcheck.timer /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now healthcheck.timer ``` - -2. Verify it's scheduled: - - ```bash - systemctl list-timers healthcheck.timer - ``` - - Expect a `NEXT`/`LEFT` time within the hour, and `LAST`/`PASSED` populated after - the first run. - + `User=ubuntu` in `healthcheck.service` needs `ubuntu` in the `docker` group + (Part 1 §2) — the timer shells out to `docker compose ps`/`exec`, not `sudo`. +2. Verify it's scheduled: `systemctl list-timers healthcheck.timer` — expect a + `NEXT`/`LEFT` time within the hour, `LAST`/`PASSED` populated after the first run. 3. Check the most recent run: - ```bash systemctl status healthcheck.service journalctl -u healthcheck.service -n 50 --no-pager ``` - - A failing run shows `Active: failed` plus the `deploy/healthcheck.sh` report in - the journal — look for `✗`/`FAIL` lines, most importantly any `beat:` - entry (stale or never-run schedule, or missing from the schedule entirely). - -4. To see failed runs across all timers: `systemctl --failed`. - -> **Expect an immediate FAIL on first install.** `scrape-sources-daily` has never run -> on prod (`last_run_at` is NULL), so the never-run rule fires until beat completes -> one. That is the outage this check exists to surface — don't silence it, fix the -> schedule. - -> **No push notification is wired up yet.** There is no `OnFailure=` target, -> email, or Slack hook — `systemctl --failed` and the journal are the only read -> paths. Check them as part of routine ops until that's built. - -## Services - -| Service | What it is | File | Notes | -|---------|-----------|------|-------| -| `gunicorn` | Django backend | `/etc/systemd/system/gunicorn.service` | `unix:/run/gunicorn/gunicorn.sock`, 3 sync workers; `RuntimeDirectory=gunicorn` creates `/run/gunicorn/` | -| `nextjs` | Next.js frontend | `/etc/systemd/system/nextjs.service` | Port 3000, `npm run start` from `theCommonsWeb/` | -| `redis-server` | Celery broker + cache | `/etc/redis/redis.conf` | localhost-bound, password-protected, 512 MB allkeys-lru | -| `celery` | Async task worker | `deploy/celery.service` | `.venv/bin/celery -A backend worker`, concurrency 2, drains Redis DB 0 | -| `celerybeat` | Scheduler | `deploy/celerybeat.service` | `.venv/bin/celery -A backend beat`, DatabaseScheduler; **exactly one** process | -| `broadcast-worker` | Playwright form-filler | `deploy/broadcast-worker.service` | `.venv/bin/celery -A backend worker -Q broadcast -c 1`, MemoryMax 2G, drains the dedicated `broadcast` queue only | -| `scrape-worker` | Ingestion scraper (Playwright) | `deploy/scrape-worker.service` | `.venv/bin/celery -A backend worker -Q scrape -c 1`, MemoryMax 2G, drains the dedicated `scrape` queue only | - -All four `ExecStart=` lines exec `/home/ubuntu/thecommons/backendServer/.venv/bin/celery` -directly rather than `uv run celery` — see §4 for why (the 2026-07-21 outage) and the -one-shot vs long-lived `uv` rule in the Facts table above. - -The three **workers** also pass `-n commons-{default,scrape,broadcast}@%%h`. Without it -they all default to `celery@`, and duplicate nodenames make -`celery -A backend inspect|control` ambiguous — it returns several replies under one -name (`DuplicateNodenameWarning`) and can misroute a revoke. The doubled `%%` is -required: systemd expands a bare `%h` to the *user home directory*, so `%%h` is what -passes a literal `%h` through for celery to expand to the hostname. Verify with: - -```bash -cd /home/ubuntu/thecommons/backendServer && .venv/bin/celery -A backend inspect ping -# expect three distinct nodes: commons-default@, commons-scrape@, commons-broadcast@ -``` - -```bash -sudo systemctl status -sudo systemctl restart -sudo journalctl -u -n 50 # add -f to follow -``` + (This one still goes through `journalctl` — the timer/service pair itself is a + host systemd unit, not a container; only the app stack it inspects moved.) +4. `systemctl --failed` shows failed runs across all timers. + +> **Expect an immediate FAIL on first install** if `scrape-sources-daily` has +> never run (`last_run_at` is NULL) — the never-run rule fires until beat +> completes one cycle. That's the outage this check exists to surface; don't +> silence it, let beat run. + +> **No push notification is wired up yet** — `systemctl --failed` and the journal +> are the only read paths until an `OnFailure=` target/email/Slack hook is built. + +## Local development (`docker-compose.override.yml`) + +Plain `docker compose up --build` (no `-f`) auto-loads +`docker-compose.override.yml` alongside `docker-compose.yml` — Compose's default +two-file merge. It swaps the baked Cloudflare-cert `nginx` config for an +HTTP-only one using `*.localhost` hostnames (no `/etc/hosts` edits needed, per RFC +6761), remaps every bind-mounted host path from `/home/ubuntu/...` to +`./.local-dev/...`, and points `REDIS_URL`/`REDIS_CACHE_URL` at the `redis` +Compose service instead of `localhost` (same cutover fix as Part 1 §6, just +pre-applied for dev). See that file's header comment for the full explanation of +why it exists as a separate file rather than a second tracked nginx config. + +> **Gotcha, discovered during verification:** Compose does **not** auto-recreate a +> container when the *content* of an inline `configs:` entry changes (the local +> `nginx_dev_conf` block in `docker-compose.override.yml` is defined this way). If +> you edit that inline config and run a plain `docker compose up -d`, `nginx` +> silently keeps running with the old config. Force it: +> ```bash +> docker compose up -d --force-recreate nginx +> ``` ## Environment variables @@ -552,7 +723,7 @@ sudo journalctl -u -n 50 # add -f to follow ``` DATABASE_URL= # Neon Postgres connection string DJANGO_SECRET_KEY= -DJANGO_ENV=prod # selects settings/prod.py (omit locally → dev) +DJANGO_ENV=prod # selects settings/prod.py — also set redundantly via compose's `environment:` on every backend/Celery service DJANGO_DEBUG=False DJANGO_ALLOWED_HOSTS=localhost,127.0.0.1,api.thecommons.town CORS_EXTRA_ORIGINS=https://thecommons.town,https://broadcast.thecommons.town @@ -561,7 +732,7 @@ GEMINI_API_KEY= CRON_SECRET= THE_COMMONS_API_KEY= SAFETY_SCORE_THRESHOLD=0.3 # optional -INGEST_SHARD_COUNT=3 # optional — see §3 +INGEST_SHARD_COUNT=3 # optional, deliberate prod setting — see note below INGEST_SCRAPER_HEADLESS=true # optional — default true INGEST_SCRAPER_TIMEOUT_MS=30000 # optional — default 30000 INGEST_SCRAPER_USER_AGENT=Mozilla/5.0 (compatible; TheCommons/1.0) # optional — default shown @@ -571,8 +742,9 @@ BETTER_AUTH_AUDIENCE= BREVO_API_KEY= DIGEST_FROM_EMAIL=digest@thecommons.town SITE_URL=https://thecommons.town -REDIS_URL=redis://:@127.0.0.1:6379/0 # Celery broker + results (DB 0) -REDIS_CACHE_URL=redis://:@127.0.0.1:6379/1 # read-endpoint cache (DB 1) +REDIS_PASSWORD= # container-only knob — must match the password embedded below +REDIS_URL=redis://:@redis:6379/0 # Celery broker + results (DB 0) — `redis`, NOT 127.0.0.1 (Part 1 §6) +REDIS_CACHE_URL=redis://:@redis:6379/1 # read-endpoint cache (DB 1) # Broadcast (see backendServer/.env.example for the full annotated block) BROADCAST_HEADLESS=true BROADCAST_DRY_RUN_DEFAULT=false @@ -580,9 +752,20 @@ BROADCAST_MAX_CONCURRENCY=1 BROADCAST_SCREENSHOT_DIR=/home/ubuntu/broadcast/screenshots BROADCAST_DOWNLOAD_DIR=/home/ubuntu/broadcast/downloads BROADCAST_TIMEOUT_MS=30000 -MEDIA_ROOT=/home/ubuntu/broadcast/media # client-uploaded event images (T7) — outside the repo checkout so `git pull` never touches it +MEDIA_ROOT=/home/ubuntu/broadcast/media # client-uploaded event images — bind-mounted into `backend`/`nginx` at this identical path ``` +**`INGEST_SHARD_COUNT=3` is a deliberate prod setting — keep it at 3.** +`_resolve_env_shard` (`backendServer/ingestion/tasks.py:22-39`) polls only the +sources where `id % INGEST_SHARD_COUNT == day_of_year % INGEST_SHARD_COUNT`, so +with a shard count of 3 any given source is actually polled roughly every **72 +hours**, not the 24h its own `poll_interval_hours` implies — sharding is disabled +(all sources polled daily) only when the var is unset, fails to parse as an int, +or is `<= 1`. Two consequences: a source that fails transiently gets 1/3 as many +chances to recover before the next poll compared to unsharded daily polling, and +monitoring thresholds must be shard-aware — a source that looks "3 days stale" is +expected and healthy under this setting, not a symptom of an outage. + ### `theCommonsWeb/.env.local` ``` @@ -595,6 +778,11 @@ NEXT_PUBLIC_BETTER_AUTH_URL=https://auth.thecommons.town BETTER_AUTH_COOKIE_DOMAIN=.thecommons.town # enables cross-subdomain sessions (SameSite=None; Secure) ``` +Loaded into the running `nextjs` container via `env_file:`, **and** its +`NEXT_PUBLIC_*`/`DATABASE_URL`/`BETTER_AUTH_*` values get baked into the compiled +JS at image build time via the `NEXTJS_BUILD_*`-prefixed build args (Part 2 step +2) — two different mechanisms consuming the same file, at two different times. + ### `broadcastWeb/.env` ``` @@ -603,47 +791,56 @@ VITE_BROADCAST_EXTENSION_ID= # Chrome extension ID for extension autof VITE_BETTER_AUTH_URL=https://auth.thecommons.town ``` -## nginx +`broadcastWeb` has no running container of its own — this file only matters at +**build** time (`broadcast-spa-build`'s `VITE_*` build args, Part 2 step 2). Vite +inlines these into the compiled bundle; there is no runtime env to load them into. -- Config: `/etc/nginx/sites-available/thecommons` (symlinked into `sites-enabled/`) — - one file, multiple `server` blocks. -- Routes: `thecommons.town` → `localhost:3000` (Next.js); `api.thecommons.town` → - `unix:/run/gunicorn/gunicorn.sock` (Django); `www` → 301 to apex; HTTP → 301 to - HTTPS; `api.thecommons.town/static/` → `backendServer/staticfiles/`; - `api.thecommons.town/media/` → `MEDIA_ROOT` (client-uploaded broadcast event - images, T7 — served by nginx directly, **never** by Django/gunicorn in prod); - `broadcast.thecommons.town` → the block from `deploy/nginx-broadcast.conf.snippet`. - - Add a sibling alias to the existing `/static/` block in the `api.thecommons.town` - server block: - - ```nginx - location /media/ { - alias /home/ubuntu/broadcast/media/; - } - ``` +## nginx - `MEDIA_ROOT` (`/home/ubuntu/broadcast/media` by default — see env vars below) - must exist and be writable by the gunicorn user (`ubuntu`) before the first - upload: `mkdir -p /home/ubuntu/broadcast/media`. It lives outside the repo - checkout alongside `BROADCAST_SCREENSHOT_DIR`/`BROADCAST_DOWNLOAD_DIR` so a - `git pull` during deploy never touches it, and it survives deploys for the - same reason. **Uploaded images are kept indefinitely — no pruning job exists.** - `MEDIA_ROOT` grows without bound; at roughly 1–3 MB per event against the VPS - block volume this is negligible for the foreseeable future, but keep it in - mind at ops time (a prune command can be added later without a migration). +- Config: [`deploy/nginx/thecommons.conf`](deploy/nginx/thecommons.conf), baked + into the `nginx` image at `/etc/nginx/conf.d/thecommons.conf` by + [`deploy/nginx/Dockerfile`](deploy/nginx/Dockerfile) — **not** a bind mount in + prod, so an edit requires a rebuild (`docker compose -f docker-compose.yml build + nginx && docker compose -f docker-compose.yml up -d --no-deps nginx`), not + `nginx -t && systemctl reload nginx`. One file, multiple `server` blocks — the + repo's long-standing convention, preserved rather than restructured. +- Routes: `thecommons.town` → `proxy_pass http://backend:8000` was the old + wording; today it's `thecommons.town` → `http://nextjs:3000` (Next.js); + `api.thecommons.town` → `http://backend:8000` (Django/gunicorn, **TCP now, not + a unix socket** — see the correction below); `www` → 301 to apex; HTTP → 301 to + HTTPS; `api.thecommons.town/static/` → the baked `collectstatic` output at + `/usr/share/nginx/html/static` (`COPY --from=` the `backend` image's + `/app/staticfiles_build/static`, matching `STATIC_ROOT` in + `backendServer/backend/settings/base.py:91`); `api.thecommons.town/media/` → + `/var/www/media` (bind-mounted `MEDIA_ROOT`, read-only from nginx's side — + **never** served by Django/gunicorn in prod); `broadcast.thecommons.town` → the + baked SPA at `/usr/share/nginx/html/broadcast` (`COPY --from=` + `broadcast-spa-build`'s `/app/dist`). +- All `proxy_pass` targets resolve through a `resolver 127.0.0.11 …` directive + (Docker's embedded DNS) via an nginx variable rather than a literal hostname — + a literal `proxy_pass http://nextjs:3000` makes nginx resolve once at boot and + **refuse to start** if that service isn't up yet; resolving through a variable + defers the lookup to request time, so one crashed upstream degrades to a 502 on + just that hostname instead of taking the whole ingress down. +- `MEDIA_ROOT` (`/home/ubuntu/broadcast/media`) must exist and be uid-1000-writable + before the first upload — covered in Part 1 §3. **Uploaded images are kept + indefinitely — no pruning job exists**, at roughly 1–3 MB/event against the VPS + block volume; negligible for now, but a prune command could be added later + without a migration. ```bash -sudo nginx -t && sudo systemctl reload nginx +docker compose -f docker-compose.yml build nginx +docker compose -f docker-compose.yml up -d --no-deps nginx ``` ## Firewall -Two layers must allow 80/443: +Unchanged by containerization — the container `nginx` binds the host's 80/443 +exactly like the systemd one did, so both layers still apply: 1. **Oracle VCN Security List** (OCI console) — ingress on 22, 80, 443. -2. **iptables on the VM** — Oracle Ubuntu images ship a catch-all `REJECT` in INPUT. - The 80/443 ACCEPT rules must sit **above** it: +2. **iptables on the VM** — Oracle Ubuntu images ship a catch-all `REJECT` in + INPUT. The 80/443 ACCEPT rules must sit **above** it: ```bash sudo iptables -L INPUT -n --line-numbers sudo iptables -I INPUT 5 -p tcp --dport 443 -m state --state NEW -j ACCEPT @@ -656,18 +853,56 @@ Two layers must allow 80/443: | Symptom | Likely cause | Check | |---------|-------------|-------| | `curl` to IP / subdomain returns nothing | iptables REJECT before ACCEPT | `sudo iptables -L INPUT -n --line-numbers` | -| nginx 502 Bad Gateway | gunicorn or nextjs down | `sudo systemctl status gunicorn nextjs` | -| broadcast subdomain → Cloudflare 526 | origin cert doesn't cover `*.thecommons.town` | reissue cert (Part 1 §5.1) | -| Django `DisallowedHost` | host missing from `ALLOWED_HOSTS` | `DJANGO_ALLOWED_HOSTS` in `.env` | -| 400 on `/events/` from browser | `NEXT_PUBLIC_API_BASE_URL` wrong or stale build | `.env.local`, then `pnpm run build` | -| Django admin has no CSS | `collectstatic` not run / `/static/` alias wrong | `manage.py collectstatic --noinput` | -| Celery worker won't start / no broker | `REDIS_URL` missing/wrong password, or Redis down | `redis-cli -a '' PING`; `journalctl -u celery -n 50` | -| Scheduled job ran twice | leftover OS cron alongside beat | `crontab -l` (Part 1 §9) | +| nginx 502 Bad Gateway | `backend` or `nextjs` container down/unhealthy | `docker compose -f docker-compose.yml ps`; `docker compose -f docker-compose.yml logs backend nextjs` | +| broadcast subdomain → Cloudflare 526 | origin cert doesn't cover `*.thecommons.town`, or the cert bind mount is missing/wrong path | reissue cert (Part 1 §4); confirm `nginx`'s `volumes:` in `docker-compose.yml` | +| Django `DisallowedHost` | `DJANGO_ALLOWED_HOSTS` missing from `backendServer/.env` — `prod.py:16` hard-crashes without it | `docker compose -f docker-compose.yml logs backend`; fix `.env`, `docker compose -f docker-compose.yml up -d --no-deps backend` | +| 400 on `/events/` from browser | `NEXT_PUBLIC_API_BASE_URL` wrong at **build** time — a runtime `.env.local` edit alone does nothing, since Next inlined the old value already | fix `theCommonsWeb/.env.local`, then `docker compose -f docker-compose.yml build nextjs && docker compose -f docker-compose.yml up -d --no-deps nextjs` | +| Django admin has no CSS | `collectstatic` output stale or missing in the `backend` image, or nginx's `/static/` alias path wrong | rebuild `backend` and `nginx`: `docker compose -f docker-compose.yml build backend nginx && docker compose -f docker-compose.yml up -d --no-deps backend nginx` | +| Celery worker won't start / no broker | `REDIS_URL` still points at `127.0.0.1`/`localhost` instead of `redis`, wrong password, or the `redis` container is down | `docker compose -f docker-compose.yml logs redis celery`; re-check Part 1 §6 | +| Scheduled job ran twice | leftover OS cron alongside beat | `crontab -l` (should be empty of `ingest_events`/`send_weekly_digest` lines — Part 1 §9) | +| Edited `docker-compose.override.yml`'s inline nginx config locally, but the container is still serving the old one | Compose doesn't auto-recreate on inline `configs:` content changes | `docker compose up -d --force-recreate nginx` | + +## Historical incident: 2026-07-21 scheduler outage (containers close this gap) + +Kept for context, not as a template to reproduce. All four Celery systemd units +used to exec `/snap/bin/uv run celery …`. Snap-packaged `uv run` spawned its child +process inside a transient `snap.astral-uv.uv-*.scope` under `user@1001.service` — +the *user* session manager, not the unit's own cgroup. With linger off, +`systemd-logind` tore that user slice down the moment the deploying SSH session +ended, and every `uv run`-based unit died within a minute of each deploy, +silently, with `status=0/SUCCESS` — a clean exit that `Restart=on-failure` +correctly declined to restart. That took the entire async stack down for 8 days +starting 2026-07-21 while `gunicorn` and `nextjs` (which never went through snap) +stayed up the whole time. Full forensics: +[`docs/prod-incident-2026-07-21-scheduler-outage.md`](docs/prod-incident-2026-07-21-scheduler-outage.md). + +The mitigation at the time was two-layered: exec the venv binary directly +(`.venv/bin/celery`, removing the snap-scope mechanism entirely) plus +`Restart=always` + `loginctl enable-linger ubuntu` as defense-in-depth. + +**Containers remove this failure mode by construction** — there is no snap, no +`logind`, no per-user systemd slice inside a container's PID namespace for a lost +SSH session to tear down. **Do not cargo-cult the `uv run` avoidance, the linger +setting, or `Restart=always`-as-a-workaround forward into the Compose/Dockerfile +setup** — none of them are solving a live problem there. The one piece that *is* +still worth keeping, for an unrelated reason, is exec'ing binaries directly rather +than through a wrapper shell (`CMD ["gunicorn", …]` / `command: ["celery", …]`, +not a shell script) — so the container's PID 1 is the real process and receives +`SIGTERM` directly on `docker stop`/`down` for a clean shutdown. Same shape, +different justification. + +What still matters from this incident either way: a stale/dead beat schedule is a +`FAIL` in `deploy/healthcheck.sh`, not a `WARN` (§Health check above) — that +distinction is what would have caught this outage in hours instead of weeks, and +it has nothing to do with systemd vs. containers. ## Deep-dive references -- `docs/redis-celery-handoff.md` — Redis/Celery internals, task conventions, beat schedules -- `docs/broadcast.md` — broadcast subsystem: routing, adapters, worker, recipe layer, extension, SPA wiring -- `docs/dev-db-isolation.md` — Neon dev branch setup for local development -- `docs/ingestion-pipeline.md` — scrape → stage → publish flow -- `docs/runbook-auth-cutover.md` — Auth-origin cutover (auth.thecommons.town subdomain, .thecommons.town cookie domain, forced re-login) +- [`docs/adr/0001-containerization.md`](docs/adr/0001-containerization.md) — the + four containerization decisions (nginx-in-a-container, Redis-in-a-container, + Postgres staying external, baked-vs-volumed artifacts) and their rationale +- [`docs/redis-celery-handoff.md`](docs/redis-celery-handoff.md) — Redis/Celery internals, task conventions, beat schedules +- [`docs/broadcast.md`](docs/broadcast.md) — broadcast subsystem: routing, adapters, worker, recipe layer, extension, SPA wiring +- [`docs/dev-db-isolation.md`](docs/dev-db-isolation.md) — Neon dev branch setup for local development +- [`docs/ingestion-pipeline.md`](docs/ingestion-pipeline.md) — scrape → stage → publish flow +- [`docs/runbook-auth-cutover.md`](docs/runbook-auth-cutover.md) — Auth-origin cutover (auth.thecommons.town subdomain, .thecommons.town cookie domain, forced re-login) diff --git a/Dockerfile.frontend b/Dockerfile.frontend new file mode 100644 index 0000000..64734b4 --- /dev/null +++ b/Dockerfile.frontend @@ -0,0 +1,187 @@ +# syntax=docker/dockerfile:1 +# +# Shared frontend Dockerfile for The Commons — builds BOTH frontend apps from +# one file, each in its own isolated stage-tree: +# +# - `commons-runtime` — theCommonsWeb (Next.js 16), a real long-lived Node +# process. Runs `node server.js` on port 3000. +# - `broadcast-build` — broadcastWeb (Vite 7 SPA), BUILD-ONLY. Produces a +# static bundle at /app/dist and stops there — a +# separate nginx image COPYs --from this stage and +# serves the files; this stage itself never runs a +# server and must not be used as a final image. +# +# theCommonsWeb/ and broadcastWeb/ are two SEPARATE pnpm workspaces (own +# pnpm-workspace.yaml + own pnpm-lock.yaml, not one monorepo), so each gets +# its own independent `pnpm install --frozen-lockfile` — nothing is shared +# or hoisted between them. +# +# Build context is the REPO ROOT (both apps' source lives under it): +# docker build -f Dockerfile.frontend --target commons-runtime . +# docker build -f Dockerfile.frontend --target broadcast-build . +# +# Target platform: Oracle Ubuntu 24.04 on ARM64 (prod VM); dev is Apple +# Silicon. node:22-slim is multi-arch, so no --platform pin is needed. + +######################################################################## +# Stage: pnpm-base +# +# Shared base only for pinning the Node/pnpm toolchain — NOT for sharing +# installed dependencies (the two apps' installs stay fully separate below). +# pnpm 11 requires Node >=22.13; node:22-slim tracks current 22.x, so it +# satisfies that floor. +######################################################################## +FROM node:22-slim AS pnpm-base + +# Pin the exact pnpm version CI uses (.github/workflows/ci.yml pins +# pnpm/action-setup@v4 to 11.1.1) via corepack, which ships with Node 22 — +# no separate npm install of pnpm itself (that would violate the +# pnpm-only/no-npm-install rule for a tool that manages pnpm). +RUN corepack enable && corepack prepare pnpm@11.1.1 --activate + +######################################################################## +# Stage: commons-deps +# +# Installs theCommonsWeb's locked dependencies only. Split from the build +# stage so this (slow) layer caches across source-only changes. +######################################################################## +FROM pnpm-base AS commons-deps + +WORKDIR /app + +# Manifests first for layer caching — copying full source here would bust +# the cache on every source edit even when dependencies didn't change. +COPY theCommonsWeb/package.json theCommonsWeb/pnpm-lock.yaml theCommonsWeb/pnpm-workspace.yaml ./ +# --frozen-lockfile: fail rather than silently re-resolve, matching CI. +# pnpm-workspace.yaml's allowBuilds (esbuild, sharp, better-sqlite3, +# @prisma/client, unrs-resolver) still applies here since it's copied in +# above — native postinstall scripts keep working. +RUN pnpm install --frozen-lockfile + +######################################################################## +# Stage: commons-build +# +# Builds the Next.js app to its standalone output. Build-time env vars +# mirror the placeholder set CI uses (.github/workflows/ci.yml +# frontend-commons "Type-check (build)" step) — `next build` prerenders +# routes that import src/lib/db.ts (throws without DATABASE_URL) and +# Better Auth (parses *_BETTER_AUTH_URL as a URL). These are placeholders +# baked only into build-time tracing, never real credentials, and are +# discarded — the running container gets its real config from +# theCommonsWeb/.env.local via compose's env_file at container start, not +# from anything in this image. +######################################################################## +FROM commons-deps AS commons-build + +WORKDIR /app +COPY theCommonsWeb/ ./ + +ARG DATABASE_URL=postgres://build:build@localhost:5432/build +ARG BETTER_AUTH_SECRET=build-only-placeholder-secret +ARG BETTER_AUTH_URL=http://localhost:3000 +ARG NEXT_PUBLIC_BETTER_AUTH_URL=http://localhost:3000 +ARG NEXT_PUBLIC_API_BASE_URL=http://localhost:8000 +ARG NEXT_PUBLIC_THE_COMMONS_API_KEY=build-placeholder +ENV DATABASE_URL=$DATABASE_URL \ + BETTER_AUTH_SECRET=$BETTER_AUTH_SECRET \ + BETTER_AUTH_URL=$BETTER_AUTH_URL \ + NEXT_PUBLIC_BETTER_AUTH_URL=$NEXT_PUBLIC_BETTER_AUTH_URL \ + NEXT_PUBLIC_API_BASE_URL=$NEXT_PUBLIC_API_BASE_URL \ + NEXT_PUBLIC_THE_COMMONS_API_KEY=$NEXT_PUBLIC_THE_COMMONS_API_KEY + +RUN pnpm run build + +######################################################################## +# Stage: commons-runtime (contract name — final Next.js image) +# +# Copies only the standalone trace, not node_modules. Next's documented +# standalone layout (see Next.js docs, "output: standalone"): +# .next/standalone/ -> server.js + a pruned node_modules, becomes /app +# .next/static/ -> NOT included in standalone automatically; must be +# copied to /app/.next/static by hand, or the site +# serves with no CSS/JS chunks (the classic failure +# mode this stage exists to avoid). +# public/ -> also not included automatically; copied to /app/public. +######################################################################## +FROM node:22-slim AS commons-runtime + +WORKDIR /app + +# Non-root runtime user (contract requirement). node:22-slim already ships a +# uid/gid 1000 `node` user — reused here rather than creating a second one +# (a fresh `groupadd --gid 1000` collides with it and fails the build). +# +# Order matters: standalone first (least likely to change), then the two +# directories standalone mode deliberately omits. +COPY --from=commons-build --chown=node:node /app/.next/standalone ./ +COPY --from=commons-build --chown=node:node /app/.next/static ./.next/static +COPY --from=commons-build --chown=node:node /app/public ./public + +USER node + +ENV NODE_ENV=production \ + PORT=3000 \ + HOSTNAME=0.0.0.0 + +EXPOSE 3000 + +# Exec-form CMD so node runs as PID 1 and receives signals directly. +# server.js is the standalone entrypoint next build emits. +CMD ["node", "server.js"] + +######################################################################## +# Stage: broadcast-deps +# +# Installs broadcastWeb's locked dependencies only — entirely separate +# pnpm workspace/lockfile from theCommonsWeb, so this does not reuse or +# extend commons-deps above. +######################################################################## +FROM pnpm-base AS broadcast-deps + +WORKDIR /app + +COPY broadcastWeb/package.json broadcastWeb/pnpm-lock.yaml broadcastWeb/pnpm-workspace.yaml ./ +# broadcastWeb's allowBuilds only permits esbuild (see pnpm-workspace.yaml); +# still copied in above so that allowlist is honored during install. +RUN pnpm install --frozen-lockfile + +######################################################################## +# Stage: broadcast-build (contract name — BUILD-ONLY, no server) +# +# `pnpm run build` = `tsc -b && vite build`, output in dist/. Vite inlines +# VITE_* build args at compile time (see broadcastWeb/.env.example) — these +# become part of the compiled JS, not runtime env, so they must be passed +# as build args here rather than left for container start. +# +# NOTE: prod CI (.github/workflows/ci.yml ~lines 274-281) greps the built +# dist/assets/*.js for a thecommons.town origin as a guard against a +# malformed VITE_BROADCAST_API_BASE_URL that "builds fine" but silently +# misroutes every API call. That check lives in CI/deploy, not here — this +# stage just needs correct build args passed in to produce a correct bundle. +# +# This stage is intentionally the last thing in this file: it has no CMD/ +# EXPOSE and must never be run as a standalone container — an nginx image +# does `COPY --from=broadcast-build /app/dist ...` to consume it. +######################################################################## +FROM broadcast-deps AS broadcast-build + +WORKDIR /app +COPY broadcastWeb/ ./ + +ARG VITE_BROADCAST_API_BASE_URL=http://localhost:8000 +ARG VITE_BETTER_AUTH_URL=http://localhost:3000 +ARG VITE_BROADCAST_EXTENSION_ID= +ENV VITE_BROADCAST_API_BASE_URL=$VITE_BROADCAST_API_BASE_URL \ + VITE_BETTER_AUTH_URL=$VITE_BETTER_AUTH_URL \ + VITE_BROADCAST_EXTENSION_ID=$VITE_BROADCAST_EXTENSION_ID + +# tsc -b writes tsconfig.tsbuildinfo and vite build writes dist/ — both as +# whatever user runs this stage. Non-root is only "where practical" per the +# contract; this stage never ships as a runtime container (nothing to run +# as root or otherwise), so it's left at the default build-time user, and +# the only thing that matters is that /app/dist exists for the nginx image +# to copy out. +RUN pnpm run build + +# Deliberately no CMD/EXPOSE/USER-drop here — this stage is a build +# artifact source, not a runnable image. diff --git a/backendServer/.dockerignore b/backendServer/.dockerignore new file mode 100644 index 0000000..3679801 --- /dev/null +++ b/backendServer/.dockerignore @@ -0,0 +1,39 @@ +# Build context is backendServer/ itself (docker build -f backendServer/Dockerfile backendServer). +# Keep this list in sync with .gitignore where the two overlap. + +# Local virtualenv — the image builds its own venv via `uv sync`, never +# reuses a host one (different OS/arch: dev is Apple Silicon, prod is +# Oracle ARM64, but a stray macOS-built .venv would be architecture-wrong +# either way). +.venv/ + +# Python bytecode / caches +__pycache__/ +**/__pycache__/ +*.pyc +*.pyo +.mypy_cache/ +.ruff_cache/ +.pytest_cache/ + +# Env files — config comes from compose's env_file at runtime, never baked +# into the image. `.env*` also catches .env.local, .env.prod, etc. +.env +.env.* +!.env.example + +# Generated/runtime data — collectstatic output is produced fresh at build +# time; media and broadcast artifacts are runtime state that lives outside +# the image (volumes in compose). +staticfiles_build/ +media/ +broadcast_artifacts/ + +# Editor/OS cruft +.DS_Store +.idea/ +.vscode/ + +# VCS (not present under backendServer/, but harmless to exclude defensively) +.git/ +.gitignore diff --git a/backendServer/AGENTS.md b/backendServer/AGENTS.md index f339514..7a48431 100644 --- a/backendServer/AGENTS.md +++ b/backendServer/AGENTS.md @@ -1,28 +1,45 @@ # backendServer — Agent Map -Django 6 + DRF backend. Python 3.13, managed by `uv`. Four apps: `events` (public API + digests), `ingestion` (LLM pipeline), `broadcast` (event syndication), `backend` (config + auth bridge + Celery). Database is Postgres on Neon. Async on Redis + Celery; broadcast dispatch is on-demand Celery too, routed to its own single-concurrency `broadcast` queue (mirrors the `scrape` queue) rather than a polling worker. See [`../ARCHITECTURE.md`](../ARCHITECTURE.md) for cross-cutting detail. +Django 6 + DRF backend. Python 3.13, managed by `uv`. Six apps: `accounts` (identity/auth-bridge), `events` (public event API), `newsletter` (subscriptions + digest engine), `ingestion` (LLM pipeline), `broadcast` (event syndication), `backend` (config + Celery, no domain logic). `devtools` is a seventh, dev/test-only app (see below). Database is Postgres on Neon. Async on Redis + Celery; broadcast dispatch is on-demand Celery too, routed to its own single-concurrency `broadcast` queue (mirrors the `scrape` queue) rather than a polling worker. See [`../ARCHITECTURE.md`](../ARCHITECTURE.md) for cross-cutting detail. + +**Isolation contract:** no app imports from `ingestion`/`broadcast`. `accounts`, `newsletter`, and `events` may read each other where the domain genuinely overlaps — e.g. `accounts.me` writes a `NewsletterSubscriber` row (email-preference sync), and `newsletter._build_recipients` reads `accounts.UserProfile` (tag-filtered digests). Both directions are deliberate. Enforced by `accounts/tests/test_isolation_fast.py` and `newsletter/tests/test_isolation_fast.py`. ## Directory Map ``` backendServer/ ├── manage.py -├── backend/ # Project config +├── backend/ # Project config — no domain logic, include()-only urlconf │ ├── settings/ # base / dev / prod / test -│ ├── urls.py # Root URLconf (cron, publish, auth/me, businesses, admin) +│ ├── urls.py # Root URLconf — include()s accounts/events/newsletter/ingestion/broadcast.urls +│ │ # (+ admin, + DEBUG-gated devtools.urls); no `from .views import ...` │ ├── celery.py # Celery app factory + autodiscover │ ├── jwt_auth.py # BearerTokenAuthentication — Better Auth JWKS (TTL + stale-grace) -│ ├── permissions.py # DRF auth/permission classes (JWT, API key) +│ ├── permissions.py # DRF auth/permission classes (JWT, API key) — imports BetterAuthUser from accounts.models │ └── test_runner.py # NeonAuthTestRunner — builds neon_auth schema for tests -├── events/ # Public app -│ ├── models.py # Event/Town/Tag/Category/UserProfile/BusinessProfile/Newsletter -│ │ # + 5 BetterAuth* mirrors (managed=False) -│ ├── views.py / serializers.py / urls.py +├── accounts/ # Identity/auth-bridge app +│ ├── models.py # 5 BetterAuth* mirrors (managed=False, neon_auth.*) + UserProfile/BusinessProfile +│ │ # (both OneToOne→BetterAuthUser; a "business" is a kind of user profile) +│ ├── views.py / serializers.py / urls.py # me (/auth/me), businesses, my_business, business_detail (/businesses...) +│ ├── permissions.py # Isolation-contract docstring (no permission classes yet) +│ └── tests/test_isolation_fast.py +├── events/ # Public event app (slimmed) +│ ├── models.py # Tag, Town, Category, Event +│ ├── views.py / serializers.py / urls.py # get_all, get_one, get_towns, get_categories, create_event, +│ │ # manage_staged_event, get_my_events, get_my_profile │ ├── cache.py # Version-keyed Redis cache for hot read endpoints │ ├── signals.py # Cache invalidation on Event/Town/Category writes -│ ├── tasks.py # Celery: ping, send_one_digest, fan_out_weekly_digest, fan_out_monthly_digest -│ ├── email_service.py # Brevo transactional email + digest builder -│ └── management/commands/ # devserver, seed_dev, healthcheck, delete_user, send_*digest +│ ├── tasks.py # Celery: ping (digest tasks moved to newsletter/tasks.py) +│ ├── email_service.py # Generic Brevo transport (send_email) — used by non-digest commands +│ └── management/commands/ # devserver, seed_dev, healthcheck, delete_user +├── newsletter/ # Subscriptions + digest engine +│ ├── models.py # NewsletterSubscriber +│ ├── views.py / urls.py # subscribe (/newsletter/subscribe), newsletter_manage (/newsletter/manage) +│ ├── email_service.py # _build_recipients, send_digest, digest_window, manage_url_for, send_newsletter_welcome +│ ├── tasks.py # Celery: send_one_digest, fan_out_weekly_digest, fan_out_monthly_digest +│ ├── templates/email/ # Digest + welcome email templates +│ ├── management/commands/ # send_digest, send_weekly_digest, send_test_digest +│ └── tests/test_isolation_fast.py ├── ingestion/ # Pipeline app │ ├── models.py # EventSource, RawEvent, StagedEvent │ ├── importers/ics_importer.py # ICS feed → RawEvent (shardable) @@ -31,7 +48,8 @@ backendServer/ │ ├── safety_scorer.py # Gemini content-safety scoring │ ├── services.py # publish_all_approved, auto_publish_safe_events │ ├── tasks.py # Celery: run_ingestion_pipeline, publish_all_approved_task -│ ├── views.py # cron_ingest, publish, admin doc pages +│ ├── views.py / urls.py # cron_ingest, publish_approved_events, direct_submit, pipeline/admin doc pages +│ │ # (own urlconf: admin/docs/*, api/cron/ingest, api/events/publish-approved, direct-submit) │ └── management/commands/ # ingest_events, cleanup_old_events ├── broadcast/ # Event syndication (see ../docs/broadcast.md) │ ├── models.py # BroadcastSubmission, BroadcastTarget, BroadcastAccess, AccessCode, AccessCodeUse @@ -44,13 +62,25 @@ backendServer/ │ └── management/commands/ # run_broadcast_worker (--once debug helper), broadcast_dry_run, capture_broadcast_form, │ # check_recipes, scaffold_adapter, set_broadcast_access, │ # generate_access_code, list_access_codes, revoke_access_code -├── templates/ # admin docs pages (docs/) + email digests (email/) +├── devtools/ # Dev-only app (INSTALLED_APPS only under dev/test settings; DEBUG-gated in urls) +│ ├── views/ # Package: playground.py, probe.py, monitor.py, _shared.py, __init__.py (re-exports) +│ └── urls.py +├── templates/ # admin docs pages (docs/) — email digest templates now live in newsletter/templates/email/ └── pyproject.toml / uv.lock ``` ## API Endpoints -Auth: `—` public · `user` Better Auth JWT · `key` `THE_COMMONS_API_KEY` · `tier≥N` broadcast tier (Bearer JWT or `X-Broadcast-Access-Code`, resolved by `broadcast/access.py`). `APPEND_SLASH=False` — slashes are exact. No global DRF config; auth/permissions are per-view. +Auth: `—` public · `user` Better Auth JWT · `key` `THE_COMMONS_API_KEY` · `tier≥N` broadcast tier (Bearer JWT or `X-Broadcast-Access-Code`, resolved by `broadcast/access.py`). `APPEND_SLASH=False` — slashes are exact. No global DRF config; auth/permissions are per-view. `backend/urls.py` is `include()`-only — every route below is owned by the app it's grouped under. + +### accounts (`accounts/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| +| GET/PATCH | `/auth/me` | user | Read / update own profile | +| GET/POST | `/businesses` · `/businesses/me` · `/businesses/` | user | Business listing CRUD | + +### events (`events/urls.py`) | Method | Path | Auth | Purpose | |--------|------|------|---------| @@ -60,23 +90,44 @@ Auth: `—` public · `user` Better Auth JWT · `key` `THE_COMMONS_API_KEY` · ` | GET/PATCH/DELETE | `/events/staged/` | user | Manage own staged submission | | GET/DELETE | `/events/` | user (delete) | Event detail / owner delete | | POST | `/events/create` | user or key | Submit event → StagedEvent | -| GET/PATCH | `/auth/me` | user | Read / update profile | + +### newsletter (`newsletter/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| | POST | `/newsletter/subscribe` | — | Newsletter signup (welcome email + manage link) | | GET/PATCH | `/newsletter/manage` | — (token) | View / change a subscription via `?token=` | -| GET/POST | `/businesses` · `/businesses/me` · `/businesses/` | user | Business listing CRUD | + +### ingestion (`ingestion/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| | GET | `/api/cron/ingest` | CRON_SECRET | Queue ingestion pipeline | | POST | `/api/events/publish-approved` | key | Queue bulk publish | +| POST | `/api/events/direct-submit` | JWT optional | Direct host event submission (broadcast SPA) | +| GET/POST | `/admin/docs/pipeline-docs/` · `/admin/docs/admin-docs/` · `/admin/docs/publish-approved/` | staff | Pipeline/admin docs pages + publish-approved button | + +### broadcast (`broadcast/urls.py`) + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| | GET | `/broadcast/access` | — | Caller's tier + trial metadata (403 for invalid creds) | | POST | `/broadcast/preview` · `/submit` | tier≥1 | Preview eligible sites / enqueue submission | | POST | `/broadcast/ai-autofill` | tier≥2 | LLM field extraction from free text | | POST | `/broadcast/direct-recipe` | tier≥1 | Recipe JSON for a site (no job) | | GET/POST | `/broadcast/jobs/[/retry\|/submit-real\|/cancel]` | tier≥1 | Job status + lifecycle ops | | GET | `/broadcast/jobs//screenshots/` · `/manual/` | tier≥1 | Screenshot / manual-review recipe | -| GET/POST | `/admin/docs/...` · `/admin/` | staff | Docs pages + django-unfold admin | + +### admin + +| Method | Path | Auth | Purpose | +|--------|------|------|---------| +| GET/POST | `/admin/` | staff | django-unfold admin | ## Management Commands -- **events:** `devserver` (auto-port runserver), `seed_dev`, `healthcheck [--json]`, `delete_user --email`, `send_digest`, `send_test_digest --email`, `send_weekly_digest`. +- **events:** `devserver` (auto-port runserver), `seed_dev`, `healthcheck [--json]`, `delete_user --email`. +- **newsletter:** `send_digest`, `send_weekly_digest`, `send_test_digest --email`. - **ingestion:** `ingest_events` (full pipeline; `--skip-*`, `--shard N/M`), `cleanup_old_events`. - **broadcast:** `run_broadcast_worker [--once]` (debug helper — drains one submission without Celery, not a service entrypoint), `broadcast_dry_run --site --fixture`, `capture_broadcast_form `, `check_recipes [--live]`, `scaffold_adapter --url --key`, `set_broadcast_access <0|1|2>`, `generate_access_code [--tier] [--label] [--expires] [--uses|--unlimited]`, `list_access_codes`, `revoke_access_code `. @@ -111,8 +162,11 @@ DJANGO_SETTINGS_MODULE=backend.settings.test uv run python manage.py test --tag= ## Devtools -`devtools/` (dev-only, `DEBUG`-gated — 404s in prod) hosts the ingestion playground and -the `/devtools/monitor` funnel dashboard + dry-run probe. See +`devtools/` (registered in `INSTALLED_APPS` only under dev/test settings; still `DEBUG`-gated +in urls — 404s in prod either way) hosts the ingestion playground and the `/devtools/monitor` +funnel dashboard + dry-run probe. `devtools/views.py` is a package (`devtools/views/`) split +by concern: `playground.py`, `probe.py`, `monitor.py`, `_shared.py`, with `__init__.py` +re-exporting the view functions `devtools/urls.py` routes to. See [`../docs/ingestion-monitoring.md`](../docs/ingestion-monitoring.md) for funnel/health semantics, `SourceRun` statuses, the probe's SSE contract, and the prod read-only setup (`PROD_DATABASE_URL`). diff --git a/backendServer/Dockerfile b/backendServer/Dockerfile new file mode 100644 index 0000000..bb337d6 --- /dev/null +++ b/backendServer/Dockerfile @@ -0,0 +1,187 @@ +# syntax=docker/dockerfile:1 +# +# Multi-stage build producing two images from one Django source tree + venv: +# - `app` — gunicorn / celery worker (default queue) / celerybeat. +# No Chromium. +# - `playwright` — `FROM app` + bundled Chromium, for the broadcast and +# scrape Celery workers (both drive headless Chromium). +# +# Build context is backendServer/ (not the repo root): +# docker build -f backendServer/Dockerfile --target app backendServer +# docker build -f backendServer/Dockerfile --target playwright backendServer +# +# Target platform: Oracle Ubuntu 24.04 on ARM64 (prod VM). python:3.13-slim +# is multi-arch, so no --platform pin is needed even though local dev is +# Apple Silicon. + +######################################################################## +# Stage: builder +# +# Resolves the locked dependency set (uv.lock) into a venv at /app/.venv +# and installs the project into it. Kept separate from `app` so build-only +# tooling (the uv binary, compilers, -dev headers) never ships at runtime. +######################################################################## +FROM python:3.13-slim AS builder + +# Recommended way to get `uv` into a Dockerfile per Astral's docs — copies +# the static binary from their distroless image rather than curl | sh. +COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /usr/local/bin/ + +WORKDIR /app + +# No compilers/headers here on purpose. Every dependency in pyproject.toml +# that would otherwise need to build from sdist — psycopg[binary], lxml, +# Pillow — publishes prebuilt manylinux aarch64 wheels, so `uv sync` never +# invokes a compiler on this platform. Installing build-essential + libpq-dev +# "just in case" cost ~250 MB and made this the slowest, most fragile layer +# in the build (it was also the layer that reliably tripped apt Hash Sum +# mismatches behind a proxy). If a future dependency genuinely lacks an +# aarch64 wheel, `uv sync` below will fail loudly with a compiler error — +# that's the signal to add the toolchain back, deliberately and with a note. + +# Copy only the dependency manifests first so this (slow) layer caches +# across source-only changes. +ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy +COPY pyproject.toml uv.lock ./ +# --frozen: fail the build if uv.lock is stale rather than silently +# re-resolving. --no-install-project: dependencies only — project source +# isn't copied in yet, so there's nothing to install as a package. +RUN uv sync --frozen --no-install-project --no-dev + +# Now bring in the full source (respecting .dockerignore: no .venv, no +# .env, no caches) and install the project itself into the same venv. +COPY . . +RUN uv sync --frozen --no-dev + +######################################################################## +# Stage: app +# +# Runtime image for gunicorn, the default celery worker, and celerybeat. +# venv + source only — no compilers, no Chromium. +######################################################################## +FROM python:3.13-slim AS app + +WORKDIR /app + +# apt hardening, applied to every apt invocation in this file via +# /etc/apt/apt.conf.d/. Retries covers transient mirror failures; disabling +# HTTP pipelining is the standard mitigation for "Hash Sum mismatch" errors, +# which is how a proxy that mangles large or pipelined responses reports +# itself. Docker Desktop ships such a proxy on by default +# (http.docker.internal:3128), and without this the Playwright stage's +# `--with-deps` — a large multi-package fetch — fails intermittently on a +# different package each run, which reads like package corruption rather +# than a network problem. +RUN printf 'Acquire::Retries "3";\nAcquire::http::Pipeline-Depth "0";\n' \ + > /etc/apt/apt.conf.d/99-robust + +# libpq5 is the runtime (not -dev) Postgres client lib psycopg needs. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libpq5 \ + && rm -rf /var/lib/apt/lists/* + +# Non-root runtime user (contract requirement). --home-dir /app rather than +# a separate /home/app: WORKDIR is already /app, and it makes $HOME sane for +# anything (e.g. playwright, in the child stage) that defaults to +# $HOME/.cache. +RUN groupadd --gid 1000 app \ + && useradd --uid 1000 --gid app --home-dir /app --no-create-home \ + --shell /usr/sbin/nologin app + +# Pull in the venv + installed project from the builder stage, owned by the +# runtime user up front so later steps (collectstatic, playwright's browser +# download in the child stage) don't need a root-owned-file cleanup pass. +COPY --from=builder --chown=app:app /app /app + +# `WORKDIR /app` above created /app itself as root, and `COPY --chown` only +# sets ownership on the entries it copies — never on a pre-existing +# destination directory. So /app stays root-owned unless we say otherwise, +# and collectstatic (below, running as `app`) then fails with EACCES trying +# to create STATIC_ROOT at /app/staticfiles_build/. +RUN chown app:app /app + +ENV PATH="/app/.venv/bin:$PATH" \ + PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 \ + HOME=/app + +USER app + +# collectstatic needs Django settings to import cleanly, but prod.py hard- +# crashes at import time without DJANGO_SECRET_KEY / DJANGO_ALLOWED_HOSTS / +# DATABASE_URL (deliberately — see prod.py), and this image never bakes in +# a real .env (config comes from compose's env_file at container start). +# So this runs under dev settings (DJANGO_ENV unset -> dev.py) with a +# throwaway DATABASE_URL just to satisfy dev.py's urlparse() call at import; +# collectstatic itself never touches the database. STATIC_ROOT +# (backend/settings/base.py) is identical across dev/prod, so the output +# lands in the right place regardless of which settings module produced it. +RUN DATABASE_URL=postgresql://collectstatic:collectstatic@localhost:5432/collectstatic \ + python manage.py collectstatic --noinput + +EXPOSE 8000 + +# Default command: gunicorn bound to TCP 0.0.0.0:8000 (replaces the old +# nginx-unix-socket setup at /run/gunicorn/gunicorn.sock), 3 sync workers, +# backend.wsgi. Exec-form CMD so gunicorn runs as PID 1 and receives signals +# directly. +# +# Compose overrides this CMD for the other two services that reuse this +# image (same command the systemd units ran, minus the systemd-specific +# bits — see note below): +# celery worker (default queue): celery -A backend worker -n commons-default@%h -l info --concurrency=2 +# celery beat: celery -A backend beat -l info +# +# Note on `.venv/bin/celery` vs `uv run celery`: the old systemd units exec +# the venv binary directly instead of going through `uv run`, to dodge a +# snap-packaged-uv bug where the child process landed in a systemd user +# slice that a logout could tear down (an 8-day silent prod outage on +# 2026-07-21 — see deploy/celery.service and +# docs/prod-incident-2026-07-21-scheduler-outage.md). That mechanism can't +# happen in a container — no snap, no logind, no user slice — so it's +# obsolete here. Still worth exec'ing the venv binary directly (it's on +# PATH via the ENV above) rather than `uv run`: it's the same correct +# behavior for PID-1 signal handling, just without the systemd-specific +# justification. +CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "3", "backend.wsgi:application"] + +######################################################################## +# Stage: playwright +# +# `FROM app` + bundled Chromium + its system libs, for the two Playwright +# Celery workers: +# broadcast worker: celery -A backend worker -Q broadcast -n commons-broadcast@%h -c 1 -l info +# scrape worker: celery -A backend worker -Q scrape -n commons-scrape@%h -c 1 -l info +# (-c 1 is mandatory for the broadcast worker — recover_orphans() assumes a +# single worker — and the scrape worker follows the same pattern; compose +# sets these commands, not this Dockerfile.) +######################################################################## +FROM app AS playwright + +USER root + +# uv is only needed transiently to invoke `playwright install` against the +# exact pinned version in uv.lock; not required at runtime, so it's added +# here rather than in `app`. +COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /usr/local/bin/ + +# Explicit browsers path rather than relying on the default +# ~/.cache/ms-playwright: guarantees the browser lands somewhere the +# non-root `app` user can read (and, since we install as root below, we +# chown it explicitly afterward rather than relying on $HOME semantics). +ENV PLAYWRIGHT_BROWSERS_PATH=/app/.playwright-browsers + +# `uv run playwright install --with-deps chromium`: +# - `uv run` resolves the playwright *package* version already pinned in +# uv.lock, so the browser build downloaded here matches it exactly. +# - `--with-deps` also apt-installs the system libraries Chromium needs +# (requires root, hence USER root above). +# - Never the branded "chrome" channel — unsupported on arm64, which is +# the actual prod target. +# Chromium is baked into the image here (build time), not downloaded at +# container start. +RUN uv run playwright install --with-deps chromium \ + && rm -rf /var/lib/apt/lists/* \ + && chown -R app:app "$PLAYWRIGHT_BROWSERS_PATH" + +USER app diff --git a/backendServer/accounts/__init__.py b/backendServer/accounts/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/accounts/admin.py b/backendServer/accounts/admin.py new file mode 100644 index 0000000..acc13b5 --- /dev/null +++ b/backendServer/accounts/admin.py @@ -0,0 +1,25 @@ +from django.contrib import admin +from unfold.admin import ModelAdmin + +from .models import BusinessProfile, UserProfile + + +@admin.register(UserProfile) +class UserProfileAdmin(ModelAdmin): + # `user` points to a managed=False model in neon_auth — no FK constraint + # and no + + + + + + diff --git a/backendServer/ingestion/tests/fixtures/patchdurham.html b/backendServer/ingestion/tests/fixtures/patchdurham.html new file mode 100644 index 0000000..ec322f6 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/patchdurham.html @@ -0,0 +1,12 @@ + + +Durham, NC Events Calendar | Durham, NC Patch + +
+

Durham Events Calendar

+

Trimmed real sample of patch.com/north-carolina/durham-nc/calendar. + The visible DOM is elided; the scraper reads __NEXT_DATA__ below.

+
+ + + diff --git a/backendServer/ingestion/tests/fixtures/patchmorrisville.html b/backendServer/ingestion/tests/fixtures/patchmorrisville.html new file mode 100644 index 0000000..ae5ecb1 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/patchmorrisville.html @@ -0,0 +1,12 @@ + + +Morrisville, NC Events Calendar | Morrisville, NC Patch + +
+

Morrisville Events Calendar

+

Trimmed real sample of patch.com/north-carolina/morrisville-nc/calendar. + The visible DOM is elided; the scraper reads __NEXT_DATA__ below.

+
+ + + diff --git a/backendServer/ingestion/tests/fixtures/patchpittsboro.html b/backendServer/ingestion/tests/fixtures/patchpittsboro.html new file mode 100644 index 0000000..bb60df6 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/patchpittsboro.html @@ -0,0 +1,12 @@ + + +Pittsboro, NC Events Calendar | Pittsboro, NC Patch + +
+

Pittsboro Events Calendar

+

Trimmed real sample of patch.com/north-carolina/pittsboro-nc/calendar. + The visible DOM is elided; the scraper reads __NEXT_DATA__ below.

+
+ + + diff --git a/backendServer/ingestion/tests/fixtures/patchraleigh.html b/backendServer/ingestion/tests/fixtures/patchraleigh.html new file mode 100644 index 0000000..46fdf4e --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/patchraleigh.html @@ -0,0 +1,12 @@ + + +Raleigh, NC Events Calendar | Raleigh, NC Patch + +
+

Raleigh Events Calendar

+

Trimmed real sample of patch.com/north-carolina/raleigh/calendar. + The visible DOM is elided; the scraper reads __NEXT_DATA__ below.

+
+ + + diff --git a/backendServer/ingestion/tests/fixtures/raleighnc.html b/backendServer/ingestion/tests/fixtures/raleighnc.html new file mode 100644 index 0000000..893413e --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/raleighnc.html @@ -0,0 +1,196 @@ + + + + +
+

Events

+
+
+

Upcoming Events

+
+ +
+

2 + +Sunday + +August 2026 + +

+ +
+ +
+

4 + +Tuesday + +August 2026 + +

+ +
+ +
+

7 + +Friday + +August 2026 + +

+ +
+ +
+

1 + +Wednesday + +January 2026 + +

+ +
+
+ +
+
+

Ongoing Events

+
+
+ +
+
+
+ + diff --git a/backendServer/ingestion/tests/fixtures/thrivinginraleigh.html b/backendServer/ingestion/tests/fixtures/thrivinginraleigh.html new file mode 100644 index 0000000..0e34422 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/thrivinginraleigh.html @@ -0,0 +1,177 @@ + + + + +
+
+ +
+
+
+
+
Aug
+
22
+
+
+
+
+

Lazy Daze Arts & Crafts Festival

+
    +
  • + +
    • +
  • + +
    • +
  • + Cary Town Hall Campus + (map) +
    • +
  • + Google Calendar + ICS +
    • +
+
+

Join us for the 49th annual Lazy Daze Arts & Crafts Festival showcasing over 250 exceptional artists from across the country! Enjoy live entertainment on four stages, savor delicious offerings from 25+ food vendors.

+
+ View Event → +
+
+ +
+
+
+
+
Aug
+
15
+
+
+
+
+

12th Annual CaribMask Carnival

+
    +
  • + +
    • +
  • + + + + + +
    • +
  • + Downtown Raleigh + (map) +
    • +
  • + Google Calendar + ICS +
    • +
+
+

In 2012, RDACA (Raleigh-Durham AfroCaribbean Association) came together to promote diversity, equality and create a deeper appreciation for Afro-Caribbean culture among the citizens of the Raleigh/Durham Metropolitan area.

+
+ View Event → +
+
+ +
+
+
+
+
Aug
+
13
+
+
+
+
+

Live After 5

+
    +
  • + +
    • +
  • + + + + + +
    • +
  • + Moore Square + (map) +
    • +
  • + Google Calendar + ICS +
    • +
+
+

DRA and Southern Bank presents Live After 5 returns this summer on a new night, at a new time, in three new locations across Downtown Raleigh for an extended six-date run.

+
+ View Event → +
+
+ +
+ +
+ +
+
+
+
+
Jun
+
17
+
+
+
+
+

Pottery Painting

+
    +
  • + +
    • +
  • + + + + + +
    • +
  • + Cozy Panda + (map) +
    • +
  • + Google Calendar + ICS +
    • +
+
+

Cozy Panda is on the move! Pick your ceramic, paint your masterpiece and take it home the same day.

+
+ View Event → +
+
+ +
+
+ + diff --git a/backendServer/ingestion/tests/fixtures/triangleonthecheap.html b/backendServer/ingestion/tests/fixtures/triangleonthecheap.html new file mode 100644 index 0000000..290883b --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/triangleonthecheap.html @@ -0,0 +1,63 @@ + + + + +
+ +

Today: Friday, July 31, 2026

+

Baskin Robbins: 31% off scoops

+

All Day | Discounted | Baskin-Robbins, multiple locations

+ +
+

Tasting at Ten -- free coffee tasting at Counter Culture Coffee

+

10:00 am | FREE | Counter Culture Coffee Headquarters and Training Center, Durham

+ +
+

Last Fridays Makers Market

+

5:30 pm to 8:30 pm | FREE | Old Courthouse–Hillsborough

+ +
+

Hillsborough Arts Council's Live on the Lawn

+

5:30 pm to 8:30 pm | FREE

+ +
+

Last Fridays Art Walk in Hillsborough

+

5:30 pm to 9:00 pm | FREE | Downtown Hillsborough

+ +
+
+

Tomorrow: Saturday, August 1, 2026

+

Museums on Us: Free admission to Marbles Kids Museum first weekend of month for Bank of America customers and employees

+

All Day | FREE | Marbles Kids Museum, Raleigh

+ +
+

National Pinball Day! Use the Pinball Map to find local pinball machines

+

All Day | Various locations

+ +
+
+

Thursday, January 1, 2026

+

Past Event (back-dated for filter test)

+

All Day | Discounted | Baskin-Robbins, multiple locations

+ +
+
+ +
+ + diff --git a/backendServer/ingestion/tests/fixtures/visitraleigh.html b/backendServer/ingestion/tests/fixtures/visitraleigh.html new file mode 100644 index 0000000..613f7e2 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/visitraleigh.html @@ -0,0 +1,119 @@ + +
+
+ +
+
+
+
+
+
+

The Rocky Horror Picture Show

+
+
+ +
+
+
+
+
+
+ +
+
+
+
+
+
+

Andrew Orolfo

+
+
+
    +
  • Dates vary between July 31, 2026 - August 1, 2026
    • +
  • Times: Fri., 9:15pm; Sat., 9pm
    • +
  • + Venue: + + Raleigh Improv + +
    • +
+
+
+
+
+
+
+ +
+
+
+
+
+
+

Beer, Bourbon & BBQ Festival 2026

+
+
+ +
+
+
+
+
+
+ + +
+
+
+
+
+
+

Past Event (back-dated for filter test)

+
+
+ +
+
+
+
+
+
+ +
+
diff --git a/backendServer/ingestion/tests/fixtures/visitraleigh_cary.html b/backendServer/ingestion/tests/fixtures/visitraleigh_cary.html new file mode 100644 index 0000000..a0b51a9 --- /dev/null +++ b/backendServer/ingestion/tests/fixtures/visitraleigh_cary.html @@ -0,0 +1,124 @@ + +
+
+ +
+
+
+
+
+
+

Comedy Show: Cyrus Steele + Joe Perrow

+
+
+ +
+
+
+
+
+
+ +
+
+
+
+
+
+

Country Grass

+
+
+ +
+
+
+
+
+
+ + +
+
+
+
+
+
+

Clue (High School Edition)

+
+
+
    +
  • Recurring weekly on Sunday, Friday, Saturday until August 9, 2026
    • +
  • Times: Fri.-Sat., 7:30pm; Sun., 3pm
    • +
  • + Venue: + + Raleigh Little Theatre + +
    • +
+
+
+
+
+
+
+ + +
+
+
+
+
+
+

Past Event (back-dated for filter test)

+
+
+ +
+
+
+
+
+
+ +
+
diff --git a/backendServer/ingestion/tests/test_carychamber_extract_fast.py b/backendServer/ingestion/tests/test_carychamber_extract_fast.py new file mode 100644 index 0000000..29127a7 --- /dev/null +++ b/backendServer/ingestion/tests/test_carychamber_extract_fast.py @@ -0,0 +1,75 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.carychamber import CarychamberScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "carychamber.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31, 12, 0, 0)) + + +@tag("fast") +class CarychamberExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.carychamber.timezone.now") + def test_extracts_future_events_and_drops_past(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarychamberScraper().extract(self.html) + + # Fixture carries 5 events: 4 future, 1 back-dated to 2020 that the + # past-event filter must drop. + self.assertEqual(len(events), 4) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.carychamber.timezone.now") + def test_start_and_end_built_from_date_plus_free_text_time(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarychamberScraper().extract(self.html) + + ambassador = next(e for e in events if e.title == "Ambassador Information Session") + # "12:00 Noon" is the site's own non-standard time string for noon. + self.assertEqual(ambassador.start.hour, 12) + self.assertEqual(ambassador.start.minute, 0) + self.assertEqual(ambassador.start.day, 5) + self.assertIsNotNone(ambassador.start.tzinfo) + self.assertEqual(ambassador.end.hour, 13) + self.assertEqual(ambassador.source_uid, "6241") + self.assertEqual( + ambassador.source_url, + "https://web.carychamber.com/events/eventdetail.aspx?eventid=6241", + ) + + @mock.patch("ingestion.scraping.scrapers.carychamber.timezone.now") + def test_description_is_html_unescaped_and_stripped(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarychamberScraper().extract(self.html) + + golf = next(e for e in events if e.title == "Education Golf Tournament") + self.assertNotIn("

", golf.description) + self.assertNotIn(" ", golf.description) + self.assertIn("beautiful day on the course", golf.description) + + @mock.patch("ingestion.scraping.scrapers.carychamber.timezone.now") + def test_titles_are_html_unescaped(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarychamberScraper().extract(self.html) + + golf_academy = next(e for e in events if "Golf Academy" in e.title) + self.assertEqual(golf_academy.title, "Ladies Lunch & Learn Golf Academy") + + def test_no_events_yields_empty_list(self): + self.assertEqual(CarychamberScraper().extract(''), []) + + def test_malformed_xml_yields_empty_list(self): + self.assertEqual(CarychamberScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_carysistercities_extract_fast.py b/backendServer/ingestion/tests/test_carysistercities_extract_fast.py new file mode 100644 index 0000000..7d865d7 --- /dev/null +++ b/backendServer/ingestion/tests/test_carysistercities_extract_fast.py @@ -0,0 +1,74 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.carysistercities import CarysistercitiesScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "carysistercities.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated event (2026-07-25, before "now") to exercise the past-event +# filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31, 12, 0, 0)) + + +@tag("fast") +class CarysistercitiesExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.carysistercities.timezone.now") + def test_extracts_future_events_and_drops_past(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarysistercitiesScraper().extract(self.html) + + # Fixture carries 4 events: 3 upcoming, 1 back-dated past event that + # the filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("CaryLIVE! The Suitcase Junket (Cary)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.carysistercities.timezone.now") + def test_google_calendar_dates_param_gives_utc_start_end(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarysistercitiesScraper().extract(self.html) + + final_friday = next(e for e in events if e.title == "Final Friday: Meet the Artists (Cary)") + self.assertEqual(final_friday.start.isoformat(), "2026-07-31T22:00:00+00:00") + self.assertEqual(final_friday.end.isoformat(), "2026-08-01T00:00:00+00:00") + self.assertIsNotNone(final_friday.start.tzinfo) + self.assertEqual( + final_friday.source_url, + "https://www.carysistercities.org/events-1/final-friday-meet-the-artists", + ) + self.assertEqual(final_friday.source_uid, final_friday.source_url) + self.assertIn("Final Fridays Art Crawl", final_friday.description) + # `description` is HTML on the wire; the extractor stores plain text. + self.assertNotIn("

", final_friday.description) + self.assertNotIn("", final_friday.description) + + @mock.patch("ingestion.scraping.scrapers.carysistercities.timezone.now") + def test_extracts_address_when_present(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarysistercitiesScraper().extract(self.html) + + lazy_daze = next(e for e in events if e.title == "Lazy Daze Beer Garden (CSC)") + self.assertEqual(lazy_daze.location, "Fred G. Bond Metro Park") + # The "(map)" link text must not leak into the venue name. + self.assertNotIn("(map)", lazy_daze.location) + + @mock.patch("ingestion.scraping.scrapers.carysistercities.timezone.now") + def test_missing_address_yields_empty_location(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = CarysistercitiesScraper().extract(self.html) + + asia_fest = next(e for e in events if e.title == "Asia Fest (CSC)") + self.assertEqual(asia_fest.location, "") + + def test_no_events_yields_empty_list(self): + self.assertEqual(CarysistercitiesScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_chapelhillarts_extract_fast.py b/backendServer/ingestion/tests/test_chapelhillarts_extract_fast.py new file mode 100644 index 0000000..6ed06a0 --- /dev/null +++ b/backendServer/ingestion/tests/test_chapelhillarts_extract_fast.py @@ -0,0 +1,66 @@ +import json +from datetime import datetime +from pathlib import Path +from unittest import mock +from zoneinfo import ZoneInfo + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.chapelhillarts import ChapelhillartsScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "chapelhillarts.html" +_LOCAL_TZ = ZoneInfo("America/New_York") + +# Fixed "now" the fixture's events (Jun 25 - Sep 25, 2026) were built to be +# future-dated against -- real data confirmed live via +# GET /wp-json/nmc-feeds/v1/events on 2026-07-31, back-dated here to before +# the earliest of those events since several had already passed by the +# capture date. The fixture also carries one back-dated 2020 event to +# exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 6, 1)) + + +@tag("fast") +class ChapelhillartsExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.chapelhillarts.timezone.now") + def test_extracts_future_events_from_json_feed(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ChapelhillartsScraper().extract(self.html) + + # Fixture carries 6 items: 1 back-dated to 2020 (dropped by the + # past-event filter) and 5 real future events. + self.assertEqual(len(events), 5) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + drone_talk = next(e for e in events if e.title == "Visiting Kalamkari Master Artist Talk") + self.assertEqual(drone_talk.start, datetime(2026, 7, 16, 17, 30, tzinfo=_LOCAL_TZ)) + self.assertIsNotNone(drone_talk.start.tzinfo) + self.assertEqual(drone_talk.source_uid, "8451") + self.assertEqual( + drone_talk.source_url, + "https://www.chapelhillarts.org/calendar/visiting-kalamkari-master-artist-talk/", + ) + # The feed's `end` field isn't valid ISO8601, so it's dropped rather + # than misparsed. + self.assertIsNone(drone_talk.end) + + def test_no_start_or_title_events_are_skipped(self): + html = json.dumps( + [ + {"id": 1, "title": "", "start": "2099-01-01T00:00:00", "url": "x"}, + {"id": 2, "start": "", "title": "No start"}, + ] + ) + self.assertEqual(ChapelhillartsScraper().extract(html), []) + + def test_malformed_json_yields_no_events(self): + self.assertEqual(ChapelhillartsScraper().extract("nope"), []) + + def test_non_list_json_yields_no_events(self): + self.assertEqual(ChapelhillartsScraper().extract('{"error": "not found"}'), []) diff --git a/backendServer/ingestion/tests/test_chapelhillnc_extract_fast.py b/backendServer/ingestion/tests/test_chapelhillnc_extract_fast.py new file mode 100644 index 0000000..e949a7e --- /dev/null +++ b/backendServer/ingestion/tests/test_chapelhillnc_extract_fast.py @@ -0,0 +1,65 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock +from zoneinfo import ZoneInfo + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.chapelhillnc import ChapelhillncScraper + +_LOCAL_TZ = ZoneInfo("America/New_York") + +FIXTURE = Path(__file__).parent / "fixtures" / "chapelhillnc.html" + +# Fixed "now" the fixture's events (July 2-29, 2026) were built to be +# future-dated against -- real data confirmed live on the site 2026-07-31, +# captured from a month-grid render whose events had already mostly passed +# by that date, so "now" is pinned to the start of that same month instead. +# The fixture also carries one back-dated 2020 event to exercise the +# past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 1)) + + +@tag("fast") +class ChapelhillncExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.chapelhillnc.timezone.now") + def test_extracts_future_events_from_calendar_grid(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ChapelhillncScraper().extract(self.html) + + # Fixture carries 6 day cells: 1 back-dated to 2020 (dropped by the + # past-event filter) and 5 real future events. + self.assertEqual(len(events), 5) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + drone_show = next(e for e in events if e.title == "Chapel Hill July 4th Drone Show") + self.assertEqual(drone_show.start, datetime(2026, 7, 4, tzinfo=_LOCAL_TZ)) + self.assertIsNotNone(drone_show.start.tzinfo) + self.assertEqual(drone_show.source_uid, "62ab25af-7708-4f73-80c3-53ad722a6312:2026-07-04") + # The DOM carries no start time, location, description, or permalink -- + # only day-level dates and a stable item id -- so these stay blank + # rather than being invented. + self.assertEqual(drone_show.location, "") + self.assertEqual(drone_show.description, "") + self.assertEqual(drone_show.source_url, "") + + @mock.patch("ingestion.scraping.scrapers.chapelhillnc.timezone.now") + def test_recurring_series_gets_per_occurrence_uid(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ChapelhillncScraper().extract(self.html) + + food_distribution = next(e for e in events if e.title == "Food Distribution") + # Same `data-item-id` as the site's other "Food Distribution" occurrence + # would reuse, disambiguated by date so each occurrence is unique. + self.assertEqual( + food_distribution.source_uid, "a5ff3873-c5e9-4f09-9449-47ca0db32b15:2026-07-08" + ) + + def test_no_calendar_cells_yields_no_events(self): + self.assertEqual(ChapelhillncScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_chathamchamber_extract_fast.py b/backendServer/ingestion/tests/test_chathamchamber_extract_fast.py new file mode 100644 index 0000000..00386f0 --- /dev/null +++ b/backendServer/ingestion/tests/test_chathamchamber_extract_fast.py @@ -0,0 +1,72 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.chathamchamber import ChathamchamberScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "chathamchamber.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# card whose dates were edited to 2020 to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class ChathamchamberExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.chathamchamber.timezone.now") + def test_extracts_future_and_ongoing_events_from_microdata(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ChathamchamberScraper().extract(self.html) + + # Fixture carries 5 cards: 4 real (one an ongoing series whose + # startDate is already past but endDate is future) + 1 back-dated + # to 2020 that the past-event filter must drop. + self.assertEqual(len(events), 4) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + film_series = next(e for e in events if e.title.startswith("Goldston America 250")) + self.assertEqual(film_series.start.isoformat(), "2026-01-17T16:00:00-05:00") + self.assertEqual(film_series.end.isoformat(), "2026-12-22T18:00:00-05:00") + self.assertIsNotNone(film_series.start.tzinfo) + self.assertEqual( + film_series.source_url, + "https://business.chathamchambernc.org/chatham-community-events/" + "Details/goldston-america-250-film-series-1591589", + ) + self.assertEqual(film_series.source_uid, film_series.source_url) + + walking_tour = next(e for e in events if e.title.startswith("Historical Walking Tour")) + self.assertEqual(walking_tour.start.isoformat(), "2026-08-09T13:00:00-04:00") + self.assertEqual(walking_tour.end.isoformat(), "2026-08-09T14:30:00-04:00") + self.assertEqual(walking_tour.description, "") + + @mock.patch("ingestion.scraping.scrapers.chathamchamber.timezone.now") + def test_description_is_html_unescaped_when_present(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ChathamchamberScraper().extract(self.html) + + race_amity = next(e for e in events if e.title == "Race Amity Day") + self.assertIn("Mindful Musical Walk", race_amity.description) + self.assertNotIn("nope"), []) diff --git a/backendServer/ingestion/tests/test_direct_ingest_db.py b/backendServer/ingestion/tests/test_direct_ingest_db.py index 5bc832f..91b2cc7 100644 --- a/backendServer/ingestion/tests/test_direct_ingest_db.py +++ b/backendServer/ingestion/tests/test_direct_ingest_db.py @@ -105,6 +105,8 @@ def test_over_threshold_holds_event(self): self.assertEqual(Event.objects.count(), 0) staged = StagedEvent.objects.get() self.assertEqual(staged.status, "pending") + # No prior Event exists on a first submission — must stay unset. + self.assertIsNone(staged.published_event) def test_blank_organizer_falls_back_to_host(self): """No organizer name on the raw row → generic 'by host' attribution.""" @@ -126,3 +128,147 @@ def test_anonymous_submission_no_user_id(self): self.assertFalse(event.is_verified) staged = StagedEvent.objects.get() self.assertIsNone(staged.submitted_by) + + +@tag("db") +class DirectIngestOrphanPreventionTests(TestCase): + """Ticket 40.4: a resubmitted-and-edited direct submission that gets + gated (duplicate / over-safety-threshold / no-town) must not strand the + Event a *prior* call to `ingest_direct_submission` already published. + `events.Event` has no soft-delete or published flag — its existence is + its publication — so the fix is to keep the terminal `StagedEvent` + pointed at that prior Event (`published_event`) rather than touch the + Event at all. Production audit (40.3) found 4 of 4 live direct-submission + Events orphaned this way. + """ + + STD_PAYLOAD = { + "title": "Test Concert", + "description": "A great show at the Cradle.", + "location_name": "Cat's Cradle", + "town": "Carrboro", + "tags": ["live-music"], + "price": 10, + } + + def setUp(self): + self.source = EventSource.objects.create( + name="Direct Feed", + source_type="ics", + url="https://feed.test/direct.ics", + ) + self.raw = RawEvent.objects.create( + source=self.source, + raw_title="Raw Concert", + raw_description="A show", + raw_location="Cat's Cradle, Carrboro, NC", + raw_start=datetime(2099, 6, 1, 18, 0, tzinfo=UTC), + source_url="", # avoids fetch_page_text network call + source_uid="direct-uid-orphan", + raw_organizer="The MAKRS Society", + ) + self.town = make_town(slug="carrboro") + self.user = make_user(user_type="BUSINESS") + + def _gemini_mocks(self, std_payload, score): + fake_std = mock.Mock() + fake_std.models.generate_content.return_value = mock.Mock(text=json.dumps(std_payload)) + fake_scorer = mock.Mock() + fake_scorer.models.generate_content.return_value = mock.Mock( + text=json.dumps({"score": score, "notes": ""}) + ) + return mock.patch( + "ingestion.standardizer.genai.Client", + side_effect=[fake_std, fake_scorer], + ) + + def test_resubmission_that_becomes_a_duplicate_keeps_prior_event_published(self): + with self._gemini_mocks(self.STD_PAYLOAD, 0.0): + first_event = ingest_direct_submission(self.raw.id, self.user.id) + self.assertEqual(Event.objects.count(), 1) + + # A genuine duplicate elsewhere in the system. find_duplicate only + # matches against a lower pk (see its docstring), and this row is + # created before the resubmission below, so it qualifies. + other = StagedEvent.objects.create( + title=self.STD_PAYLOAD["title"], + description="d", + location_name=self.STD_PAYLOAD["location_name"], + town=self.STD_PAYLOAD["town"], + start_datetime=self.raw.raw_start, + status="published", + ) + + with self._gemini_mocks(self.STD_PAYLOAD, 0.0): + result = ingest_direct_submission(self.raw.id, self.user.id) + + self.assertIsNone(result) + # The prior Event stays live and moderatable — not orphaned, not deleted. + self.assertEqual(Event.objects.count(), 1) + self.assertTrue(Event.objects.filter(pk=first_event.pk).exists()) + + staged = StagedEvent.objects.get(raw_event=self.raw) + self.assertEqual(staged.status, "duplicate") + self.assertEqual(staged.duplicate_of, other) + self.assertEqual(staged.published_event_id, first_event.pk) + + def test_resubmission_over_safety_threshold_keeps_prior_event_published(self): + with self._gemini_mocks(self.STD_PAYLOAD, 0.0): + first_event = ingest_direct_submission(self.raw.id, self.user.id) + self.assertEqual(Event.objects.count(), 1) + + with self._gemini_mocks(self.STD_PAYLOAD, 0.9): + result = ingest_direct_submission(self.raw.id, self.user.id) + + self.assertIsNone(result) + self.assertEqual(Event.objects.count(), 1) + self.assertTrue(Event.objects.filter(pk=first_event.pk).exists()) + + staged = StagedEvent.objects.get(raw_event=self.raw) + # Held for review, same terminal value the automated pipeline uses + # (see auto_publish_safe_events / monitoring.py's held_for_review + # bucket) — not silently left at whatever standardize_event defaulted to. + self.assertEqual(staged.status, "pending") + self.assertEqual(staged.published_event_id, first_event.pk) + + def test_resubmission_with_no_matching_town_keeps_prior_event_published(self): + with self._gemini_mocks(self.STD_PAYLOAD, 0.0): + first_event = ingest_direct_submission(self.raw.id, self.user.id) + self.assertEqual(Event.objects.count(), 1) + + out_of_coverage_payload = {**self.STD_PAYLOAD, "town": "Greensboro"} + with self._gemini_mocks(out_of_coverage_payload, 0.0): + result = ingest_direct_submission(self.raw.id, self.user.id) + + self.assertIsNone(result) + self.assertEqual(Event.objects.count(), 1) + staged = StagedEvent.objects.get(raw_event=self.raw) + self.assertEqual(staged.status, "skipped_no_town") + self.assertEqual(staged.published_event_id, first_event.pk) + + # The gated resubmission's content must never have touched the Event — + # only a fully successful re-publish is allowed to mutate it. + first_event.refresh_from_db() + self.assertEqual(first_event.town.slug, "carrboro") + + def test_first_submission_duplicate_has_no_published_event(self): + """No prior_event exists on a first-time duplicate — published_event + must stay None, not resurrect a reference to a nonexistent Event.""" + other = StagedEvent.objects.create( + title=self.STD_PAYLOAD["title"], + description="d", + location_name=self.STD_PAYLOAD["location_name"], + town=self.STD_PAYLOAD["town"], + start_datetime=self.raw.raw_start, + status="published", + ) + + with self._gemini_mocks(self.STD_PAYLOAD, 0.0): + result = ingest_direct_submission(self.raw.id, self.user.id) + + self.assertIsNone(result) + self.assertFalse(Event.objects.exists()) + staged = StagedEvent.objects.get(raw_event=self.raw) + self.assertEqual(staged.status, "duplicate") + self.assertEqual(staged.duplicate_of, other) + self.assertIsNone(staged.published_event) diff --git a/backendServer/ingestion/tests/test_downtowncarync_extract_fast.py b/backendServer/ingestion/tests/test_downtowncarync_extract_fast.py new file mode 100644 index 0000000..be20d9d --- /dev/null +++ b/backendServer/ingestion/tests/test_downtowncarync_extract_fast.py @@ -0,0 +1,72 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.downtowncarync import DowntowncarynScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "downtowncarync.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31, mid-afternoon UTC). The fixture keeps +# two back-dated (2026-07-03) events and one same-day (2026-07-31, 7pm ET, +# still future relative to this "now") plus one clearly-future (2026-08-02) +# event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31, 12, 0)) + + +@tag("fast") +class DowntowncarynExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.downtowncarync.timezone.now") + def test_extracts_future_events_from_month_grid(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntowncarynScraper().extract(self.html) + + # Fixture carries 5 events: 2 back-dated to July 3, 1 later today + # (July 31, 7pm), and 2 in the future (Aug 2). + self.assertEqual(len(events), 2) + titles = [e.title for e in events] + self.assertNotIn("July 3rd Celebration", titles) + self.assertNotIn( + "Cary Town Band presents America’s 250th Anniversary Patriotic Celebration", + titles, + ) + + dance = next(e for e in events if e.title == "Discover Dance: DISHOOM") + self.assertEqual(dance.start.isoformat(), "2026-07-31T19:00:00-04:00") + self.assertIsNotNone(dance.start.tzinfo) + self.assertEqual(dance.end.isoformat(), "2026-07-31T22:30:00-04:00") + self.assertEqual( + dance.source_url, "https://downtowncarync.org/event/discover-dance-dishoom-2/" + ) + self.assertEqual(dance.source_uid, dance.source_url) + self.assertEqual(dance.location, "") + self.assertIn("DISHOOM returns to Downtown Cary Park", dance.description) + + artists = next(e for e in events if e.title == "Meet the Artists: Fine Arts League of Cary") + self.assertEqual(artists.start.isoformat(), "2026-08-02T14:00:00-04:00") + + @mock.patch("ingestion.scraping.scrapers.downtowncarync.timezone.now") + def test_titles_are_html_unescaped(self, mock_now): + # Use a "now" before all fixture events so the back-dated (but + # entity-escaped) title survives the past-event filter. + mock_now.return_value = timezone.make_aware(datetime(2000, 1, 1)) + events = DowntowncarynScraper().extract(self.html) + + band = next(e for e in events if "Cary Town Band" in e.title) + self.assertEqual( + band.title, + "Cary Town Band presents America’s 250th Anniversary Patriotic Celebration", + ) + self.assertNotIn("’", band.description) + self.assertIn("America’s Semi-Quincentennial", band.description) + + def test_no_matching_articles_yields_no_events(self): + self.assertEqual(DowntowncarynScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_downtownraleigh_extract_fast.py b/backendServer/ingestion/tests/test_downtownraleigh_extract_fast.py new file mode 100644 index 0000000..2773cf2 --- /dev/null +++ b/backendServer/ingestion/tests/test_downtownraleigh_extract_fast.py @@ -0,0 +1,81 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.downtownraleigh import DowntownraleighScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "downtownraleigh.html" + +# Fixed "now" the fixture's events were built to be future-dated against (real +# data confirmed live 2026-07-31); the fixture also carries one back-dated +# "Jan 1" row (same header year, earlier month) to exercise the past-event +# filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class DowntownraleighExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.downtownraleigh.timezone.now") + def test_extracts_future_events_grouped_by_day(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntownraleighScraper().extract(self.html) + + # Fixture carries 7 events: 1 back-dated (Jan 1) that the filter must + # drop, 3 on Jul 31, 3 on Aug 1. The "Ongoing" bucket has no day/month + # anchor and is skipped entirely regardless of date. + self.assertEqual(len(events), 6) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + self.assertNotIn("Staying Alive Special Exhibition", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.downtownraleigh.timezone.now") + def test_time_range_parses_start_hour(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntownraleighScraper().extract(self.html) + + karaoke = next(e for e in events if e.title == "Crank Karaoke @ Crank Arm Brewing") + self.assertEqual(karaoke.start.isoformat(), "2026-07-31T20:00:00-04:00") + self.assertIsNotNone(karaoke.start.tzinfo) + self.assertIsNone(karaoke.end) + self.assertEqual(karaoke.location, "Crank Arm Brewing, Raleigh, NC") + self.assertEqual( + karaoke.source_url, + "https://downtownraleigh.org/do/crank-karaoke--crank-arm-brewing", + ) + self.assertEqual(karaoke.source_uid, "/do/crank-karaoke--crank-arm-brewing#2026-07-31") + + @mock.patch("ingestion.scraping.scrapers.downtownraleigh.timezone.now") + def test_range_without_leading_meridiem_uses_trailing_one(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntownraleighScraper().extract(self.html) + + reset = next(e for e in events if e.title == "Friday Reset for the Weekend") + # "1-3pm" -> starts at 1pm, not 3pm. + self.assertEqual(reset.start.hour, 13) + + @mock.patch("ingestion.scraping.scrapers.downtownraleigh.timezone.now") + def test_single_time_token_uppercase(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntownraleighScraper().extract(self.html) + + golf = next(e for e in events if e.title == "National Disc Golf Day") + self.assertEqual(golf.start.hour, 11) + + @mock.patch("ingestion.scraping.scrapers.downtownraleigh.timezone.now") + def test_no_time_text_defaults_to_midnight(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = DowntownraleighScraper().extract(self.html) + + cozy = next(e for e in events if e.title.startswith("Cozy Saturday")) + self.assertEqual((cozy.start.hour, cozy.start.minute), (0, 0)) + self.assertEqual(cozy.location, "Haymaker, Raleigh, NC") + + def test_no_rows_yields_no_events(self): + self.assertEqual(DowntownraleighScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_eventbritemorrisville_extract_fast.py b/backendServer/ingestion/tests/test_eventbritemorrisville_extract_fast.py new file mode 100644 index 0000000..d3f0255 --- /dev/null +++ b/backendServer/ingestion/tests/test_eventbritemorrisville_extract_fast.py @@ -0,0 +1,57 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.eventbritemorrisville import EventbritemorrisvilleScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "eventbritemorrisville.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class EventbritemorrisvilleExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network. + + Delegates to the shared `extract_eventbrite_events` helper tested more + thoroughly in `test_eventbriteraleigh_extract_fast.py`. This fixture is + also the receipt for the quality warning in + `eventbritemorrisville.py`'s docstring: none of the "Popular" bucket + events on the real Morrisville discovery page (captured 2026-07-31) are + actually Morrisville-located — extraction still works correctly, it + just doesn't return Morrisville events. + """ + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.eventbriteraleigh.timezone.now") + def test_extracts_future_events_none_of_which_are_morrisville(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = EventbritemorrisvilleScraper().extract(self.html) + + # Fixture carries 5 events: 3 future, 1 online (dropped), 1 + # back-dated to 2020 (dropped by the past-event filter). + self.assertEqual(len(events), 3) + # Real spillover: the Morrisville discovery page's "Popular" bucket + # returned only Raleigh-located events on this capture. + self.assertTrue(all("Raleigh" in e.location for e in events)) + self.assertTrue(all("Morrisville" not in e.location for e in events)) + + jersey = next(e for e in events if e.title.startswith("The 4th Annual Jersey Fest")) + self.assertEqual(jersey.start.isoformat(), "2026-08-01T18:00:00-04:00") + self.assertEqual(jersey.source_uid, "1992735047169") + self.assertEqual( + jersey.location, "The London Bridge Pub, 110 East Hargett Street, Raleigh, NC" + ) + + def test_no_server_data_yields_no_events(self): + self.assertEqual( + EventbritemorrisvilleScraper().extract("nope"), [] + ) diff --git a/backendServer/ingestion/tests/test_eventbritepittsboro_extract_fast.py b/backendServer/ingestion/tests/test_eventbritepittsboro_extract_fast.py new file mode 100644 index 0000000..50f6aae --- /dev/null +++ b/backendServer/ingestion/tests/test_eventbritepittsboro_extract_fast.py @@ -0,0 +1,50 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.eventbritepittsboro import EventbritepittsboroScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "eventbritepittsboro.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class EventbritepittsboroExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network. + + Delegates to the shared `extract_eventbrite_events` helper tested more + thoroughly in `test_eventbriteraleigh_extract_fast.py`; this test just + confirms the Pittsboro-specific fixture wires through correctly. + """ + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.eventbriteraleigh.timezone.now") + def test_extracts_future_local_events(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = EventbritepittsboroScraper().extract(self.html) + + # Fixture carries 5 events: 3 future local, 1 online (dropped), 1 + # back-dated to 2020 (dropped by the past-event filter). + self.assertEqual(len(events), 3) + + trivia = next(e for e in events if e.title == "Doherty's Pub Trivia w/ Patrick W") + self.assertEqual(trivia.start.isoformat(), "2026-08-04T19:00:00-04:00") + self.assertIsNotNone(trivia.start.tzinfo) + self.assertEqual(trivia.source_uid, "1989861724985") + self.assertEqual( + trivia.location, + "Doherty's Irish Pub & Restaurant, 56 Sanford Road, Pittsboro, NC", + ) + self.assertIn("Pittsboro, NC", trivia.description) + + def test_no_server_data_yields_no_events(self): + self.assertEqual(EventbritepittsboroScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_eventbriteraleigh_extract_fast.py b/backendServer/ingestion/tests/test_eventbriteraleigh_extract_fast.py new file mode 100644 index 0000000..37d5dd8 --- /dev/null +++ b/backendServer/ingestion/tests/test_eventbriteraleigh_extract_fast.py @@ -0,0 +1,66 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.eventbriteraleigh import EventbriteraleighScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "eventbriteraleigh.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class EventbriteraleighExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.eventbriteraleigh.timezone.now") + def test_extracts_future_local_events_from_server_data(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = EventbriteraleighScraper().extract(self.html) + + # Fixture carries 5 events: 3 future local, 1 online (dropped), 1 + # back-dated to 2020 (dropped by the past-event filter). + self.assertEqual(len(events), 3) + titles = [e.title for e in events] + self.assertNotIn( + "LITERACY. NOW. A National Parent-Led Conference to End the Literacy Crisis", titles + ) + self.assertNotIn( + "250 aka Finesse2tymes takes over CLUB LOVE (back-dated for filter test)", titles + ) + + club = next(e for e in events if e.title == "250 aka Finesse2tymes takes over CLUB LOVE") + self.assertEqual(club.start.isoformat(), "2026-07-31T22:00:00-04:00") + self.assertIsNotNone(club.start.tzinfo) + self.assertEqual(club.end.isoformat(), "2026-08-01T02:00:00-04:00") + self.assertEqual(club.source_uid, "1995343344657") + self.assertEqual( + club.source_url, + "https://www.eventbrite.com/e/250-aka-finesse2tymes-takes-over-club-love-tickets-1995343344657", + ) + # Venue name plus street address, so the standardizer can infer the town. + self.assertEqual(club.location, "4400 Craftsman Dr, 4400 Craftsman Drive, Raleigh, NC") + self.assertIn("free for everybody", club.description) + + @mock.patch("ingestion.scraping.scrapers.eventbriteraleigh.timezone.now") + def test_online_events_are_dropped(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = EventbriteraleighScraper().extract(self.html) + + self.assertTrue(all("LITERACY" not in e.title for e in events)) + + def test_no_server_data_yields_no_events(self): + self.assertEqual(EventbriteraleighScraper().extract("nope"), []) + + def test_malformed_server_data_yields_no_events(self): + html = "" + self.assertEqual(EventbriteraleighScraper().extract(html), []) diff --git a/backendServer/ingestion/tests/test_morrisvillechamber_extract_fast.py b/backendServer/ingestion/tests/test_morrisvillechamber_extract_fast.py new file mode 100644 index 0000000..6f93602 --- /dev/null +++ b/backendServer/ingestion/tests/test_morrisvillechamber_extract_fast.py @@ -0,0 +1,67 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.morrisvillechamber import MorrisvillechamberScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "morrisvillechamber.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter, and one event +# whose title the chamber marked *CANCELLED* to exercise that filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class MorrisvillechamberExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.morrisvillechamber.timezone.now") + def test_extracts_future_uncancelled_events(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvillechamberScraper().extract(self.html) + + # Fixture carries 5 events: 3 plain future events, 1 future event + # marked *CANCELLED* (dropped), 1 back-dated to 2020 (dropped). + self.assertEqual(len(events), 3) + titles = [e.title for e in events] + self.assertNotIn("*CANCELLED* Cregger Showroom Ribbon-Cutting After Hours Social", titles) + self.assertFalse(any("back-dated for filter test" in t for t in titles)) + + @mock.patch("ingestion.scraping.scrapers.morrisvillechamber.timezone.now") + def test_in_person_event_fields(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvillechamberScraper().extract(self.html) + + pizza = next(e for e in events if e.title.startswith("Bambino's Pizza")) + self.assertEqual(pizza.start.isoformat(), "2026-08-04T20:00:00-04:00") + self.assertIsNotNone(pizza.start.tzinfo) + self.assertEqual(pizza.end.isoformat(), "2026-08-04T21:00:00-04:00") + self.assertEqual( + pizza.source_url, + "https://morrisvillechamber.org/event-detail?e=EXCq5cP0tOLp7S7CEjtAWg2", + ) + self.assertEqual(pizza.source_uid, "EXCq5cP0tOLp7S7CEjtAWg2") + self.assertEqual(pizza.location, "4129 Davis Drive, Morrisville, NC, 27560") + self.assertIn("Grand Opening and Ribbon Cutting", pizza.description) + + @mock.patch("ingestion.scraping.scrapers.morrisvillechamber.timezone.now") + def test_online_event_has_no_street_address(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvillechamberScraper().extract(self.html) + + virtual = next(e for e in events if "Virtual" in e.title) + self.assertEqual(virtual.location, "Online") + + def test_invalid_json_yields_no_events(self): + self.assertEqual(MorrisvillechamberScraper().extract("not json"), []) + + def test_no_events_key_yields_no_events(self): + self.assertEqual(MorrisvillechamberScraper().extract('{"data": {}}'), []) diff --git a/backendServer/ingestion/tests/test_morrisvilleevents_extract_fast.py b/backendServer/ingestion/tests/test_morrisvilleevents_extract_fast.py new file mode 100644 index 0000000..de9cad7 --- /dev/null +++ b/backendServer/ingestion/tests/test_morrisvilleevents_extract_fast.py @@ -0,0 +1,70 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.morrisvilleevents import MorrisvilleeventsScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "morrisvilleevents.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class MorrisvilleeventsExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.morrisvilleevents.timezone.now") + def test_extracts_future_events_and_drops_past(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvilleeventsScraper().extract(self.html) + + # Fixture carries 5 events: 4 future, 1 back-dated to 2020 that the + # past-event filter must drop. + self.assertEqual(len(events), 4) + self.assertNotIn("Founders Day (back-dated for filter test)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.morrisvilleevents.timezone.now") + def test_event_fields(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvilleeventsScraper().extract(self.html) + + psac = next(e for e in events if e.title == "Public Safety Advisory Committee Meeting") + self.assertEqual( + psac.start.isoformat(), + datetime(2026, 8, 4, tzinfo=psac.start.tzinfo).isoformat(), + ) + self.assertIsNotNone(psac.start.tzinfo) + self.assertIsNone(psac.end) + self.assertEqual( + psac.source_url, + "https://www.morrisvillenc.gov/Events-directory/" + "Public-Safety-Advisory-Committee-Meeting", + ) + self.assertEqual(psac.source_uid, psac.source_url) + self.assertIn( + "health, safety, and welfare", + psac.description, + ) + self.assertEqual(psac.location, "Fire Station No. 1, 200 Town Hall Drive, 27560") + + @mock.patch("ingestion.scraping.scrapers.morrisvilleevents.timezone.now") + def test_address_trailing_comma_stripped_when_no_zip(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = MorrisvilleeventsScraper().extract(self.html) + + music = next(e for e in events if e.title == "Music in the Park") + # Fixture's address has no zip: "Indian Creek Trailhead, 101 Town + # Hall Drive,  " -- trailing comma/whitespace must be trimmed. + self.assertEqual(music.location, "Indian Creek Trailhead, 101 Town Hall Drive") + + def test_no_items_yields_no_events(self): + self.assertEqual(MorrisvilleeventsScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_patchdurham_extract_fast.py b/backendServer/ingestion/tests/test_patchdurham_extract_fast.py new file mode 100644 index 0000000..fa929d5 --- /dev/null +++ b/backendServer/ingestion/tests/test_patchdurham_extract_fast.py @@ -0,0 +1,62 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.patchdurham import PatchdurhamScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "patchdurham.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class PatchdurhamExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture — no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.patchdurham.timezone.now") + def test_extracts_future_events_from_next_data(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchdurhamScraper().extract(self.html) + + # Fixture carries 4 events across day buckets: 3 future, 1 back-dated + # to 2020 that the past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + concert = next(e for e in events if e.title.startswith("Beer Garden Concert Series")) + self.assertEqual(concert.start.isoformat(), "2026-07-31T22:00:00+00:00") + self.assertIsNotNone(concert.start.tzinfo) + self.assertIsNone(concert.end) + self.assertEqual(concert.source_uid, "20fdd37c-7c6b-4b89-9d06-e09553ee9838") + self.assertEqual( + concert.source_url, + "https://patch.com/north-carolina/durham-nc/calendar/event/20260731/" + "20fdd37c-7c6b-4b89-9d06-e09553ee9838/" + "beer-garden-concert-series-friday-night-edition-with-the-hourglass-kids", + ) + # Venue name plus street address, so the standardizer can infer the town. + self.assertEqual(concert.location, "The Glass Jug Beer Lab - RTP, 5410 NC-55, Durham, NC") + # `body` is HTML on the wire; the extractor stores plain text. + self.assertNotIn("

", concert.description) + self.assertNotIn("", concert.description) + self.assertIn("Glass Jug RTP", concert.description) + + @mock.patch("ingestion.scraping.scrapers.patchdurham.timezone.now") + def test_titles_are_html_unescaped(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchdurhamScraper().extract(self.html) + + soap = next(e for e in events if "Soap Making" in e.title) + self.assertEqual(soap.title, "Sip & Create: Soap Making Workshop") + + def test_no_next_data_yields_no_events(self): + self.assertEqual(PatchdurhamScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_patchmorrisville_extract_fast.py b/backendServer/ingestion/tests/test_patchmorrisville_extract_fast.py new file mode 100644 index 0000000..ead7be7 --- /dev/null +++ b/backendServer/ingestion/tests/test_patchmorrisville_extract_fast.py @@ -0,0 +1,64 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.patchmorrisville import PatchmorrisvilleScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "patchmorrisville.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# synthetic back-dated event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class PatchmorrisvilleExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network. + + Like Pittsboro, Morrisville's calendar carries only `patchAmFreeEvent` + nodes -- fetched from https://patch.com/north-carolina/morrisville-nc/calendar, + not the assigned section-landing URL (which has no `allEvents` at all). + """ + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_extracts_future_events_from_next_data(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchmorrisvilleScraper().extract(self.html) + + # Fixture carries 4 events across day buckets: 3 future, 1 back-dated + # to 2020 that the past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + rock_climb = next(e for e in events if "Homeschool Rock Climb" in e.title) + self.assertEqual(rock_climb.start.isoformat(), "2026-08-27T10:00:00-04:00") + self.assertIsNotNone(rock_climb.start.tzinfo) + self.assertIsNone(rock_climb.end) + self.assertEqual(rock_climb.source_uid, "10717592") + self.assertEqual( + rock_climb.source_url, + "https://raleighfamilyadventure.com/event/homeschool-rock-climb-at-trc-morrisville/2026-08-27/", + ) + self.assertEqual( + rock_climb.location, + "Triangle Rock Club - Morrisville, 102 Pheasant Wood Ct, Morrisville, NC", + ) + self.assertEqual(rock_climb.description, "") + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_evening_event_parses_pm_time(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchmorrisvilleScraper().extract(self.html) + + parents_night = next(e for e in events if "Parents" in e.title) + self.assertEqual(parents_night.start.isoformat(), "2026-08-21T18:00:00-04:00") + + def test_no_next_data_yields_no_events(self): + self.assertEqual(PatchmorrisvilleScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_patchpittsboro_extract_fast.py b/backendServer/ingestion/tests/test_patchpittsboro_extract_fast.py new file mode 100644 index 0000000..9666dde --- /dev/null +++ b/backendServer/ingestion/tests/test_patchpittsboro_extract_fast.py @@ -0,0 +1,68 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.patchpittsboro import PatchpittsboroScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "patchpittsboro.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# synthetic back-dated event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class PatchpittsboroExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network. + + Pittsboro's calendar carries only `patchAmFreeEvent` nodes (no + Patch-authored `event` nodes at all), so this exercises the aggregator + node shape: date + separate 12-hour time string, no body/description, + `externalUrl` as the source link. + """ + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_extracts_future_events_from_next_data(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchpittsboroScraper().extract(self.html) + + # Fixture carries 4 events across day buckets: 3 future, 1 back-dated + # to 2020 that the past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + okra = next(e for e in events if "Okra Jamboree" in e.title) + self.assertEqual(okra.start.isoformat(), "2026-08-02T12:00:00-04:00") + self.assertIsNotNone(okra.start.tzinfo) + self.assertIsNone(okra.end) + self.assertEqual(okra.source_uid, "10837944") + self.assertEqual( + okra.source_url, + "https://thetriangleweekender.com/event/third-annual-okra-jamboree/", + ) + # `patchAmAddressStr` already reads "Venue, Street, City"; the region + # gets appended since it isn't part of that string. + self.assertEqual(okra.location, "The Plant, 220 Lorax Ln, Pittsboro, NC") + # No body/summary field exists on this node shape. + self.assertEqual(okra.description, "") + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_titles_are_html_unescaped(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchpittsboroScraper().extract(self.html) + + dance = next(e for e in events if "Inclusive Dance" in e.title) + self.assertEqual( + dance.title, "ALL-Together! Friends Group – Inclusive Dance and Activities" + ) + self.assertEqual(dance.start.isoformat(), "2026-08-06T11:00:00-04:00") + + def test_no_next_data_yields_no_events(self): + self.assertEqual(PatchpittsboroScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_patchraleigh_extract_fast.py b/backendServer/ingestion/tests/test_patchraleigh_extract_fast.py new file mode 100644 index 0000000..fc58f2c --- /dev/null +++ b/backendServer/ingestion/tests/test_patchraleigh_extract_fast.py @@ -0,0 +1,65 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.patchraleigh import PatchraleighScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "patchraleigh.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# synthetic back-dated event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class PatchraleighExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_extracts_future_events_from_next_data(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchraleighScraper().extract(self.html) + + # Fixture carries 4 events across day buckets: 3 future, 1 back-dated + # to 2020 that the past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + concert = next(e for e in events if e.title.startswith("Beer Garden Concert Series")) + self.assertEqual(concert.start.isoformat(), "2026-07-31T22:00:00+00:00") + self.assertIsNotNone(concert.start.tzinfo) + self.assertIsNone(concert.end) + self.assertEqual(concert.source_uid, "c65e76f8-3165-4bab-a398-76a4c6d8fbe6") + self.assertEqual( + concert.source_url, + "https://patch.com/north-carolina/raleigh/calendar/event/20260731/" + "c65e76f8-3165-4bab-a398-76a4c6d8fbe6/" + "beer-garden-concert-series-friday-night-edition-with-the-hourglass-kids", + ) + self.assertEqual(concert.location, "The Glass Jug Beer Lab - RTP, 5410 NC-55, Durham, NC") + self.assertNotIn("

", concert.description) + self.assertNotIn("", concert.description) + + @mock.patch("ingestion.scraping.scrapers.patch.timezone.now") + def test_address_with_only_a_name_field_falls_back_gracefully(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = PatchraleighScraper().extract(self.html) + + soap = next(e for e in events if "Soap Making" in e.title) + # streetAddress/city/region are all empty on this real record; only + # the concatenated `name` field carries the full address (with the + # site's real non-breaking spaces between segments, preserved as-is). + self.assertEqual( + soap.location, + "The Glass Jug Downtown\xa0545 Foster Street, Suite 10\xa0Durham, NC 27701", + ) + + def test_no_next_data_yields_no_events(self): + self.assertEqual(PatchraleighScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_raleighnc_extract_fast.py b/backendServer/ingestion/tests/test_raleighnc_extract_fast.py new file mode 100644 index 0000000..6939bf2 --- /dev/null +++ b/backendServer/ingestion/tests/test_raleighnc_extract_fast.py @@ -0,0 +1,65 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.raleighnc import RaleighncScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "raleighnc.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated (Jan 1, 2026) teaser -- a copy of a real one with only its +# title and `datetime` edited -- to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class RaleighncExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.raleighnc.timezone.now") + def test_extracts_future_events_from_teaser_list(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = RaleighncScraper().extract(self.html) + + # Fixture carries 5 "Upcoming Events" teasers: 4 future, 1 back-dated + # to Jan 1 that the past-event filter must drop. The "Ongoing Events" + # sidebar article (no `

", carnival.description) + + @mock.patch("ingestion.scraping.scrapers.thrivinginraleigh.timezone.now") + def test_address_deduplicated_when_map_link_repeats_venue_name(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = ThrivinginraleighScraper().extract(self.html) + + live_after_5 = next(e for e in events if e.title.strip() == "Live After 5") + # The maps.google.com link for this event is just "?q=Moore Square" -- + # identical to the venue text -- so location must not repeat itself. + self.assertEqual(live_after_5.location, "Moore Square") + + def test_no_events_yields_no_events(self): + self.assertEqual(ThrivinginraleighScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_triangleonthecheap_extract_fast.py b/backendServer/ingestion/tests/test_triangleonthecheap_extract_fast.py new file mode 100644 index 0000000..7ab6c74 --- /dev/null +++ b/backendServer/ingestion/tests/test_triangleonthecheap_extract_fast.py @@ -0,0 +1,83 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.triangleonthecheap import TriangleonthecheapScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "triangleonthecheap.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated (Jan 1, 2026) day bucket -- a copy of a real event with only +# its header date and title edited -- to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class TriangleonthecheapExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.triangleonthecheap.timezone.now") + def test_extracts_future_events_grouped_by_day_header(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = TriangleonthecheapScraper().extract(self.html) + + # Fixture carries 8 events: 5 under "Today" + 2 under "Tomorrow" (7 + # future) and 1 back-dated to Jan 1 that the filter must drop. + self.assertEqual(len(events), 7) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.triangleonthecheap.timezone.now") + def test_time_range_parses_start_hour_with_explicit_meridiem(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = TriangleonthecheapScraper().extract(self.html) + + tasting = next(e for e in events if e.title.startswith("Tasting at Ten")) + self.assertEqual(tasting.start.isoformat(), "2026-07-31T10:00:00-04:00") + self.assertIsNotNone(tasting.start.tzinfo) + self.assertEqual( + tasting.location, "Counter Culture Coffee Headquarters and Training Center, Durham" + ) + self.assertEqual( + tasting.source_url, + "https://triangleonthecheap.com/tasting-ten-free-tasting-tour-counter-culture-coffee-hq/", + ) + + @mock.patch("ingestion.scraping.scrapers.triangleonthecheap.timezone.now") + def test_all_day_defaults_to_midnight(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = TriangleonthecheapScraper().extract(self.html) + + baskin = next(e for e in events if e.title.startswith("Baskin Robbins")) + self.assertEqual((baskin.start.hour, baskin.start.minute), (0, 0)) + self.assertEqual(baskin.location, "Baskin-Robbins, multiple locations") + + @mock.patch("ingestion.scraping.scrapers.triangleonthecheap.timezone.now") + def test_duplicate_permalink_stays_unique_per_title_same_day(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = TriangleonthecheapScraper().extract(self.html) + + hillsborough = [ + e + for e in events + if e.source_url == "https://triangleonthecheap.com/hillsborough-last-fridays-art-walk/" + ] + # Three distinct listings share one permalink on the same date -- + # source_uid must disambiguate all three. + self.assertEqual(len(hillsborough), 3) + self.assertEqual(len({e.source_uid for e in hillsborough}), 3) + + live_on_lawn = next( + e for e in hillsborough if e.title == "Hillsborough Arts Council's Live on the Lawn" + ) + # 2-segment meta ("time | price", no venue segment) -> no location. + self.assertEqual(live_on_lawn.location, "") + + def test_no_day_headers_yields_no_events(self): + self.assertEqual(TriangleonthecheapScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_visitraleigh_cary_extract_fast.py b/backendServer/ingestion/tests/test_visitraleigh_cary_extract_fast.py new file mode 100644 index 0000000..156336c --- /dev/null +++ b/backendServer/ingestion/tests/test_visitraleigh_cary_extract_fast.py @@ -0,0 +1,68 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.visitraleigh_cary import VisitraleighCaryScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "visitraleigh_cary.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class VisitraleighCaryExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network. + + Exercises the same `_extract_events` helper as `visitraleigh.py` via the + Cary-scoped subclass, so this focuses on cases that module's own test + doesn't cover (the "Recurring ... until " no-start-date case) + rather than re-testing shared parsing logic end to end. + """ + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_extracts_future_events(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighCaryScraper().extract(self.html) + + # Fixture carries 4 events: 3 future, 1 back-dated to 2020 that the + # past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_recurring_listing_falls_back_to_until_date(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighCaryScraper().extract(self.html) + + clue = next(e for e in events if e.title == "Clue (High School Edition)") + # "Recurring weekly on Sunday, Friday, Saturday until August 9, 2026" + # has only one parseable date, so it becomes `start` with no `end`. + self.assertEqual(clue.start.isoformat(), "2026-08-09T19:30:00-04:00") + self.assertIsNone(clue.end) + self.assertEqual(clue.source_uid, "105919") + self.assertEqual(clue.location, "Raleigh Little Theatre") + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_source_url_and_name(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighCaryScraper().extract(self.html) + + comedy = next(e for e in events if e.title.startswith("Comedy Show")) + self.assertEqual( + comedy.source_url, + "https://www.visitraleigh.com/event/comedy-show%3a-cyrus-steele-%2b-joe-perrow/108510/", + ) + self.assertEqual(VisitraleighCaryScraper.name, "Visit Raleigh (Cary)") + self.assertEqual(VisitraleighCaryScraper.key, "visitraleigh_cary") + + def test_no_recid_items_yields_no_events(self): + self.assertEqual(VisitraleighCaryScraper().extract("nope"), []) diff --git a/backendServer/ingestion/tests/test_visitraleigh_extract_fast.py b/backendServer/ingestion/tests/test_visitraleigh_extract_fast.py new file mode 100644 index 0000000..f10e5ef --- /dev/null +++ b/backendServer/ingestion/tests/test_visitraleigh_extract_fast.py @@ -0,0 +1,80 @@ +from datetime import datetime +from pathlib import Path +from unittest import mock + +from django.test import SimpleTestCase, tag +from django.utils import timezone + +from ingestion.scraping.scrapers.visitraleigh import VisitraleighScraper + +FIXTURE = Path(__file__).parent / "fixtures" / "visitraleigh.html" + +# Fixed "now" the fixture's events were built to be future-dated against +# (real data confirmed live 2026-07-31); the fixture also carries one +# back-dated 2020 event to exercise the past-event filter. +_FIXTURE_BUILD_TIME = timezone.make_aware(datetime(2026, 7, 31)) + + +@tag("fast") +class VisitraleighExtractTests(SimpleTestCase): + """Pure `extract()` against a saved real fixture -- no DB, no browser, no network.""" + + def setUp(self): + self.html = FIXTURE.read_text(encoding="utf-8") + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_extracts_future_events(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighScraper().extract(self.html) + + # Fixture carries 4 events: 3 future, 1 back-dated to 2020 that the + # past-event filter must drop. + self.assertEqual(len(events), 3) + self.assertNotIn("Past Event (back-dated for filter test)", [e.title for e in events]) + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_single_date_and_time(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighScraper().extract(self.html) + + rocky = next(e for e in events if e.title == "The Rocky Horror Picture Show") + self.assertEqual(rocky.start.isoformat(), "2026-07-31T20:30:00-04:00") + self.assertIsNotNone(rocky.start.tzinfo) + self.assertIsNone(rocky.end) + self.assertEqual(rocky.source_uid, "109424") + self.assertEqual( + rocky.source_url, + "https://www.visitraleigh.com/event/the-rocky-horror-picture-show/109424/", + ) + self.assertEqual(rocky.location, "The Rialto Theatre") + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_date_range_with_vary_prefix(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighScraper().extract(self.html) + + andrew = next(e for e in events if e.title == "Andrew Orolfo") + # "Dates vary between July 31, 2026 - August 1, 2026", first time + # token "9:15pm" applied to the start date. + self.assertEqual(andrew.start.isoformat(), "2026-07-31T21:15:00-04:00") + self.assertEqual(andrew.end.isoformat(), "2026-08-01T00:00:00-04:00") + + @mock.patch("ingestion.scraping.scrapers.visitraleigh.timezone.now") + def test_date_range_without_vary_prefix(self, mock_now): + mock_now.return_value = _FIXTURE_BUILD_TIME + events = VisitraleighScraper().extract(self.html) + + bbb = next(e for e in events if e.title.startswith("Beer, Bourbon")) + self.assertEqual(bbb.start.isoformat(), "2026-07-31T18:00:00-04:00") + self.assertEqual(bbb.end.isoformat(), "2026-08-01T00:00:00-04:00") + self.assertEqual(bbb.location, "Koka Booth Amphitheatre") + + def test_no_recid_items_yields_no_events(self): + self.assertEqual(VisitraleighScraper().extract("nope"), []) + + def test_unrendered_template_placeholder_is_skipped(self): + html = ( + '

' + '

{{title}}

' + ) + self.assertEqual(VisitraleighScraper().extract(html), []) diff --git a/backendServer/ingestion/urls.py b/backendServer/ingestion/urls.py new file mode 100644 index 0000000..c63121c --- /dev/null +++ b/backendServer/ingestion/urls.py @@ -0,0 +1,20 @@ +from django.urls import path + +from . import views + +urlpatterns = [ + path("admin/docs/pipeline-docs/", views.pipeline_docs, name="pipeline-docs"), + path("admin/docs/admin-docs/", views.admin_docs, name="admin-docs"), + path( + "admin/docs/publish-approved/", + views.publish_approved_admin, + name="publish-approved-admin", + ), + path("api/cron/ingest", views.cron_ingest, name="cron-ingest"), + path( + "api/events/publish-approved", + views.publish_approved_events, + name="publish-approved-events", + ), + path("api/events/direct-submit", views.direct_submit, name="direct-submit"), +] diff --git a/backendServer/newsletter/__init__.py b/backendServer/newsletter/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/newsletter/admin.py b/backendServer/newsletter/admin.py new file mode 100644 index 0000000..a5535fa --- /dev/null +++ b/backendServer/newsletter/admin.py @@ -0,0 +1,11 @@ +from django.contrib import admin +from unfold.admin import ModelAdmin + +from .models import NewsletterSubscriber + + +@admin.register(NewsletterSubscriber) +class NewsletterSubscriberAdmin(ModelAdmin): + list_display = ["email", "frequency", "is_active", "subscribed_at"] + list_filter = ["frequency", "is_active"] + search_fields = ["email"] diff --git a/backendServer/newsletter/apps.py b/backendServer/newsletter/apps.py new file mode 100644 index 0000000..f76bbe1 --- /dev/null +++ b/backendServer/newsletter/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class NewsletterConfig(AppConfig): + name = "newsletter" diff --git a/backendServer/newsletter/email_service.py b/backendServer/newsletter/email_service.py new file mode 100644 index 0000000..4bff063 --- /dev/null +++ b/backendServer/newsletter/email_service.py @@ -0,0 +1,124 @@ +import logging +import os +from datetime import timedelta + +from django.template.loader import render_to_string +from django.utils import timezone + +from events.email_service import send_email + +logger = logging.getLogger(__name__) + + +def send_newsletter_welcome(email: str, manage_token) -> bool: + """Send the welcome email for a new (or re-)subscription. + + Includes the login-free manage link keyed by the subscriber's manage_token. + Best-effort — a Brevo failure here should never block the subscribe response. + """ + manage_url = manage_url_for(manage_token) + subject = "You're subscribed to The Commons" + html = ( + "

Thanks for subscribing to The Commons newsletter.

" + "

You can change your frequency or unsubscribe anytime, no login required, " + f'at {manage_url}.

' + ) + return send_email(email, subject, html) + + +def digest_window(frequency: str) -> tuple: + """Return (cutoff, subject) for a digest frequency. Shared by send_digest + and the per-recipient Celery task so the two paths can't drift apart. + """ + if frequency == "WEEKLY": + return timezone.now() + timedelta(days=7), "This Week in The Commons" + return timezone.now() + timedelta(days=31), "This Month in The Commons" + + +def manage_url_for(manage_token) -> str: + site_url = os.environ.get("SITE_URL", "https://www.thecommons.town") + return f"{site_url}/newsletter/manage?token={manage_token}" + + +def _build_recipients(frequency: str) -> list[dict]: + """Return [{email, tags: set[str], manage_token}] for all active subscribers. + + NewsletterSubscriber is the single source of truth for digest recipients — + every row (anonymous or account-holding) carries a manage_token, so this is + the only resolver that can produce a working manage/unsubscribe link. When + a subscriber's email also has a UserProfile, its tags narrow the digest to + matching events; otherwise (anonymous) the empty set sends everything. + Both the Celery fan-out and the send_digest command call this — no + divergent recipient logic anywhere else. + """ + from accounts.models import UserProfile + from newsletter.models import NewsletterSubscriber + + profile_tags_by_email = { + profile.user.email.lower(): {t.name for t in profile.tags.all()} + for profile in UserProfile.objects.select_related("user").prefetch_related("tags") + } + + recipients = [] + for sub in NewsletterSubscriber.objects.filter(frequency=frequency, is_active=True): + recipients.append( + { + "email": sub.email, + "tags": profile_tags_by_email.get(sub.email.lower(), set()), + "manage_token": sub.manage_token, + } + ) + return recipients + + +def send_digest(frequency: str) -> dict: + """Send the weekly or monthly digest to all active subscribers. + + Returns a dict with 'sent' and 'failed' counts. + """ + from events.models import Event + + cutoff, subject = digest_window(frequency) + + all_events = list( + Event.objects.filter(date__gte=timezone.now(), date__lte=cutoff) + .select_related("town") + .prefetch_related("tags") + .order_by("date") + ) + + recipients = _build_recipients(frequency) + if not recipients: + logger.info("No active %s subscribers — skipping digest.", frequency) + return {"sent": 0, "failed": 0} + + site_url = os.environ.get("SITE_URL", "https://www.thecommons.town") + + sent = failed = 0 + for recipient in recipients: + tag_filter = recipient["tags"] + if tag_filter: + events = [ + e for e in all_events if tag_filter.intersection({t.name for t in e.tags.all()}) + ] + else: + events = all_events + + html_body = render_to_string( + "email/digest.html", + { + "events": events, + "frequency": frequency, + "subject": subject, + "site_url": site_url, + "manage_url": manage_url_for(recipient["manage_token"]), + }, + ) + + if send_email(recipient["email"], subject, html_body): + sent += 1 + else: + failed += 1 + + logger.info("Digest sent: %d succeeded, %d failed.", sent, failed) + return {"sent": sent, "failed": failed} diff --git a/backendServer/newsletter/management/__init__.py b/backendServer/newsletter/management/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/newsletter/management/commands/__init__.py b/backendServer/newsletter/management/commands/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/events/management/commands/send_digest.py b/backendServer/newsletter/management/commands/send_digest.py similarity index 93% rename from backendServer/events/management/commands/send_digest.py rename to backendServer/newsletter/management/commands/send_digest.py index ae42a6e..c802b2c 100644 --- a/backendServer/events/management/commands/send_digest.py +++ b/backendServer/newsletter/management/commands/send_digest.py @@ -1,6 +1,6 @@ from django.core.management.base import BaseCommand -from events.email_service import send_digest +from newsletter.email_service import send_digest class Command(BaseCommand): diff --git a/backendServer/events/management/commands/send_test_digest.py b/backendServer/newsletter/management/commands/send_test_digest.py similarity index 100% rename from backendServer/events/management/commands/send_test_digest.py rename to backendServer/newsletter/management/commands/send_test_digest.py diff --git a/backendServer/events/management/commands/send_weekly_digest.py b/backendServer/newsletter/management/commands/send_weekly_digest.py similarity index 90% rename from backendServer/events/management/commands/send_weekly_digest.py rename to backendServer/newsletter/management/commands/send_weekly_digest.py index 37db4cb..2b8deb5 100644 --- a/backendServer/events/management/commands/send_weekly_digest.py +++ b/backendServer/newsletter/management/commands/send_weekly_digest.py @@ -1,6 +1,6 @@ from django.core.management.base import BaseCommand -from events.tasks import fan_out_weekly_digest +from newsletter.tasks import fan_out_weekly_digest class Command(BaseCommand): diff --git a/backendServer/newsletter/migrations/0001_initial.py b/backendServer/newsletter/migrations/0001_initial.py new file mode 100644 index 0000000..5929e28 --- /dev/null +++ b/backendServer/newsletter/migrations/0001_initial.py @@ -0,0 +1,59 @@ +# NewsletterSubscriber now lives in newsletter/ (see events/migrations/ +# 0022_move_newsletter_to_newsletter.py for the paired events-side move). +# state_operations only — no DDL runs; the physical table +# (events_newslettersubscriber) is untouched. + +import uuid + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ("events", "0021_move_identity_to_accounts"), + ] + + operations = [ + migrations.SeparateDatabaseAndState( + database_operations=[], + state_operations=[ + migrations.CreateModel( + name="NewsletterSubscriber", + fields=[ + ( + "id", + models.BigAutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ("email", models.EmailField(max_length=254, unique=True)), + ( + "frequency", + models.CharField( + choices=[("WEEKLY", "Weekly"), ("MONTHLY", "Monthly")], + default="WEEKLY", + max_length=10, + ), + ), + ("is_active", models.BooleanField(default=True)), + ("subscribed_at", models.DateTimeField(auto_now_add=True)), + ( + "manage_token", + models.UUIDField( + default=uuid.uuid4, editable=False, unique=True, db_index=True + ), + ), + ], + options={ + "db_table": "events_newslettersubscriber", + }, + ), + ], + ), + ] diff --git a/backendServer/newsletter/migrations/0002_repoint_digest_beat.py b/backendServer/newsletter/migrations/0002_repoint_digest_beat.py new file mode 100644 index 0000000..6d7781a --- /dev/null +++ b/backendServer/newsletter/migrations/0002_repoint_digest_beat.py @@ -0,0 +1,43 @@ +"""Repoint the beat PeriodicTask rows seeded by events/migrations/0015 and +events/migrations/0020 to the digest engine's new home in newsletter/tasks.py +(ticket 41.8). + +The historical seed migrations (events 0015_seed_digest_beat, +0020_seed_monthly_digest_beat) are left untouched — they created the rows +with task="events.tasks.fan_out_weekly_digest" / "...fan_out_monthly_digest". +This migration only UPDATEs the `task` dotted-path on those existing rows so +django-celery-beat's DatabaseScheduler dispatches to the tasks' new Celery +names (newsletter.tasks.fan_out_weekly_digest / fan_out_monthly_digest, +auto-derived from the module @shared_task now lives in). Fully reversible. +""" +from django.db import migrations + +WEEKLY_OLD = "events.tasks.fan_out_weekly_digest" +WEEKLY_NEW = "newsletter.tasks.fan_out_weekly_digest" +MONTHLY_OLD = "events.tasks.fan_out_monthly_digest" +MONTHLY_NEW = "newsletter.tasks.fan_out_monthly_digest" + + +def repoint_forward(apps, schema_editor): + PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask") + PeriodicTask.objects.filter(task=WEEKLY_OLD).update(task=WEEKLY_NEW) + PeriodicTask.objects.filter(task=MONTHLY_OLD).update(task=MONTHLY_NEW) + + +def repoint_reverse(apps, schema_editor): + PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask") + PeriodicTask.objects.filter(task=WEEKLY_NEW).update(task=WEEKLY_OLD) + PeriodicTask.objects.filter(task=MONTHLY_NEW).update(task=MONTHLY_OLD) + + +class Migration(migrations.Migration): + + dependencies = [ + ("newsletter", "0001_initial"), + ("events", "0020_seed_monthly_digest_beat"), + ("django_celery_beat", "0019_alter_periodictasks_options"), + ] + + operations = [ + migrations.RunPython(repoint_forward, repoint_reverse), + ] diff --git a/backendServer/newsletter/migrations/__init__.py b/backendServer/newsletter/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/newsletter/models.py b/backendServer/newsletter/models.py new file mode 100644 index 0000000..9ba3e30 --- /dev/null +++ b/backendServer/newsletter/models.py @@ -0,0 +1,21 @@ +import uuid + +from django.db import models + + +class NewsletterSubscriber(models.Model): + class Frequency(models.TextChoices): + WEEKLY = "WEEKLY", "Weekly" + MONTHLY = "MONTHLY", "Monthly" + + email = models.EmailField(unique=True) + frequency = models.CharField(max_length=10, choices=Frequency.choices, default=Frequency.WEEKLY) + is_active = models.BooleanField(default=True) + subscribed_at = models.DateTimeField(auto_now_add=True) + manage_token = models.UUIDField(default=uuid.uuid4, editable=False, unique=True, db_index=True) + + class Meta: + db_table = "events_newslettersubscriber" + + def __str__(self): + return f"{self.email} ({self.frequency})" diff --git a/backendServer/newsletter/tasks.py b/backendServer/newsletter/tasks.py new file mode 100644 index 0000000..eaaab8e --- /dev/null +++ b/backendServer/newsletter/tasks.py @@ -0,0 +1,91 @@ +import logging +import os + +from celery import shared_task +from django.template.loader import render_to_string +from django.utils import timezone + +from events.email_service import send_email +from newsletter.email_service import _build_recipients, digest_window, manage_url_for + +logger = logging.getLogger(__name__) + + +@shared_task(bind=True, max_retries=3, default_retry_delay=300) +def send_one_digest(self, email, tags, manage_token, frequency): + """Render and send the personalized digest to one resolved recipient. + + Takes an already-resolved recipient (email/tags/manage_token/frequency) + rather than a UserProfile id, so it serves both authenticated and + anonymous NewsletterSubscriber rows alike — the fan-out tasks are the only + callers, and both go through email_service._build_recipients first. `tags` + arrives as a list (Celery JSON-serializes task args) and is treated as a + set of interest-tag names to filter events by; an empty list sends + everything. send_email swallows Brevo errors and returns False, so a + falsy return triggers a retry (3x, 5-min backoff) without affecting other + recipients. + """ + from events.models import Event + + tag_filter = set(tags) + now = timezone.now() + cutoff, subject = digest_window(frequency) + + events = ( + Event.objects.filter(date__gte=now, date__lte=cutoff) + .select_related("town") + .prefetch_related("tags") + .order_by("date") + ) + if tag_filter: + events = [e for e in events if tag_filter.intersection({t.name for t in e.tags.all()})] + else: + events = list(events) + + if not events: + logger.info("send_one_digest: %s has no matching events — skipping.", email) + return + + site_url = os.environ.get("SITE_URL", "https://www.thecommons.town") + html = render_to_string( + "email/digest.html", + { + "events": events, + "frequency": frequency, + "subject": subject, + "site_url": site_url, + "manage_url": manage_url_for(manage_token), + }, + ) + + if send_email(email, subject, html): + logger.info("send_one_digest: sent to %s (%d events).", email, len(events)) + return + + logger.warning("send_one_digest: send to %s failed — retrying.", email) + raise self.retry() + + +def _queue_digest_fan_out(frequency): + recipients = _build_recipients(frequency) + for recipient in recipients: + send_one_digest.delay( + recipient["email"], + list(recipient["tags"]), + str(recipient["manage_token"]), + frequency, + ) + logger.info("fan_out_%s_digest: queued %d digest subtasks.", frequency.lower(), len(recipients)) + return len(recipients) + + +@shared_task +def fan_out_weekly_digest(): + """Queue one send_one_digest subtask per WEEKLY subscriber. Returns the count.""" + return _queue_digest_fan_out("WEEKLY") + + +@shared_task +def fan_out_monthly_digest(): + """Queue one send_one_digest subtask per MONTHLY subscriber. Returns the count.""" + return _queue_digest_fan_out("MONTHLY") diff --git a/backendServer/templates/email/digest.html b/backendServer/newsletter/templates/email/digest.html similarity index 100% rename from backendServer/templates/email/digest.html rename to backendServer/newsletter/templates/email/digest.html diff --git a/backendServer/templates/email/weekly_digest.html b/backendServer/newsletter/templates/email/weekly_digest.html similarity index 100% rename from backendServer/templates/email/weekly_digest.html rename to backendServer/newsletter/templates/email/weekly_digest.html diff --git a/backendServer/newsletter/tests/__init__.py b/backendServer/newsletter/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backendServer/events/tests/test_digest.py b/backendServer/newsletter/tests/test_digest.py similarity index 82% rename from backendServer/events/tests/test_digest.py rename to backendServer/newsletter/tests/test_digest.py index 1ee9f05..0b26cdc 100644 --- a/backendServer/events/tests/test_digest.py +++ b/backendServer/newsletter/tests/test_digest.py @@ -5,10 +5,10 @@ from django.test import TestCase, tag from django.utils import timezone -from events.models import Event, NewsletterSubscriber -from events.tasks import fan_out_monthly_digest, fan_out_weekly_digest, send_one_digest - -from .factories import make_town +from events.models import Event +from events.tests.factories import make_town +from newsletter.models import NewsletterSubscriber +from newsletter.tasks import fan_out_monthly_digest, fan_out_weekly_digest, send_one_digest @tag("db") @@ -44,7 +44,7 @@ def test_fan_out_weekly_queues_one_subtask_per_weekly_subscriber(self): self._make_subscriber("MONTHLY", email="c@example.com") # excluded self._make_subscriber("WEEKLY", is_active=False, email="d@example.com") # excluded - with mock.patch("events.tasks.send_one_digest.delay") as delay: + with mock.patch("newsletter.tasks.send_one_digest.delay") as delay: count = fan_out_weekly_digest.delay().get() self.assertEqual(count, 2) @@ -54,26 +54,26 @@ def test_fan_out_monthly_queues_one_subtask_per_monthly_subscriber(self): self._make_subscriber("MONTHLY", email="a@example.com") self._make_subscriber("WEEKLY", email="b@example.com") # excluded - with mock.patch("events.tasks.send_one_digest.delay") as delay: + with mock.patch("newsletter.tasks.send_one_digest.delay") as delay: count = fan_out_monthly_digest.delay().get() self.assertEqual(count, 1) delay.assert_called_once_with("a@example.com", [], mock.ANY, "MONTHLY") def test_send_one_digest_sends_for_matching_events(self): - with mock.patch("events.tasks.send_email", return_value=True) as send: + with mock.patch("newsletter.tasks.send_email", return_value=True) as send: send_one_digest.delay("reader@example.com", [], "some-token", "WEEKLY") send.assert_called_once() self.assertEqual(send.call_args.args[0], "reader@example.com") def test_send_one_digest_skips_when_no_events_match_tags(self): # The one seeded event has no tags, so a tag-filtered recipient gets nothing. - with mock.patch("events.tasks.send_email") as send: + with mock.patch("newsletter.tasks.send_email") as send: send_one_digest.delay("reader@example.com", ["music"], "some-token", "WEEKLY") send.assert_not_called() def test_send_one_digest_retries_on_brevo_failure(self): - with mock.patch("events.tasks.send_email", return_value=False) as send: + with mock.patch("newsletter.tasks.send_email", return_value=False) as send: # In eager mode self.retry() raises Retry; confirms a Brevo failure # requests a retry of this one subtask. with self.assertRaises(Retry): diff --git a/backendServer/events/tests/test_digest_db.py b/backendServer/newsletter/tests/test_digest_db.py similarity index 92% rename from backendServer/events/tests/test_digest_db.py rename to backendServer/newsletter/tests/test_digest_db.py index c00a845..1de6b9a 100644 --- a/backendServer/events/tests/test_digest_db.py +++ b/backendServer/newsletter/tests/test_digest_db.py @@ -5,10 +5,10 @@ from django.utils import timezone from django_celery_beat.models import PeriodicTask -from events.email_service import _build_recipients, send_digest -from events.models import Event, NewsletterSubscriber, Tag - -from .factories import make_town, make_user +from events.models import Event, Tag +from events.tests.factories import make_town, make_user +from newsletter.email_service import _build_recipients, send_digest +from newsletter.models import NewsletterSubscriber @tag("db") @@ -19,7 +19,7 @@ class MonthlyBeatScheduleSeedTests(TestCase): def test_monthly_digest_schedule_seeded(self): pt = PeriodicTask.objects.get(name="monthly-digest-first") - self.assertEqual(pt.task, "events.tasks.fan_out_monthly_digest") + self.assertEqual(pt.task, "newsletter.tasks.fan_out_monthly_digest") self.assertTrue(pt.enabled) self.assertEqual(pt.crontab.minute, "0") self.assertEqual(pt.crontab.hour, "18") @@ -105,7 +105,7 @@ def fake_send_email(to, subject, html, text=None): captured["html"] = html return True - with mock.patch("events.email_service.send_email", side_effect=fake_send_email): + with mock.patch("newsletter.email_service.send_email", side_effect=fake_send_email): result = send_digest("WEEKLY") self.assertEqual(result, {"sent": 1, "failed": 0}) diff --git a/backendServer/newsletter/tests/test_isolation_fast.py b/backendServer/newsletter/tests/test_isolation_fast.py new file mode 100644 index 0000000..412fb46 --- /dev/null +++ b/backendServer/newsletter/tests/test_isolation_fast.py @@ -0,0 +1,32 @@ +import ast +import pathlib + +from django.test import SimpleTestCase, tag + +# newsletter legitimately imports events (events.email_service.send_email is +# the shared Brevo transport) and accounts (newsletter.email_service._build_ +# recipients reads UserProfile.tags to narrow digests — mirrors accounts' +# own sanctioned import of newsletter for its `me` view). It must never +# reach into ingestion or broadcast. +FORBIDDEN_ROOTS = {"ingestion", "broadcast"} + + +@tag("fast") +class IsolationTest(SimpleTestCase): + def test_newsletter_imports_nothing_from_ingestion_or_broadcast(self): + root = pathlib.Path(__file__).resolve().parents[1] + offenders = [] + for path in root.rglob("*.py"): + if "tests" in path.parts or "migrations" in path.parts: + continue + tree = ast.parse(path.read_text(), filename=str(path)) + for node in ast.walk(tree): + mods = [] + if isinstance(node, ast.ImportFrom) and node.module: + mods.append(node.module) + if isinstance(node, ast.Import): + mods.extend(a.name for a in node.names) + for m in mods: + if m.split(".")[0] in FORBIDDEN_ROOTS: + offenders.append((str(path), m)) + self.assertFalse(offenders, f"isolation breach: {offenders}") diff --git a/backendServer/events/tests/test_newsletter_db.py b/backendServer/newsletter/tests/test_newsletter_db.py similarity index 94% rename from backendServer/events/tests/test_newsletter_db.py rename to backendServer/newsletter/tests/test_newsletter_db.py index 7f8791f..96aebed 100644 --- a/backendServer/events/tests/test_newsletter_db.py +++ b/backendServer/newsletter/tests/test_newsletter_db.py @@ -3,14 +3,14 @@ from django.test import TestCase, tag from django.urls import reverse -from events.models import NewsletterSubscriber +from newsletter.models import NewsletterSubscriber @tag("db") class NewsletterSubscribeTests(TestCase): def setUp(self): # subscribe() now sends a welcome email; keep these tests off the network. - patcher = mock.patch("events.views.send_newsletter_welcome", return_value=True) + patcher = mock.patch("newsletter.views.send_newsletter_welcome", return_value=True) patcher.start() self.addCleanup(patcher.stop) @@ -54,7 +54,7 @@ def test_missing_email_is_400(self): self.assertEqual(resp.json()["error"], "email is required") def test_subscribe_attempts_welcome_email_with_manage_link(self): - with mock.patch("events.views.send_newsletter_welcome") as welcome: + with mock.patch("newsletter.views.send_newsletter_welcome") as welcome: resp = self.client.post( reverse("subscribe"), {"email": "reader@example.com", "frequency": "WEEKLY"}, @@ -65,7 +65,7 @@ def test_subscribe_attempts_welcome_email_with_manage_link(self): welcome.assert_called_once_with(subscriber.email, subscriber.manage_token) def test_subscribe_survives_welcome_send_failure(self): - with mock.patch("events.views.send_newsletter_welcome", return_value=False): + with mock.patch("newsletter.views.send_newsletter_welcome", return_value=False): resp = self.client.post( reverse("subscribe"), {"email": "reader@example.com", "frequency": "WEEKLY"}, diff --git a/backendServer/events/tests/test_send_digest_command_db.py b/backendServer/newsletter/tests/test_send_digest_command_db.py similarity index 85% rename from backendServer/events/tests/test_send_digest_command_db.py rename to backendServer/newsletter/tests/test_send_digest_command_db.py index 177a797..a82c82a 100644 --- a/backendServer/events/tests/test_send_digest_command_db.py +++ b/backendServer/newsletter/tests/test_send_digest_command_db.py @@ -10,7 +10,7 @@ def test_command_enqueues_fan_out_task(self): # The command was changed to enqueue the fan-out task rather than send # inline — assert it calls .delay(), not the old inline path. with mock.patch( - "events.management.commands.send_weekly_digest.fan_out_weekly_digest.delay" + "newsletter.management.commands.send_weekly_digest.fan_out_weekly_digest.delay" ) as delay: delay.return_value = mock.Mock(id="task-xyz") call_command("send_weekly_digest") diff --git a/backendServer/newsletter/urls.py b/backendServer/newsletter/urls.py new file mode 100644 index 0000000..4596ca3 --- /dev/null +++ b/backendServer/newsletter/urls.py @@ -0,0 +1,8 @@ +from django.urls import path + +from . import views + +urlpatterns = [ + path("newsletter/subscribe", views.subscribe, name="subscribe"), + path("newsletter/manage", views.newsletter_manage, name="newsletter-manage"), +] diff --git a/backendServer/newsletter/views.py b/backendServer/newsletter/views.py new file mode 100644 index 0000000..90c4636 --- /dev/null +++ b/backendServer/newsletter/views.py @@ -0,0 +1,72 @@ +from django.core.exceptions import ValidationError +from rest_framework import status +from rest_framework.decorators import api_view +from rest_framework.response import Response + +from newsletter.email_service import send_newsletter_welcome + +from .models import NewsletterSubscriber + + +@api_view(["POST"]) +def subscribe(request): + email = request.data.get("email", "").strip().lower() + frequency = request.data.get("frequency", "WEEKLY").upper() + + if not email: + return Response({"error": "email is required"}, status=status.HTTP_400_BAD_REQUEST) + + if frequency not in ("WEEKLY", "MONTHLY"): + return Response( + {"error": "frequency must be WEEKLY or MONTHLY"}, status=status.HTTP_400_BAD_REQUEST + ) + + subscriber, created = NewsletterSubscriber.objects.update_or_create( + email=email, + defaults={"frequency": frequency, "is_active": True}, + ) + + send_newsletter_welcome(subscriber.email, subscriber.manage_token) + + return Response( + {"email": subscriber.email, "frequency": subscriber.frequency}, + status=status.HTTP_201_CREATED if created else status.HTTP_200_OK, + ) + + +@api_view(["GET", "PATCH"]) +def newsletter_manage(request): + token = request.query_params.get("token") + + subscriber = None + if token: + try: + subscriber = NewsletterSubscriber.objects.filter(manage_token=token).first() + except ValidationError: + subscriber = None + + if subscriber is None: + return Response({"error": "Unknown or invalid token."}, status=status.HTTP_404_NOT_FOUND) + + if request.method == "PATCH": + frequency = (request.data.get("frequency") or "").upper() + if frequency not in ("WEEKLY", "MONTHLY", "NEVER"): + return Response( + {"error": "frequency must be WEEKLY, MONTHLY, or NEVER"}, + status=status.HTTP_400_BAD_REQUEST, + ) + + if frequency == "NEVER": + subscriber.is_active = False + else: + subscriber.frequency = frequency + subscriber.is_active = True + subscriber.save() + + return Response( + { + "email": subscriber.email, + "frequency": subscriber.frequency, + "is_active": subscriber.is_active, + } + ) diff --git a/deploy/healthcheck.service b/deploy/healthcheck.service index 41d75d8..72c8fd9 100644 --- a/deploy/healthcheck.service +++ b/deploy/healthcheck.service @@ -2,20 +2,26 @@ # then: sudo systemctl daemon-reload && sudo systemctl enable --now healthcheck.timer # (enable the .timer, not this .service — see healthcheck.timer) # -# Runs deploy/healthcheck.sh (system checks + `manage.py healthcheck`, which -# includes beat schedule freshness) on a systemd timer, independent of Celery -# beat. This is intentional: beat is one of the things being watched, so a -# beat-driven schedule would go silent in exactly the failure mode this exists -# to catch (see the 2026-07-21 -> 2026-07-29 outage, where beat stopped firing -# for 5 weeks and nothing noticed). +# Runs deploy/healthcheck.sh (RAM/disk + `docker compose ps` / `docker +# inspect` liveness for every long-running compose service + `manage.py +# healthcheck`, now executed via `docker compose exec` inside the backend +# container, which includes beat schedule freshness) on a systemd timer, +# independent of Celery beat AND independent of the compose stack itself. +# This is intentional: beat (and the rest of the app stack) is one of the +# things being watched, so scheduling this through beat — or replacing it +# with a Docker Compose `healthcheck:` block on a service — would go silent +# in exactly the failure mode this exists to catch (see the 2026-07-21 -> +# 2026-07-29 outage, where beat stopped firing for 5 weeks and nothing +# noticed). A host-level systemd timer is the one thing that keeps firing +# even when every container in docker-compose.yml is down. # -# Type=oneshot, not a long-lived unit: /snap/bin/uv is fine here even though -# sibling services are moving off it. The snap per-user-cgroup teardown that -# kills long-lived `celery worker`/`beat` processes on SSH logout (see -# celery.service) only bites a process that's still running when the session -# tears down — a oneshot invocation that starts, runs to completion in -# seconds, and exits does not outlive any single login session. Don't "fix" -# this to a venv binary path; it isn't the same bug. +# Type=oneshot, not a long-lived unit: the script only shells out to `docker +# compose ps` / `docker inspect` / `docker compose exec -T` and returns — it +# never holds a long-lived process of its own open, so there's nothing here +# for the snap per-user-cgroup teardown (see the old celery.service note) +# to kill on SSH logout. `docker` itself still needs to be reachable by +# `User=ubuntu` (i.e. ubuntu must be in the `docker` group on the VM) — +# that's a host provisioning step, not something this unit configures. [Unit] Description=The Commons VM health check (one-shot) @@ -25,8 +31,6 @@ Type=oneshot User=ubuntu Group=ubuntu WorkingDirectory=/home/ubuntu/thecommons -EnvironmentFile=/home/ubuntu/thecommons/backendServer/.env -Environment=UV_BIN=/snap/bin/uv ExecStart=/usr/bin/env bash /home/ubuntu/thecommons/deploy/healthcheck.sh --no-color # Non-zero exit on FAIL (see healthcheck.sh summary logic) lands here as a # failed unit — `systemctl --failed` and `journalctl -u healthcheck` are the diff --git a/deploy/healthcheck.sh b/deploy/healthcheck.sh index 849dc8f..d3a60c9 100755 --- a/deploy/healthcheck.sh +++ b/deploy/healthcheck.sh @@ -2,24 +2,28 @@ # # The Commons — VM health check. # -# A single, scannable report of the box's health: RAM/disk, every systemd unit, -# and (via the Django `healthcheck` command) Redis, Postgres, the Celery worker, -# and the beat schedule. Run it on the VM: +# A single, scannable report of the box's health: RAM/disk, every long-running +# compose service (T6/T7 moved the app stack into containers — see +# docker-compose.yml), and (via the Django `healthcheck` command, now run +# *inside* the backend container) Redis, Postgres, the Celery worker, and the +# beat schedule. Run it on the VM: # # bash deploy/healthcheck.sh # bash deploy/healthcheck.sh --no-color | tee health.log # -# System-level checks (RAM/disk/systemd/cron) are done here in bash; app-level -# checks are delegated to `manage.py healthcheck`, whose STATUS|name|detail lines -# are colorized below. Exits non-zero if any critical check fails so it can feed -# monitoring — including a stale/never-run beat schedule, which is now a FAIL, -# not a WARN (a dead scheduler is an outage, not a suggestion). +# System-level checks (RAM/disk/containers/cron) are done here in bash; +# app-level checks are delegated to `manage.py healthcheck`, whose +# STATUS|name|detail lines are colorized below. Exits non-zero if any +# critical check fails so it can feed monitoring — including a stale/never-run +# beat schedule, which is a FAIL, not a WARN (a dead scheduler is an outage, +# not a suggestion). # # Tunables (env vars, with defaults): -# RAM_WARN=80 RAM_FAIL=95 # % memory used -# DISK_WARN=80 DISK_FAIL=95 # % of / used -# UV_BIN=uv # path to uv (VM: /snap/bin/uv) -# CELERY_TIMEOUT=1.0 # seconds to wait for a worker ping +# RAM_WARN=80 RAM_FAIL=95 # % memory used +# DISK_WARN=80 DISK_FAIL=95 # % of / used +# CELERY_TIMEOUT=1.0 # seconds to wait for a worker ping +# RESTART_WARN=3 # container RestartCount that trips a WARN +# COMPOSE_FILE=docker-compose.yml set -uo pipefail # ── config ─────────────────────────────────────────────────────────────────── @@ -27,15 +31,19 @@ RAM_WARN="${RAM_WARN:-80}" RAM_FAIL="${RAM_FAIL:-95}" DISK_WARN="${DISK_WARN:-80}" DISK_FAIL="${DISK_FAIL:-95}" -UV_BIN="${UV_BIN:-uv}" CELERY_TIMEOUT="${CELERY_TIMEOUT:-1.0}" - -SERVICES=(redis-server celery celerybeat gunicorn nextjs broadcast-worker scrape-worker) +RESTART_WARN="${RESTART_WARN:-3}" +COMPOSE_FILE="${COMPOSE_FILE:-docker-compose.yml}" + +# The long-running services from docker-compose.yml. `migrate` and +# `broadcast-spa-build` are deliberately excluded: both are one-shot/build-only +# (restart: "no", exit 0 by design — see their comments in docker-compose.yml) +# and would misreport as dead services if checked here. +SERVICES=(redis backend celery celerybeat broadcast-worker scrape-worker nextjs nginx) LEGACY_CRON='manage.py (ingest_events|send_weekly_digest)' SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(dirname "$SCRIPT_DIR")" -BACKEND="$REPO_ROOT/backendServer" # ── flags ──────────────────────────────────────────────────────────────────── USE_COLOR=auto @@ -105,19 +113,67 @@ else report WARN "disk" "could not read df -P /" fi -# ── systemd units ──────────────────────────────────────────────────────────── +# ── compose services ───────────────────────────────────────────────────────── +# Container-aware replacement for the old `systemctl is-active` loop. Always +# pass -f explicitly and run from REPO_ROOT: a bare `docker compose` (no -f) +# auto-loads docker-compose.override.yml, which swaps in local-dev-only +# config and must never be what a prod healthcheck inspects. section "Services" -if command -v systemctl >/dev/null 2>&1; then +DOCKER_OK=1 +if ! command -v docker >/dev/null 2>&1; then + report WARN "docker" "docker not available (not a VM with the stack deployed?)" + DOCKER_OK=0 +elif ! docker compose version >/dev/null 2>&1; then + report WARN "docker" "docker compose plugin not available" + DOCKER_OK=0 +fi + +BACKEND_UP=0 +if [ "$DOCKER_OK" -eq 1 ]; then + pushd "$REPO_ROOT" >/dev/null for svc in "${SERVICES[@]}"; do - state="$(systemctl is-active "$svc" 2>/dev/null || true)" - if [ "$state" = active ]; then - report OK "$svc" "active" + # --all so a stopped/crashed container is still found (a bare `ps -q` + # only lists running ones, which would collapse "never started" and + # "exited" into the same unhelpful "no container" report). + cid="$(docker compose -f "$COMPOSE_FILE" ps --all -q "$svc" 2>/dev/null | head -n1)" + if [ -z "$cid" ]; then + report FAIL "$svc" "no container found" + continue + fi + + # {{if .State.Health}} guards services with no HEALTHCHECK (only + # redis defines one) — asking for .State.Health.Status on those + # would error the template instead of returning empty. + inspect="$(docker inspect --format \ + '{{.State.Status}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}|{{.RestartCount}}' \ + "$cid" 2>/dev/null || true)" + status="${inspect%%|*}" + rest="${inspect#*|}" + health="${rest%%|*}" + restarts="${rest#*|}" + + if [ -z "$status" ]; then + report FAIL "$svc" "docker inspect failed for $cid" + elif [ "$status" = running ]; then + [ "$svc" = backend ] && BACKEND_UP=1 + if [ "$health" = unhealthy ]; then + report FAIL "$svc" "running but unhealthy (restarts=$restarts)" + elif [ "${restarts:-0}" -ge "$RESTART_WARN" ]; then + report WARN "$svc" "running, but restarted $restarts times (possible restart-loop)" + elif [ "$health" != none ]; then + report OK "$svc" "running, health=$health" + else + report OK "$svc" "running" + fi + elif [ "$status" = restarting ]; then + # Caught mid-crash-loop: docker is actively bouncing this + # container, distinct from a container that's cleanly exited. + report FAIL "$svc" "restarting (crash-looping, restarts=$restarts)" else - report FAIL "$svc" "${state:-unknown}" + report FAIL "$svc" "$status (restarts=$restarts)" fi done -else - report WARN "systemd" "systemctl not available (not a VM?)" + popd >/dev/null fi # ── legacy OS cron (must be gone — beat owns these now) ─────────────────────── @@ -133,17 +189,22 @@ else report OK "legacy-cron" "no crontab on this host" fi -# ── app-level checks (Django command) ──────────────────────────────────────── +# ── app-level checks (Django command, now run inside the backend container) ── +# `exec` needs a running container; DOCKER_OK/BACKEND_UP come from the +# Services loop above, so a down stack reports one clear FAIL here instead of +# an ambiguous `docker compose exec` error. section "Application" app_out="" -if [ -d "$BACKEND" ]; then - # Mirror the systemd units: run from the backend dir with .env loaded. - pushd "$BACKEND" >/dev/null - if [ -f .env ]; then set -a; . ./.env; set +a; fi - app_out="$("$UV_BIN" run python manage.py healthcheck --require-prod --celery-timeout "$CELERY_TIMEOUT" 2>/dev/null || true)" - popd >/dev/null +if [ "$DOCKER_OK" -ne 1 ]; then + report FAIL "app-checks" "docker/compose not available — cannot exec into backend" +elif [ "$BACKEND_UP" -ne 1 ]; then + report FAIL "app-checks" "backend container is not running — cannot run manage.py healthcheck" else - report FAIL "app-checks" "backendServer not found at $BACKEND" + pushd "$REPO_ROOT" >/dev/null + # -T: no TTY allocation, required for a non-interactive/systemd context. + app_out="$(docker compose -f "$COMPOSE_FILE" exec -T backend \ + python manage.py healthcheck --require-prod --celery-timeout "$CELERY_TIMEOUT" 2>/dev/null || true)" + popd >/dev/null fi if [ -n "$app_out" ]; then @@ -151,8 +212,8 @@ if [ -n "$app_out" ]; then [ -z "$status" ] && continue report "$status" "$name" "$detail" done <<< "$app_out" -elif [ -d "$BACKEND" ]; then - report FAIL "app-checks" "manage.py healthcheck produced no output (uv/Django error?)" +elif [ "$DOCKER_OK" -eq 1 ] && [ "$BACKEND_UP" -eq 1 ]; then + report FAIL "app-checks" "manage.py healthcheck produced no output (container exec error?)" fi # ── summary ────────────────────────────────────────────────────────────────── diff --git a/deploy/healthcheck.timer b/deploy/healthcheck.timer index 8319038..94a4c7c 100644 --- a/deploy/healthcheck.timer +++ b/deploy/healthcheck.timer @@ -1,11 +1,20 @@ # Runs healthcheck.service hourly — copy to /etc/systemd/system/healthcheck.timer # then: sudo systemctl daemon-reload && sudo systemctl enable --now healthcheck.timer # -# Deliberately NOT a django-celery-beat entry: beat itself is one of the -# things this checks, so scheduling it through beat would mean a dead beat -# silences its own alarm. A systemd timer runs independent of the app stack — -# it fires even if Postgres, Redis, and every Celery process are down, so -# those failures actually surface as FAILs instead of just not running. +# Deliberately a HOST systemd timer, not: +# - a django-celery-beat entry: beat itself is one of the things this +# checks (via manage.py healthcheck's PeriodicTask freshness probe), so +# scheduling the check through beat would mean a dead beat silences its +# own alarm. +# - a Docker Compose `healthcheck:` block on one of the app services: that +# only reports the liveness of the one container it's attached to, and +# goes silent the moment that container (or the whole compose project) +# stops existing — exactly the "everything is down" case this needs to +# catch. A container can't watch the engine it runs on; the host can. +# A systemd timer on the VM runs independent of the compose stack — it fires +# even if Postgres (Neon, external), Redis, and every container in +# docker-compose.yml are down, so those failures actually surface as FAILs +# instead of just not running. [Unit] Description=Run The Commons health check hourly diff --git a/deploy/nginx/Dockerfile b/deploy/nginx/Dockerfile new file mode 100644 index 0000000..e27dda7 --- /dev/null +++ b/deploy/nginx/Dockerfile @@ -0,0 +1,37 @@ +# syntax=docker/dockerfile:1 +# +# nginx image for The Commons — fronts nextjs (3000) and backend/gunicorn +# (8000), and directly serves two BAKED build artifacts produced by other +# services' Dockerfiles, plus the media bind mount. +# +# This Dockerfile is never built standalone: `docker build -f +# deploy/nginx/Dockerfile deploy/nginx` would fail, because `backend-static` +# and `broadcast-spa` below are Buildx *named build contexts* wired up in +# docker-compose.yml's `nginx.build.additional_contexts`, bound to the +# `backend` and `broadcast-spa-build` compose services respectively (each +# building a specific stage of backendServer/Dockerfile and Dockerfile.frontend). +# Always build via `docker compose build nginx` (or `up --build`) — verified +# that compose resolves and builds those named-context services first even +# when only `nginx` is requested. +# +# backend-static -> backendServer/Dockerfile `app` stage +# /app/staticfiles_build/static (collectstatic output) +# broadcast-spa -> Dockerfile.frontend `broadcast-build` stage +# /app/dist (compiled broadcastWeb SPA) +# +# Baking these in (rather than a volume) is deliberate: a volume here would +# let a stale build survive a fresh deploy. Both artifacts are pure build +# output, never mutated at runtime. +FROM nginx:1.27-alpine + +COPY --from=backend-static /app/staticfiles_build/static /usr/share/nginx/html/static +COPY --from=broadcast-spa /app/dist /usr/share/nginx/html/broadcast + +# Replace nginx's stock default server block with ours — the stock +# default.conf listens on 80 with its own catch-all server, which would +# otherwise compete with the HTTP->HTTPS redirect block below. +COPY thecommons.conf /etc/nginx/conf.d/thecommons.conf +RUN rm -f /etc/nginx/conf.d/default.conf + +# nginx:1.27-alpine's own CMD/ENTRYPOINT (start nginx in the foreground) is +# unchanged — no override needed. diff --git a/deploy/nginx/thecommons.conf b/deploy/nginx/thecommons.conf new file mode 100644 index 0000000..da85ea0 --- /dev/null +++ b/deploy/nginx/thecommons.conf @@ -0,0 +1,158 @@ +# The Commons — nginx config. ONE file, many server blocks (repo +# convention — see the old DEPLOY.md §nginx / deploy/nginx-broadcast.conf.snippet, +# which this supersedes now that nginx runs containerized). +# +# Baked into the image at /etc/nginx/conf.d/thecommons.conf by +# deploy/nginx/Dockerfile. This is the PRODUCTION config — it requires the +# Cloudflare origin cert to be bind-mounted at container start (see +# docker-compose.yml's nginx volumes). It only actually runs when compose is +# invoked as `-f docker-compose.yml` alone (the VM). Plain `docker compose +# up` (local dev) auto-loads docker-compose.override.yml, which replaces +# this file's content at the same container path with an HTTP-only +# equivalent via an inline `configs:` block, since a dev machine has neither +# the cert nor real DNS for the *.thecommons.town hostnames below. +# +# TLS: Cloudflare Full-strict. Cert/key bind-mounted read-only — never +# baked into this or any other image layer. + +# Docker's embedded DNS server. Required because every proxy_pass below +# resolves its upstream through a *variable* rather than a literal hostname. +# +# Why bother: with a literal `proxy_pass http://nextjs:3000`, nginx resolves +# the name once at startup and REFUSES TO BOOT if it doesn't resolve +# ("[emerg] host not found in upstream"). Since this service is +# `restart: unless-stopped`, a nextjs container that is down or still +# starting when nginx restarts would take nginx down with it — and with it +# api.thecommons.town and broadcast.thecommons.town, which have nothing to +# do with nextjs. One crashed frontend becomes a total ingress outage. +# +# Resolving through a variable defers the lookup to request time, so nginx +# always boots and a dead upstream degrades to a 502 on just that one +# hostname. `valid=10s` re-resolves regularly, which also means a recreated +# container's new IP is picked up without an nginx reload. ipv6=off because +# the compose network is v4-only and the AAAA lookup just adds latency. +resolver 127.0.0.11 valid=10s ipv6=off; + +# ── HTTP -> HTTPS redirect, all hostnames ────────────────────────────────── +server { + listen 80; + server_name thecommons.town www.thecommons.town api.thecommons.town broadcast.thecommons.town auth.thecommons.town; + return 301 https://$host$request_uri; +} + +# ── www -> apex redirect ──────────────────────────────────────────────────── +server { + listen 443 ssl; + server_name www.thecommons.town; + + ssl_certificate /etc/ssl/cloudflare/thecommons.town.pem; + ssl_certificate_key /etc/ssl/cloudflare/thecommons.town.key; + + return 301 https://thecommons.town$request_uri; +} + +# ── thecommons.town -> Next.js ────────────────────────────────────────────── +server { + listen 443 ssl; + server_name thecommons.town; + + ssl_certificate /etc/ssl/cloudflare/thecommons.town.pem; + ssl_certificate_key /etc/ssl/cloudflare/thecommons.town.key; + + location / { + # Variable upstream — see the `resolver` note at the top of this file. + # No URI part after the port, so the original request URI is passed + # through unchanged. + set $upstream_nextjs nextjs; + proxy_pass http://$upstream_nextjs:3000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +# ── auth.thecommons.town -> Next.js (Better Auth portal) ──────────────────── +# Same upstream as the apex (one Next.js app serves both hostnames); the +# split exists so Better Auth owns a dedicated origin, which is what makes +# the `.thecommons.town` cookie domain work across subdomains (suite 37 — +# docs/runbook-auth-cutover.md). +# +# This block is NOT optional and must never be dropped: `backendServer/.env` +# points BETTER_AUTH_JWKS_URL/BETTER_AUTH_ISSUER at this hostname, so every +# broadcast JWT verification fetches https://auth.thecommons.town/api/auth/jwks +# through it. Without this server block, requests to auth.thecommons.town fall +# through to the first `listen 443 ssl` block (the www -> apex redirect), which +# 301s the JWKS fetch and 403s every tier-2 broadcast request — the same +# symptom as the 2026-07-20 JWKS outage, from a different cause. +server { + listen 443 ssl; + server_name auth.thecommons.town; + + ssl_certificate /etc/ssl/cloudflare/thecommons.town.pem; + ssl_certificate_key /etc/ssl/cloudflare/thecommons.town.key; + + location / { + # Variable upstream — see the `resolver` note at the top of this file. + set $upstream_nextjs_auth nextjs; + proxy_pass http://$upstream_nextjs_auth:3000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +# ── api.thecommons.town -> Django (gunicorn) ──────────────────────────────── +server { + listen 443 ssl; + server_name api.thecommons.town; + + ssl_certificate /etc/ssl/cloudflare/thecommons.town.pem; + ssl_certificate_key /etc/ssl/cloudflare/thecommons.town.key; + + # Event image uploads (T7) pass through this proxy to gunicorn. + client_max_body_size 20m; + + # Baked collectstatic output — never proxied to gunicorn; Django's + # staticfiles app isn't even asked. + location /static/ { + alias /usr/share/nginx/html/static/; + } + + # Client-uploaded event images (MEDIA_ROOT) — bind-mounted real host + # data, served by nginx directly. NEVER proxied to gunicorn in prod. + # Kept indefinitely; no pruning job (see docker-compose.yml volumes). + location /media/ { + alias /var/www/media/; + } + + location / { + # Variable upstream — see the `resolver` note at the top of this file. + set $upstream_backend backend; + proxy_pass http://$upstream_backend:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +# ── broadcast.thecommons.town -> static SPA ───────────────────────────────── +server { + listen 443 ssl; + server_name broadcast.thecommons.town; + + ssl_certificate /etc/ssl/cloudflare/thecommons.town.pem; + ssl_certificate_key /etc/ssl/cloudflare/thecommons.town.key; + + # Baked SPA build — not a bind mount (a volume here would let a stale + # build survive a deploy). No Node process behind this. + root /usr/share/nginx/html/broadcast; + index index.html; + + location / { + # Client-side routing: unknown paths fall back to index.html. + try_files $uri /index.html; + } +} diff --git a/docker-compose.override.yml b/docker-compose.override.yml new file mode 100644 index 0000000..3013d15 --- /dev/null +++ b/docker-compose.override.yml @@ -0,0 +1,161 @@ +# Local-dev override — auto-loaded by plain `docker compose up`/`build` +# alongside docker-compose.yml (Compose's default two-file merge). The VM +# must NEVER pick this up: prod invocations always pass `-f +# docker-compose.yml` explicitly, e.g. `docker compose -f docker-compose.yml +# up -d --build`, which skips this file entirely. +# +# Why this file exists: docker-compose.yml's nginx bakes in a config +# (deploy/nginx/thecommons.conf) that hard-requires the Cloudflare origin +# cert at /etc/ssl/cloudflare/thecommons.town.{pem,key} — that cert doesn't +# exist on a dev machine, and nginx refuses to start at all with `listen 443 +# ssl` blocks pointing at missing files. Rather than maintaining a second +# tracked nginx.conf (this ticket's scope is one thecommons.conf, matching +# the "one file" convention), this override replaces JUST that one file's +# *content* at the same in-container path via an inline `configs:` entry — +# an HTTP-only equivalent using `*.localhost` hostnames (which resolve to +# 127.0.0.1 with no /etc/hosts edits needed, per RFC 6761) instead of the +# real *.thecommons.town domains. No cert, no SSL directives, nothing to +# configure. +# +# It also remaps every bind-mounted host path from the Linux VM's +# /home/ubuntu/... layout to plain repo-relative directories, since those +# absolute paths don't exist (and shouldn't be assumed) on a dev machine. +# Docker creates them automatically on first `up` if missing. + +# Local-dev environment shared by every Django/Celery service. +# +# Two things here are NOT cosmetic: +# +# 1. DJANGO_ENV=dev. The base compose file sets `prod` on all of these, and +# prod.py does `os.environ["DJANGO_ALLOWED_HOSTS"]` (prod.py:16) — a hard +# KeyError. That key isn't in backendServer/.env, so without this override +# every backend container would crash-loop on startup locally. `dev` is an +# explicit valid value rather than unsetting the var, because +# backend/settings/__init__.py treats an unrecognized DJANGO_ENV as a hard +# error and only unset/empty as dev — being explicit documents the intent. +# +# 2. REDIS_URL / REDIS_CACHE_URL. base.py:146,193 default these to +# redis://localhost:6379/{0,1}. Inside a container "localhost" is that +# container itself, so the default silently points Celery at nothing. +# They must name the compose service (`redis`). Keeps the DB 0 = broker / +# DB 1 = cache split. No password locally (the redis service starts +# unauthenticated when REDIS_PASSWORD is unset). +x-local-django-env: &local-django-env + DJANGO_ENV: dev + REDIS_URL: redis://redis:6379/0 + REDIS_CACHE_URL: redis://redis:6379/1 + MEDIA_ROOT: /home/ubuntu/broadcast/media + BROADCAST_SCREENSHOT_DIR: /home/ubuntu/broadcast/screenshots + BROADCAST_DOWNLOAD_DIR: /home/ubuntu/broadcast/downloads + +services: + # Only port 80 locally — no TLS, so no reason to publish 443. + # (`!override` replaces the base list outright instead of merging into it.) + nginx: + ports: !override + - "80:80" + # `!override` (not a plain list) drops the base's Cloudflare cert bind + # mount entirely — it's not just unused here, the host path it points at + # (/etc/ssl/cloudflare) usually doesn't exist on a dev machine at all. + volumes: !override + - ./.local-dev/media:/var/www/media:ro + configs: + - source: nginx_dev_conf + target: /etc/nginx/conf.d/thecommons.conf + + backend: + environment: *local-django-env + volumes: + - ./.local-dev/media:/home/ubuntu/broadcast/media + - ./.local-dev/screenshots:/home/ubuntu/broadcast/screenshots + - ./.local-dev/downloads:/home/ubuntu/broadcast/downloads + + celery: + environment: *local-django-env + + celerybeat: + environment: *local-django-env + + scrape-worker: + environment: *local-django-env + + broadcast-worker: + environment: *local-django-env + volumes: + - ./.local-dev/screenshots:/home/ubuntu/broadcast/screenshots + - ./.local-dev/downloads:/home/ubuntu/broadcast/downloads + + migrate: + environment: *local-django-env + volumes: + - ./.local-dev/backups:/home/ubuntu/backups + +# Inline config content (Compose Spec top-level `configs:`, written directly +# into this YAML file — no second nginx.conf file needed on disk). Mirrors +# deploy/nginx/thecommons.conf's routes 1:1 (apex/api/broadcast), minus TLS. +configs: + nginx_dev_conf: + content: | + # Variable upstreams + Docker's embedded resolver, mirroring + # deploy/nginx/thecommons.conf. Without this nginx resolves upstreams at + # startup and refuses to boot when one is missing — which locally is the + # common case (bringing up only part of the stack), not an edge case. + resolver 127.0.0.11 valid=10s ipv6=off; + + server { + listen 80; + server_name localhost; + + location / { + set $$upstream_nextjs nextjs; + proxy_pass http://$$upstream_nextjs:3000; + proxy_set_header Host $$host; + proxy_set_header X-Real-IP $$remote_addr; + proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for; + } + } + + server { + listen 80; + server_name api.localhost; + + client_max_body_size 20m; + + location /static/ { + alias /usr/share/nginx/html/static/; + } + + location /media/ { + alias /var/www/media/; + } + + location / { + set $$upstream_backend backend; + proxy_pass http://$$upstream_backend:8000; + # Forward "localhost", NOT $$host. dev.py:16 hardcodes + # ALLOWED_HOSTS = ["localhost", "127.0.0.1"] with no env + # override, so passing through "api.localhost" makes Django + # reject every request with DisallowedHost (HTTP 400) — the + # same failure shape as the June 2026 "no events" outage. + # Rewriting the header here keeps this a local-dev nginx + # concern instead of editing application settings. The prod + # config (deploy/nginx/thecommons.conf) correctly passes + # $$host, because prod.py's ALLOWED_HOSTS does include + # api.thecommons.town. + proxy_set_header Host localhost; + proxy_set_header X-Real-IP $$remote_addr; + proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for; + } + } + + server { + listen 80; + server_name broadcast.localhost; + + root /usr/share/nginx/html/broadcast; + index index.html; + + location / { + try_files $$uri /index.html; + } + } diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..871aa6f --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,318 @@ +# The Commons — production compose file (T6/T7 of the Dockerization suite). +# +# Replaces the systemd unit graph in deploy/*.service (gunicorn, celery, +# celerybeat, broadcast-worker, scrape-worker) plus the hand-maintained +# nginx config described in DEPLOY.md §nginx. Postgres is NOT here — +# DATABASE_URL always points at Neon (external), in every environment. +# +# Usage: +# VM (prod): docker compose -f docker-compose.yml up -d --build +# local dev: docker compose up --build +# (docker-compose.override.yml is auto-loaded and swaps +# in plain-HTTP nginx + local bind-mount paths — see +# that file's header for the full explanation) +# +# Build-time secrets gotcha: `nextjs` and `broadcast-spa-build` below bake +# NEXT_PUBLIC_*/VITE_* values into their JS bundles via build args. Compose +# variable interpolation (the ${VAR:-default} syntax used throughout this +# file) only reads the shell environment or a `.env` file living next to +# this compose file — it CANNOT read theCommonsWeb/.env.local or +# broadcastWeb/.env directly (those are consumed by env_file: at container +# *runtime*, a completely different mechanism). Every build arg below has a +# safe CI-style placeholder default so `docker compose build` always +# succeeds with zero setup, but a REAL prod build needs the real values +# exported into the shell (or placed in a root .env) first, e.g.: +# set -a; source theCommonsWeb/.env.local; source broadcastWeb/.env; set +a +# docker compose build nextjs broadcast-spa-build +# CI already greps the compiled broadcast bundle for a thecommons.town +# origin (.github/workflows/ci.yml) as a guard against exactly this being +# forgotten. + +x-logging: &default-logging + # journalctl dies with systemd — plain json-file with rotation is the + # replacement. `docker logs ` still works identically. + driver: json-file + options: + max-size: "10m" + max-file: "3" + +services: + # ── Redis — Celery broker/results (DB 0) + read-endpoint cache (DB 1) ──── + # Was the system-installed redis-server.service. + # + # ⚠️ CUTOVER REQUIREMENT — prod's backendServer/.env MUST change. + # REDIS_URL and REDIS_CACHE_URL currently point at 127.0.0.1/localhost + # (that's also base.py's default: base.py:146,193). Inside a container + # "localhost" is the container itself, so leaving them as-is doesn't error — + # Celery just silently connects to nothing and no task ever runs. That is + # the same shape of failure as the 2026-07-21 outage (async stack dead, + # everything else green), so treat it as a release blocker, not a nit: + # + # REDIS_URL=redis://:@redis:6379/0 + # REDIS_CACHE_URL=redis://:@redis:6379/1 + # + # The password must match REDIS_PASSWORD below. `manage.py healthcheck` + # pings DB 0 and round-trips DB 1, so the hourly watchdog catches this if it + # is missed — but it should be done as part of the cutover, not discovered. + # (Not set here in `environment:` because that would override the whole URL + # and drop the password embedded in it.) + redis: + image: redis:7-alpine + restart: unless-stopped + env_file: + - backendServer/.env + # REDIS_PASSWORD is a container-only knob — Django/Celery never read it + # directly, only REDIS_URL/REDIS_CACHE_URL (which already embed a + # password in their `:@` segment). It must be set in + # backendServer/.env to the SAME password embedded in those URLs, or + # this container's --requirepass and the app's connection string drift + # out of sync. Left unset (local dev — see docker-compose.override.yml + # and .env.example's unauthenticated `redis://localhost:6379/0`), redis + # starts with no auth. + command: > + sh -c ' + if [ -n "$$REDIS_PASSWORD" ]; then + exec redis-server --requirepass "$$REDIS_PASSWORD" --appendonly yes + else + exec redis-server --appendonly yes + fi' + healthcheck: + test: ["CMD-SHELL", "redis-cli -a \"$$REDIS_PASSWORD\" ping 2>/dev/null | grep -q PONG || redis-cli ping | grep -q PONG"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 5s + volumes: + - redis-data:/data + logging: *default-logging + + # ── One-shot migration — replaces the manual `manage.py migrate` deploy + # step. Also seeds django_celery_beat's schedule tables, so celerybeat + # needs this to have run at least once. Not a Celery service, so it has + # no reason to wait on redis. + migrate: + build: + context: backendServer + dockerfile: Dockerfile + target: app + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + command: ["python", "manage.py", "migrate", "--noinput"] + restart: "no" + volumes: + # Real, persistent host storage for the pre-migrate pg_dump the CI + # pipeline takes today (.github/workflows/ci.yml) — an ephemeral + # container filesystem here would make that dump vanish on the next + # `docker compose down`, silently turning the safety net into a + # no-op. `docker compose run --rm migrate` doesn't currently invoke + # pg_dump itself (that's still the deploy pipeline's job, out of + # scope for this ticket) — this mount just makes sure the path is + # real and ready for whichever process writes to it. + - ${BACKUPS_HOST:-/home/ubuntu/backups}:/home/ubuntu/backups + logging: *default-logging + + # ── Django / gunicorn ──────────────────────────────────────────────────── + backend: + build: + context: backendServer + dockerfile: Dockerfile + target: app + restart: unless-stopped + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + depends_on: + redis: + condition: service_healthy + migrate: + condition: service_completed_successfully + # Gunicorn binds TCP 0.0.0.0:8000 now (no more unix socket) — reachable + # only from other containers on the compose network, never the host. + expose: + - "8000" + volumes: + # Same absolute path on both sides on purpose: MEDIA_ROOT / + # BROADCAST_SCREENSHOT_DIR / BROADCAST_DOWNLOAD_DIR in + # backendServer/.env are real host paths already in prod's .env + # today (DEPLOY.md); mounting the container at the identical path + # means that .env file needs zero changes for the cutover. The host + # directories must exist and be writable by uid 1000 (the image's + # non-root `app` user — matches the `ubuntu` user's usual uid 1000 + # on the Oracle VM) before first run. + - ${MEDIA_ROOT_HOST:-/home/ubuntu/broadcast/media}:/home/ubuntu/broadcast/media + - ${BROADCAST_SCREENSHOT_HOST:-/home/ubuntu/broadcast/screenshots}:/home/ubuntu/broadcast/screenshots + - ${BROADCAST_DOWNLOAD_HOST:-/home/ubuntu/broadcast/downloads}:/home/ubuntu/broadcast/downloads + logging: *default-logging + + # ── Default Celery worker (digest emails etc. — everything not routed to + # the broadcast/scrape queues) ─────────────────────────────────────────── + celery: + build: + context: backendServer + dockerfile: Dockerfile + target: app + restart: unless-stopped + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + command: ["celery", "-A", "backend", "worker", "-n", "commons-default@%h", "-l", "info", "--concurrency=2"] + depends_on: + redis: + condition: service_healthy + migrate: + condition: service_completed_successfully + mem_limit: 1g + logging: *default-logging + + # ── Celery beat — exactly one process (Celery best practice; do not + # scale this service). DatabaseScheduler keeps the schedule in Postgres, + # seeded by `migrate` above. ───────────────────────────────────────────── + celerybeat: + build: + context: backendServer + dockerfile: Dockerfile + target: app + restart: unless-stopped + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + command: ["celery", "-A", "backend", "beat", "-l", "info"] + depends_on: + redis: + condition: service_healthy + migrate: + condition: service_completed_successfully + logging: *default-logging + + # ── Broadcast Playwright worker — drains the `broadcast` queue. -c 1 is + # mandatory, not tuning: recover_orphans() assumes a single worker, so + # concurrency > 1 would let a live queue-drain race orphan recovery. ──── + broadcast-worker: + build: + context: backendServer + dockerfile: Dockerfile + target: playwright + restart: unless-stopped + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + command: ["celery", "-A", "backend", "worker", "-Q", "broadcast", "-n", "commons-broadcast@%h", "-c", "1", "-l", "info"] + depends_on: + redis: + condition: service_healthy + migrate: + condition: service_completed_successfully + mem_limit: 2g + volumes: + - ${BROADCAST_SCREENSHOT_HOST:-/home/ubuntu/broadcast/screenshots}:/home/ubuntu/broadcast/screenshots + - ${BROADCAST_DOWNLOAD_HOST:-/home/ubuntu/broadcast/downloads}:/home/ubuntu/broadcast/downloads + logging: *default-logging + + # ── Ingestion scrape worker — drains the `scrape` queue. Same -c 1 + # rationale as broadcast-worker's queue-drain note above; here it's just + # to keep headless-Chromium memory off the default worker. ────────────── + scrape-worker: + build: + context: backendServer + dockerfile: Dockerfile + target: playwright + restart: unless-stopped + env_file: + - backendServer/.env + environment: + DJANGO_ENV: prod + command: ["celery", "-A", "backend", "worker", "-Q", "scrape", "-n", "commons-scrape@%h", "-c", "1", "-l", "info"] + depends_on: + redis: + condition: service_healthy + migrate: + condition: service_completed_successfully + mem_limit: 2g + logging: *default-logging + + # ── Next.js (theCommonsWeb) ────────────────────────────────────────────── + nextjs: + build: + context: . + dockerfile: Dockerfile.frontend + target: commons-runtime + args: + # See the file header re: why these come from shell/root-.env + # interpolation rather than theCommonsWeb/.env.local directly. + DATABASE_URL: ${NEXTJS_BUILD_DATABASE_URL:-postgres://build:build@localhost:5432/build} + BETTER_AUTH_SECRET: ${NEXTJS_BUILD_BETTER_AUTH_SECRET:-build-only-placeholder-secret} + BETTER_AUTH_URL: ${NEXTJS_BUILD_BETTER_AUTH_URL:-https://auth.thecommons.town} + NEXT_PUBLIC_BETTER_AUTH_URL: ${NEXTJS_BUILD_NEXT_PUBLIC_BETTER_AUTH_URL:-https://auth.thecommons.town} + NEXT_PUBLIC_API_BASE_URL: ${NEXTJS_BUILD_NEXT_PUBLIC_API_BASE_URL:-https://api.thecommons.town} + NEXT_PUBLIC_THE_COMMONS_API_KEY: ${NEXTJS_BUILD_NEXT_PUBLIC_THE_COMMONS_API_KEY:-build-placeholder} + restart: unless-stopped + env_file: + - theCommonsWeb/.env.local + # Standalone Next server on 3000 — internal only, nginx fronts it. + expose: + - "3000" + logging: *default-logging + + # ── Build-only helper: produces the compiled broadcastWeb SPA so nginx + # can COPY --from it (see nginx's `additional_contexts` below). This + # stage (Dockerfile.frontend `broadcast-build`) has no CMD/EXPOSE and + # must never actually run — `command`/`restart: "no"` mirror the + # `migrate` one-shot pattern so a stray `docker compose up` (which starts + # every service by default) just runs a harmless no-op and exits 0 + # instead of crash-looping on a missing entrypoint. + broadcast-spa-build: + build: + context: . + dockerfile: Dockerfile.frontend + target: broadcast-build + args: + VITE_BROADCAST_API_BASE_URL: ${BROADCAST_BUILD_VITE_BROADCAST_API_BASE_URL:-https://api.thecommons.town} + VITE_BETTER_AUTH_URL: ${BROADCAST_BUILD_VITE_BETTER_AUTH_URL:-https://auth.thecommons.town} + VITE_BROADCAST_EXTENSION_ID: ${BROADCAST_BUILD_VITE_BROADCAST_EXTENSION_ID:-} + command: ["true"] + restart: "no" + + # ── nginx — the ONLY service publishing host ports. ───────────────────── + # Bakes in two build artifacts produced by sibling Dockerfiles via Buildx + # named build contexts bound to the compose services above (verified: + # `docker compose build` resolves `service:` contexts and builds + # them first even when only `nginx` is requested): + # backend-static -> backend's `app` stage: /app/staticfiles_build/static + # broadcast-spa -> broadcast-spa-build's stage: /app/dist + # See deploy/nginx/Dockerfile for the COPY --from= lines that consume + # these names. + nginx: + build: + context: deploy/nginx + dockerfile: Dockerfile + additional_contexts: + backend-static: "service:backend" + broadcast-spa: "service:broadcast-spa-build" + restart: unless-stopped + ports: + - "80:80" + - "443:443" + depends_on: + - backend + - nextjs + - broadcast-spa-build + volumes: + # Client-uploaded event images (MEDIA_ROOT) — nginx serves these + # directly, never through gunicorn. Read-only: nginx never writes here. + - ${MEDIA_ROOT_HOST:-/home/ubuntu/broadcast/media}:/var/www/media:ro + # Cloudflare Full-strict origin cert. Bind-mounted read-only — NEVER + # copied into an image layer. Doesn't exist on a dev machine, which is + # why docker-compose.override.yml replaces this whole service's nginx + # config (via an inline `configs:` entry) with an HTTP-only one that + # never references this path. + - ${CLOUDFLARE_CERT_DIR_HOST:-/etc/ssl/cloudflare}:/etc/ssl/cloudflare:ro + logging: *default-logging + +volumes: + redis-data: diff --git a/docs/admin-backend.md b/docs/admin-backend.md index d231290..fd2dbe0 100644 --- a/docs/admin-backend.md +++ b/docs/admin-backend.md @@ -31,7 +31,7 @@ The sidebar is organized into three sections, configured in `backend/settings.py | **Events** | Published Events | `/admin/events/event/` | | | Tags | `/admin/events/tag/` | | **Users** | Users | `/admin/auth/user/` | -| | User Profiles | `/admin/events/userprofile/` | +| | User Profiles | `/admin/accounts/userprofile/` | --- @@ -184,7 +184,7 @@ Standard Django user management. Use this to: - Reset passwords - Grant/revoke `is_staff` or `is_superuser` flags -### User Profiles — `/admin/events/userprofile/` +### User Profiles — `/admin/accounts/userprofile/` Extended profile data attached to each Django user. diff --git a/docs/adr/0001-containerization.md b/docs/adr/0001-containerization.md new file mode 100644 index 0000000..dfed651 --- /dev/null +++ b/docs/adr/0001-containerization.md @@ -0,0 +1,287 @@ +# ADR 0001: Containerization of The Commons + +## Status + +Accepted — 2026-08-01. This ADR records four decisions already made for the +Dockerization suite; later tickets in the suite (Dockerfiles, compose file, +CI changes, `DEPLOY.md` rewrite) build on these without re-litigating them. + +## Context + +The Commons runs today as a hand-provisioned Oracle Cloud VM (Ubuntu 24.04, +ARM64, 1 OCPU / 6 GB — `DEPLOY.md` §Facts) with seven long-lived processes +wired up as individual systemd units (`gunicorn`, `nextjs`, `redis-server`, +`celery`, `celerybeat`, `broadcast-worker`, `scrape-worker`) plus a hand-edited +nginx config. There is no container tooling anywhere in this repo today — no +Dockerfile, no compose file, not even a stray reference to "docker" in the +docs. Provisioning a second environment (or recovering this one from scratch) +means re-reading all of `DEPLOY.md` Part 1 by hand. + +Two incidents make the cost of that gap concrete: + +- The 2026-07-21 scheduler outage (`docs/prod-incident-2026-07-21-scheduler-outage.md`): + all four Celery units execed `/snap/bin/uv run celery …`, and snap's `uv` + spawns its child inside a transient scope under `user@1001.service` (the + *user* session manager, not the unit's own cgroup). A post-deploy SSH logout + tore down `user-1001.slice` and killed the whole async stack with + `status=0/SUCCESS` — a clean exit `Restart=on-failure` correctly declined to + restart. `gunicorn` and `nextjs`, which exec their binaries directly, never + went through snap and stayed up 68 days straight over the same window. The + fix in `deploy/*.service` today is `ExecStart=.venv/bin/celery …` plus + `loginctl enable-linger ubuntu` as defense-in-depth. +- The nginx config is one file, `/etc/nginx/sites-available/thecommons`, with + `server` blocks for all three subdomains hand-appended over several suites + (`DEPLOY.md` §nginx, `deploy/nginx-broadcast.conf.snippet`). There is no way + to diff, review, or roll back a change to it short of SSHing in. + +This ADR records the four load-bearing decisions for moving this stack into +containers, and the reasoning behind each, so a future engineer sees *why* +rather than re-deriving it from the compose file. + +--- + +## Decision 1 — nginx runs in a container, as the single ingress + +**Decision:** nginx moves into a container and becomes the sole entrypoint for +all three subdomains (`thecommons.town`, `api.thecommons.town`, +`broadcast.thecommons.town`), replacing the host-installed nginx. + +**Rationale:** + +- **Local/prod parity.** Today's nginx config exists only on the VM; nothing + about routing, TLS termination, or the static/media aliasing is exercised + locally. Containerizing it means the same image and (mostly) the same + config file run in both places — the current one-file-many-`server`-blocks + structure (`DEPLOY.md` §nginx, `deploy/nginx-broadcast.conf.snippet`) is + preserved, not restructured, since that convention is explicitly called out + in the repo as deliberate ("do NOT create a separate sites-available file"). +- **The Cloudflare origin cert is bind-mounted, never baked in.** TLS + termination happens with the Cloudflare origin cert at + `/etc/ssl/cloudflare/thecommons.town.{pem,key}` under Full (strict) mode + (`DEPLOY.md` §Facts, line 22). This must be a **read-only bind mount**, not + a `COPY` into an image layer — an image layer would (a) require a rebuild + and re-push on every cert rotation and (b) risk the private key ending up + in an image that could be pulled or inspected outside the VM. A bind mount + keeps the key exactly where it is today, filesystem-permissioned, outside + the image entirely. + +**Consequence — gunicorn moves from a unix socket to TCP.** Today gunicorn +listens on `unix:/run/gunicorn/gunicorn.sock` (`DEPLOY.md` §Services, line +518) with `RuntimeDirectory=gunicorn` creating the socket dir. A unix socket +can't cross a container boundary the way it's used today (nginx and gunicorn +sharing a host filesystem path), so nginx talks to the backend over the +compose network instead: `proxy_pass http://backend:8000;`. This is a strictly +weaker isolation posture than a filesystem-permissioned socket (any container +on the compose network can reach port 8000), but it's the standard pattern +for containerized nginx+app pairs and the compose network is not +host-reachable. + +**Cutover risk to flag explicitly:** the host nginx must be **stopped and +disabled** (`sudo systemctl disable --now nginx`) before the container nginx +binds 80/443 — two processes cannot bind the same ports. This is a +first-deploy-only step, but it is the one step in this whole suite where a +mistake produces an immediate, visible outage (container fails to bind, or +host nginx silently keeps serving the old config). The iptables ACCEPT rules +for 80/443 already in place (`DEPLOY.md` §Firewall) are unaffected — they act +on the port, not on which process holds it. + +--- + +## Decision 2 — Redis runs in a container + +**Decision:** Redis moves into a container (`redis:7-alpine`), replacing the +`apt install redis-server` + `/etc/redis/redis.conf` setup in `DEPLOY.md` +Part 1 §2. + +**Rationale:** + +- **Self-contained one-command local bring-up.** Redis is currently the only + piece of async infrastructure that has no local-dev story at all — `docs/broadcast.md` + and `DEPLOY.md` both describe it as prod-only apt install. Containerizing + it means `docker compose up` gives a working broker with no host packages. +- **Clean path to a managed service later.** `redis:7-alpine` in a container + today, backed by a named volume, ports directly to ElastiCache (or any + managed Redis) later — swap the connection URL, drop the container. No + application code changes either way, for the same reason the DB-split + already requires none (next point). +- **No code changes, because the existing DB-split already lives in env + vars.** The one-instance/two-logical-DB split — **DB 0 = Celery + broker + results, DB 1 = Django cache** — is not hardcoded; it falls + straight out of `REDIS_URL` (`backend/settings/base.py:146`, feeding + `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`) and `REDIS_CACHE_URL` + (`backend/settings/base.py:190-193`, the `CACHES` `LOCATION`). Point both + env vars at the same containerized Redis with `/0` and `/1` suffixes and + the split is unchanged. +- **`requirepass` is kept**, sourced from `.env` exactly as today + (`DEPLOY.md` §2: "The password lives only in `backendServer/.env`, never in + git") — containerizing Redis is not a reason to relax that. +- **Named volume for persistence**, since Redis here is not purely a cache — + DB 0 holds in-flight Celery task state and `django-celery-beat`'s schedule + metadata (the schedule itself is in Postgres via `DatabaseScheduler`, but + broker state is Redis-resident) — an unpersisted Redis would drop + in-flight/queued tasks on every container restart. + +--- + +## Decision 3 — Postgres is NOT containerized; it stays on Neon + +**Decision:** Postgres is not part of the compose stack. Both prod and local +development continue to point at Neon (external, managed) — prod at the prod +branch, local dev at a Neon dev branch per `docs/dev-db-isolation.md`. + +**Rationale:** Neon already gives us managed backups, branching, and +point-in-time recovery — the actual restore mechanism `DEPLOY.md`'s guarded +migrate calls out explicitly ("this dump is belt-and-suspenders; Neon +PITR/branching is the real restore mechanism"). Running a containerized +Postgres alongside a service that's already fully externalized would just be +two sources of truth for the same data, for local dev only, with no benefit +prod can use. + +**Consequence — the CI pre-migrate `pg_dump` needs a new home.** The deploy +job's guarded-migrate step (`.github/workflows/ci.yml:230-262`) takes a +`pg_dump` before applying any pending migration and currently depends on a +**host-installed** `postgresql-client` (`DEPLOY.md` line 412: "fails the +deploy if `pg_dump` is missing — one-time VM prep: `sudo apt install -y +postgresql-client`"). Once nginx/Redis/the app tiers are containerized, we +don't want a bare-metal apt package as the one remaining thing keeping the +backup step alive — it becomes a throwaway `postgres:18-alpine` container, +invoked for the duration of the dump and discarded (`docker run --rm +postgres:18-alpine pg_dump "$DATABASE_URL" | gzip > …`). + +**Why version 18 specifically, not "latest" or matched-to-Neon:** `pg_dump` +can dump a server *older* than itself, but refuses to dump one *newer* than +itself. Pinning the **newest** available client image is therefore +unconditionally safe regardless of what Postgres major version the Neon +branch is actually running — there's no version-matching exercise to redo +every time Neon upgrades its own server version, and no risk of the dump +silently failing the week Neon ships a major bump ahead of a stale pin. + +--- + +## Decision 4 — build artifacts are baked into images; only real state gets volumes + +This is the split most likely to be gotten wrong later, so it's spelled out +explicitly both ways. + +### Baked at build time (NOT volumes) + +| Artifact | Source | Why baked, not a volume | +|---|---|---| +| Django `collectstatic` output (`staticfiles/`) | `manage.py collectstatic --noinput` (`DEPLOY.md` §3, run on every deploy) | Regenerated from source on every build. A volume here would be strictly **worse** than baking: stale static files from a previous deploy could survive into a new container and be served alongside new code, reintroducing exactly the "admin has no CSS / stale bundle" class of bug `DEPLOY.md`'s troubleshooting table already lists. | +| `broadcastWeb`'s compiled `dist/` | `pnpm run build` (`DEPLOY.md` §7, currently served by nginx directly off the VM filesystem, "static → dist/, served directly by nginx, no service") | Same reasoning — a build artifact, not state. The CI smoke check that greps the built JS for a `thecommons.town` API origin (`ci.yml:274-281`) exists precisely because a stale/misconfigured `dist/` is a real failure mode; a volume would make that class of bug *stickier*, not safer. | + +The nginx image's Dockerfile is expected to `COPY --from=` both of these out +of dedicated build stages, so the running nginx container always serves +exactly what was built alongside the code it's serving — never a leftover +from a previous image. + +### Volumes (real state that must survive a container restart) + +| Path | Env var | What it holds | Why it must be a volume | +|---|---|---|---| +| `/home/ubuntu/broadcast/media` | `MEDIA_ROOT` | Client-uploaded broadcast event images | `DEPLOY.md` (§nginx, line 631) states these are **kept indefinitely with no pruning job** — this is genuine, growing user data, not a cache or a rebuildable artifact. Losing it on a restart would delete client uploads outright. | +| `/home/ubuntu/backups` | — (target of the CI pre-migrate `pg_dump`, `ci.yml:246`) | Pre-migrate `.sql.gz` dumps, newest 5 kept | Call this out explicitly: **if this path were ephemeral, the dump would be written and then discarded the moment the container that wrote it exits**, silently turning the guarded-migrate safety net into a no-op — the deploy would still *look* successful (the dump command exits 0), but there would be nothing to restore from after the fact. This is the one volume in the whole list where getting it wrong fails silently rather than loudly. | +| `/home/ubuntu/broadcast/screenshots`, `/home/ubuntu/broadcast/downloads` | `BROADCAST_SCREENSHOT_DIR`, `BROADCAST_DOWNLOAD_DIR` | Playwright debug artifacts from broadcast form-fill runs | Not user-facing data, but operators reference these when triaging a `needs_manual`/failed broadcast target after the fact (`docs/broadcast.md` §Models) — losing them on every restart would remove the only forensic trail for a failed submission. | +| (unnamed, Redis's data dir) | — | Celery broker state (DB 0), Django cache (DB 1) | Covered under Decision 2 — a named volume so queued/in-flight task state survives a Redis container restart. | + +The dividing line, stated once for reuse: **if a path can be fully +regenerated by re-running the build (or re-running `collectstatic`/`pnpm +build`), it's baked. If losing it on restart would destroy something no +rebuild can recreate — user uploads, a backup, a debug trail, cache/broker +state that's actively in flight — it's a volume.** + +--- + +## Base images + +All images must have working **arm64** variants — the only deploy target +today is the Oracle Cloud VM, Ubuntu 24.04 **ARM64** (`DEPLOY.md` §Facts, +line 15). This ruled out anything without a maintained arm64 build. + +| Component | Image | Notes | +|---|---|---| +| Backend (Django/gunicorn) + Playwright workers (`scrape-worker`, `broadcast-worker`) | `python:3.13-slim` | Matches the `python-version: "3.13"` pinned in CI (`ci.yml:23,82`) and `backendServer/pyproject.toml`. | +| `theCommonsWeb`, `broadcastWeb` | `node:22-slim` | Matches Node 22 pinned in CI (`ci.yml:32,109`) and the `pnpm 11.1.1` requirement (pnpm 11 needs Node ≥22.13, `ci.yml:104`). | +| Redis | `redis:7-alpine` | See Decision 2. | +| nginx | `nginx:1.27-alpine` | See Decision 1. | +| Postgres client (pre-migrate dump sidecar only — not a running service) | `postgres:18-alpine` | See Decision 3; deliberately newer than any Postgres version Neon is expected to run. | + +**Chromium is installed into the worker image via `uv run playwright install +--with-deps chromium`, not via a pinned `mcr.microsoft.com/playwright` base +image.** `backendServer/pyproject.toml:31` pins `playwright>=1.60.0`, and +`uv.lock` locks the resolved version exactly (`uv.lock:1006`, `1.60.0`). +Installing the browser build through that same `playwright` package +guarantees the Chromium build matches the library version resolved by +`uv.lock` — that's exactly what `playwright install` does: it fetches the +browser build matching the installed package's version, not a browser +version chosen independently. A pinned Microsoft Playwright base image +instead fixes the Chromium+library pair to whatever version *that image* +shipped with; the moment someone bumps `playwright` in `pyproject.toml` +without also bumping the base image tag (or vice versa), the library and +browser drift out of sync — the exact class of bug Playwright's own docs +warn causes cryptic `Executable doesn't exist` / protocol-mismatch failures. +Installing through the locked library keeps one version number +(`uv.lock`'s) as the single source of truth instead of two that can silently +diverge. This also matches current prod practice (`DEPLOY.md` Part 1 §5.3 / +§6.1: `uv run playwright install chromium` + `install-deps`, bundled Chromium +only — arm64 has no branded "Chrome"). + +--- + +## Historical note: why the systemd units exec binaries directly (and why containers make it moot) + +Every unit in `deploy/*.service` execs `.venv/bin/celery` directly instead of +going through `uv run`. This looks like an odd style choice until you know +why: snap-packaged `uv run` spawned its child process inside a transient +`snap.astral-uv.uv-*.scope` under `user@1001.service` — the *user* session +manager, not the unit's own cgroup. With linger off, `systemd-logind` tore +that user slice down the moment the deploying SSH session ended, and every +`uv run`-based unit died within a minute of each deploy, silently, with +`status=0/SUCCESS` — a clean exit that `Restart=on-failure` correctly +declined to restart. That's what took the entire async stack down for 8 +days starting 2026-07-21 while `gunicorn` and `nextjs` (which never went +through snap) stayed up the whole time (full forensics: +`docs/prod-incident-2026-07-21-scheduler-outage.md`). The current mitigation +is two-layered: exec the venv binary directly (removes the mechanism +entirely) plus `Restart=always` + `loginctl enable-linger ubuntu` as +defense-in-depth in case a unit is ever reverted. + +**Containers remove this failure mode by construction** — there is no snap, +no `logind`, no per-user systemd slice inside a container's PID namespace for +a lost SSH session to tear down. The elaborate two-layer mitigation +(exec-direct + linger + `Restart=always`) is a workaround for a systemd/snap +interaction that containers simply don't have, and should **not** be +cargo-culted forward into the compose/Dockerfile setup as if it were still +solving a live problem. + +One piece *is* still worth carrying forward, for an unrelated reason: **exec +binaries directly rather than through a wrapper shell**, so the container's +PID 1 is the actual process (`celery`, `gunicorn`, …) and receives `SIGTERM` +directly on `docker stop` / `docker compose down` for a clean shutdown, +rather than a shell that may not forward signals to its child. Same +`ExecStart=` shape as today, different reason. + +--- + +## Doc drift observed while researching this ADR + +- `DEPLOY.md` line 412 ("one-time VM prep: `sudo apt install -y + postgresql-client`") describes the pre-containerization world and will be + stale once the pre-migrate dump moves into the `postgres:18-alpine` + sidecar (Decision 3). Not fixed here — `DEPLOY.md` is explicitly out of + scope for this ticket and owned by a later one in this suite — flagging so + whoever writes that ticket knows this line needs to go. +- `DEPLOY.md` §Services (line 518) documents gunicorn on + `unix:/run/gunicorn/gunicorn.sock`. This ADR's Decision 1 changes that to + TCP (`backend:8000`) once nginx is containerized. This isn't contradictory + doc drift so much as a documented, deliberate divergence this ADR + introduces — noting it here so the DEPLOY.md rewrite ticket doesn't miss + the socket→TCP change as one more thing to update, beyond the nginx + section itself. +- No contradiction found between the brief and the repo on any of the four + decisions themselves — `REDIS_URL`/`REDIS_CACHE_URL` really do carry the + whole DB-split with no code changes needed (verified at + `backend/settings/base.py:146-193`), `MEDIA_ROOT` really is documented as + unpruned (`DEPLOY.md` line 631), and `playwright` really is a `uv.lock`-pinned + dependency rather than a loose one (`pyproject.toml:31`, resolved to exactly + `1.60.0` in `uv.lock`). diff --git a/docs/handoff-suite-42-2026-08-02.md b/docs/handoff-suite-42-2026-08-02.md new file mode 100644 index 0000000..1306145 --- /dev/null +++ b/docs/handoff-suite-42-2026-08-02.md @@ -0,0 +1,155 @@ +# Handoff — Suite 42 (Dockerize the stack), 2026-08-02 + +Paste the block below to a fresh Claude Code instance. Everything it needs is inline; +it does not need to read this preamble. + +--- + +## PROMPT FOR NEXT INSTANCE + +You are picking up **suite 42 — Dockerize the stack** on `The Commons` +(`/Users/ErenYeager/Desktop/hw/thecommons`, branch `all-things-ingestion`). +Read `CLAUDE.md`, `AGENTS.md`, and `DEPLOY.md` first — DEPLOY.md was just rewritten +and is the system of record for everything below. + +### Read this before you touch anything + +**⚠️ There is a live landmine. Do not merge this branch to `main` yet.** + +`.github/workflows/ci.yml:171` runs the `deploy` job on every push to `main`. That job +was rewritten in suite 42 to build images and run `docker compose -f docker-compose.yml +up -d` over SSH — **but the production VM has no Docker installed.** The first merge to +`main` will fail the deploy, and the VM's current systemd-based stack will keep running +the old code with no indication anything was attempted. The branch is currently 3 commits +ahead of `main` with all of suite 42 uncommitted, so nothing has fired yet. + +Sequence matters: **VM prep must land before the merge**, or the merge must be gated. + +### What suite 42 already did (all verified, none of it committed) + +The stack is containerized and was proven end-to-end on a local machine against the Neon +dev branch. Postgres is deliberately NOT containerized. + +New files: +- `backendServer/Dockerfile` — multi-stage: `app` (gunicorn/celery/celerybeat, 492 MB, no + Chromium) and `playwright` (`FROM app` + bundled Chromium 148, 1.97 GB) for the + broadcast and scrape workers. +- `Dockerfile.frontend` — shared: `commons-runtime` (Next.js standalone, port 3000, + 284 MB) and `broadcast-build` (build-only, SPA at `/app/dist`, 387 MB). + `theCommonsWeb` and `broadcastWeb` are SEPARATE pnpm workspaces with separate lockfiles. +- `deploy/nginx/Dockerfile` + `deploy/nginx/thecommons.conf` — nginx as sole ingress, + baking in collectstatic output and the broadcast SPA via Buildx named build contexts. +- `docker-compose.yml` — replaces all seven systemd units. +- `docker-compose.override.yml` — local dev only (plain HTTP, no cert, `DJANGO_ENV=dev`, + repo-relative mounts under `.local-dev/`). +- `.dockerignore`, `backendServer/.dockerignore`. +- `docs/adr/0001-containerization.md` — the architecture decisions and rationale. + +Modified: `.github/workflows/ci.yml` (deploy job), `DEPLOY.md` (full rewrite), +`deploy/healthcheck.{sh,service,timer}` (systemd → container checks), +`theCommonsWeb/next.config.ts` (added `output: 'standalone'`), `.gitignore`, +`notion-sync/{OUTBOX.md,STATE.md}`. + +**Verified locally:** all 8 long-running services up; apex / `/events/` / `/admin/login/` +/ baked static / media / broadcast SPA incl. deep-link fallback all 200 through nginx; +3 distinct Celery nodes each draining only its own queue; Chromium launches in both +Playwright workers; Redis DB 0 (broker) + DB 1 (cache) both exercised; `docker compose +down` clean. + +### Rules that will bite you + +1. **Every prod compose command needs `-f docker-compose.yml` explicitly.** A bare + `docker compose` auto-loads `docker-compose.override.yml`, which is local-dev-only — + you would deploy plain-HTTP, cert-less, `DJANGO_ENV=dev` config to production. +2. `-c 1` on the broadcast worker is mandatory, not tuning — `recover_orphans()` assumes + a single worker. Exactly one `celerybeat`. +3. Compose does **not** auto-recreate a container when inline `configs:` content changes. + Use `--force-recreate`. +4. Docker Desktop's proxy (`http.docker.internal:3128`) corrupts large apt fetches with + `Hash Sum mismatch` on a different package each run. `backendServer/Dockerfile` + mitigates it via `/etc/apt/apt.conf.d/99-robust`. +5. `.dockerignore` patterns need an explicit `**/` to match inside subdirectories on this + BuildKit — stricter than `.gitignore`. A bare `*.pem` does NOT exclude + `broadcastWeb/commons-broadcast.pem`. +6. The tree is dirty with three overlapping suites (41 backend refactor, 42 docker, + 43 human docs). **Do not `git add -A`** — a blanket commit sweeps unrelated in-flight + work. Commit suite 42's files explicitly by path. + +### Next steps, in order + +**1. Decide the merge-safety strategy (blocking, do this first).** +Either complete VM prep before merging, or temporarily gate the deploy job. Do not leave +`main` armed against a Docker-less VM. This is a judgement call — surface the options +rather than picking silently. + +**2. Do the one-time VM prep — `DEPLOY.md` Part 1, an 11-step numbered checklist.** +Target: Oracle Ubuntu 24.04 **ARM64**, user `ubuntu`, repo at `/home/ubuntu/thecommons`. +Highlights, but follow the doc: +- Install Docker Engine + the compose v2 plugin (arm64). +- Add `ubuntu` to the `docker` group and **verify over a real non-interactive SSH session** + (`ssh host 'docker ps'`, not an interactive login, and not `newgrp docker` — both paper + over a broken setup; CI connects non-interactively). +- Create `/home/ubuntu/broadcast/{media,screenshots,downloads}` and `/home/ubuntu/backups`, + writable by **uid 1000** (the image's non-root `app` user). +- Confirm the Cloudflare origin cert at `/etc/ssl/cloudflare/thecommons.town.{pem,key}`. + +**3. Edit prod `backendServer/.env` — two release blockers.** +- `REDIS_URL` / `REDIS_CACHE_URL` currently point at `localhost`/`127.0.0.1`. **Inside a + container `localhost` is the container itself**, so Celery silently connects to nothing, + no task ever runs, and every other signal stays green — the same failure shape as the + 2026-07-21 scheduler outage. Must become + `redis://:@redis:6379/0` and `.../1`. +- Add `REDIS_PASSWORD` (it is NOT in `.env.example` today; the `redis` container reads it + directly, Django/Celery only read the URLs) — it must match the password in those URLs. +- Confirm `DJANGO_ALLOWED_HOSTS` is set — `backend/settings/prod.py:16` does + `os.environ["DJANGO_ALLOWED_HOSTS"]`, a hard `KeyError` with no default. +- Confirm `DJANGO_ENV=prod`. If unset the app silently serves dev settings, whose + localhost-only `ALLOWED_HOSTS` rejects `api.thecommons.town` with a 400. + +**4. Host nginx → container nginx cutover.** Keep the host nginx additive until cutover, +then stop/disable it so the container can bind 80/443. Retire the old app systemd units +and the sudoers `systemctl` drop-in. + +**5. Run the test suites — NOT done in suite 42.** +- Backend: `cd backendServer && DJANGO_SETTINGS_MODULE=backend.settings.test uv run python manage.py test` + (`--tag=fast` no-DB tier, `--tag=db` DB tier). +- Frontend: `cd theCommonsWeb && pnpm build` — **specifically verify the new + `output: 'standalone'` doesn't break the host build or CI's `frontend-commons` job.** + This was only ever exercised inside Docker, never on the host. + +**6. First automated container deploy**, then watch `docker compose -f docker-compose.yml +logs` and run `deploy/healthcheck.sh` on the VM. + +### Verification gaps — be honest about these, don't assume + +- **Nothing was tested on the actual VM.** All verification was local, on Apple Silicon. +- The CI `deploy` job has **never run** — it needs the VM plus repo secrets. It is + YAML-valid, `actionlint`-clean, and its script is shellcheck-clean at warning level, + but its runtime behavior is unproven. +- The **Django test suite was not run** during suite 42 (no Python app source changed — + only `next.config.ts`), so it is untested against the suite-41 refactor in this tree. +- `pnpm build` on the host with `output: 'standalone'` is unverified. +- The broadcast end-to-end submit flow (enqueue → worker drains → Playwright fills a real + form) was not exercised; only that Chromium launches in both workers. + +### Already done, don't redo + +- The 6 pending suite-41 migrations were applied to Neon branch `test_neondb_cacheverify` + with the owner's approval. `migrate --check` is clean there. 5 were + `SeparateDatabaseAndState` (zero DDL); 1 was a reversible `RunPython` repointing beat + task paths. +- Suite 42 is queued in `notion-sync/OUTBOX.md` as `Needs QA` (tickets 42.1–42.8) and + recorded in `notion-sync/STATE.md`. Don't re-add it. Follow `CLAUDE.md`'s Notion rules + for any new board changes. +- Three DEPLOY.md doc-drift items are already fixed: the static path (`staticfiles_build/ + static`, not `staticfiles/`), the obsolete `apt install postgresql-client`, and the + gunicorn unix socket (now TCP `backend:8000`). + +### Useful context + +The six bugs suite 42 found were all invisible to code review and only surfaced by +running things — root-owned `/app` breaking collectstatic, nginx refusing to boot when any +upstream was missing (one dead frontend = total ingress outage), the `REDIS_URL` localhost +trap, the `DJANGO_ALLOWED_HOSTS` KeyError, a `DisallowedHost` 400, and `.dockerignore` +leaking private keys into the build context. **Prefer executing over reading** when +verifying anything here. diff --git a/docs/redis-celery-handoff.md b/docs/redis-celery-handoff.md index fe6f18c..ae1a2af 100644 --- a/docs/redis-celery-handoff.md +++ b/docs/redis-celery-handoff.md @@ -60,8 +60,8 @@ the code and reproduces on a fresh DB. Current entries: | Task | Path | Schedule (seeded by) | |------|------|----------------------| | Ingestion pipeline | `ingestion.tasks.run_ingestion_pipeline` | 04:00 daily, `America/New_York` (`ingestion/migrations/0007_seed_ingest_beat.py`) | -| Weekly digest fan-out | `events.tasks.fan_out_weekly_digest` | Sundays 18:00, `America/New_York` (`events/migrations/0015_seed_digest_beat.py`) | -| Monthly digest fan-out | `events.tasks.fan_out_monthly_digest` | 1st of month 18:00, `America/New_York` (`events/migrations/0020_seed_monthly_digest_beat.py`, task name `monthly-digest-first`) | +| Weekly digest fan-out | `newsletter.tasks.fan_out_weekly_digest` | Sundays 18:00, `America/New_York` (`events/migrations/0015_seed_digest_beat.py`; repointed from `events.tasks.fan_out_weekly_digest` by `newsletter/migrations/0002_repoint_digest_beat.py`) | +| Monthly digest fan-out | `newsletter.tasks.fan_out_monthly_digest` | 1st of month 18:00, `America/New_York` (`events/migrations/0020_seed_monthly_digest_beat.py`, task name `monthly-digest-first`; repointed from `events.tasks.fan_out_monthly_digest` by `newsletter/migrations/0002_repoint_digest_beat.py`) | The `CrontabSchedule.timezone` is set to `America/New_York` (not UTC) so beat tracks US-Eastern DST exactly like the OS cron these replaced. diff --git a/human-docs/HANDOFF_PLAN.md b/human-docs/HANDOFF_PLAN.md new file mode 100644 index 0000000..8a5f2cb --- /dev/null +++ b/human-docs/HANDOFF_PLAN.md @@ -0,0 +1,142 @@ +# Human-docs handoff plan + +> The paste-ready prompt for a `/handoff-report` session that authors the human-facing +> onboarding docs. The repo has none yet — this plan enumerates every non-obvious subsystem +> to document. Foundational subsystems first; a natural first cut is **overview → auth → +> ingestion → data-model** (what a new owner needs to be dangerous), then the rest. +> +> Naming note: doc filenames below match the routes in [`start-here.md`](start-here.md). +> Register each finished doc in [`README.md`](README.md)'s index. + +--- + +## Prompt + +``` +Write human-facing onboarding documentation for The Commons into human-docs/. The repo +has NONE yet (human-docs/ holds only README.md + an empty index). Audience: an inheriting +owner / new teammate with general web-dev skill but zero context on THIS codebase — not +agents. Use the /handoff-report skill for each doc (it defines structure, grounding rules, +and the publish checklist): each doc = what the subsystem does + who depends on it, Mermaid +diagrams of REAL behaviour read from the code, data-model / interface tables, and the sharp +edges that bite newcomers. Ground every claim in code — read it, don't trust prose or these +notes blindly; flag drift. Register each doc in human-docs/README.md's index. Keep the +agent-facing docs/ tree untouched (link to it where a human should go deeper). Prose, not +bullet-dumps; the digital-newspaper voice is fine. + +The non-obvious "sharp edges" below are SEEDS — confirm each against current code and expand. +Run these as separate /handoff-report passes, foundational subsystems first. + +1. SYSTEM OVERVIEW — human-docs/overview.md + What the product is (local events aggregator for Chapel Hill/Carrboro/Pittsboro), the + monorepo layout (backendServer / theCommonsWeb / broadcastWeb / broadcastExtension / + deploy), the end-to-end lifecycle of an event (ingested → standardized → published → + optionally broadcast → emailed in a digest), and a single architecture diagram. The map + a newcomer reads first. Sources: README.md, ARCHITECTURE.md, PROJECT_CONTEXT.md, code. + +2. AUTH BRIDGE — human-docs/auth.md + How Better Auth (Next.js) is the source of truth for identity and Django only mirrors it. + The `accounts` app's neon_auth mirror models (managed=False + the double-quote db_table + trick), JWKS/JWT verification (backend/jwt_auth.py, backend/permissions.py), the + auth.thecommons.town cross-subdomain cookie setup. Sharp edges: JWKS fetch needs a + browser User-Agent or Cloudflare 403s every verify (see docs/... incident); DJANGO_ENV + must be "prod" in .env or prod silently falls back to dev settings → DisallowedHost 400 → + "no events"; BetterAuthAccount.user_id is uuid not text (past ORM-join failures); + password-required accounts + the passwordless-account rollover story. Sources: accounts/, + backend/jwt_auth.py, backend/permissions.py, theCommonsWeb/src/lib/auth*.ts, + docs/runbook-auth-cutover.md, docs/prd-centralized-auth.md. + +3. INGESTION PIPELINE — human-docs/ingestion.md + The poll → Gemini-standardize → dedup → safety-score → stage → publish flow, the three + source types (ICS feed / web scraper / HTTP fetch) and how a source is classified, + RawEvent/StagedEvent/EventSource, and where the LLM sits. Sharp edges: events.Event PK + is `uuid` not `id` (Count("id") raises FieldError — use Count("pk")); INGEST_SHARD_COUNT + makes plain ingest_events poll only a subset (use --shard for all); auto_publish + early-returns when nothing is pending (call publish_all_approved to flush); source-classifier + traps (CivicPlus real-ICS path, Tribe ?ical=1 false positive, Akamai vs AWS-WAF, Chatham + County = Akamai-blocked/headless-only, GET-only no-scroll limits); never use the ORM inside + sync_playwright. Sources: ingestion/, docs/ingestion-pipeline.md, docs/safety-scoring.md, + the source-creation skill, memory of classification gotchas. + +4. BROADCAST (event syndication) — human-docs/broadcast.md + What broadcasting is (pushing published events onto third-party town calendars via + Playwright + a Chrome extension), the adapter pattern, the operator SPA (broadcastWeb), + and access codes. Sharp edges: broadcast/ is isolated by contract — routing.py must not + import from events, and no ORM inside sync_playwright; broadcast runs on its own Celery + queue drained by a SINGLE -c 1 worker (orphan-recovery correctness depends on it); the + extension only autofills calendar hosts listed in manifest host_permissions (missing host + = silent no-fill). This is a HUMAN summary — docs/broadcast.md stays the agent source of + truth; link to it. Sources: broadcast/, broadcastWeb/, broadcastExtension/, docs/broadcast.md. + +5. NEWSLETTER & DIGESTS — human-docs/newsletter.md + The `newsletter` app: email-only subscribe, manage-token (login-free) preference page, + the weekly + monthly digest engine (recipient resolution across subscribers + user + profiles, Brevo send), and the Celery-beat schedule. Sharp edge: digest fan-out tasks + live at newsletter.tasks.* (beat PeriodicTask rows were repointed in Suite 41). Sources: + newsletter/, backend beat migrations. + +6. ASYNC: REDIS + CELERY — human-docs/async-jobs.md + Redis layout (DB 0 = broker/results, DB 1 = cache), the three queues (default / scrape / + broadcast) and which worker drains each, and how beat schedules work. Sharp edges: + CELERY_BEAT_MAX_LOOP_INTERVAL=6h starves beat's _do_sync so last_run_at lags up to 6h + (fix = CELERY_BEAT_SYNC_EVERY=1) — never tune a staleness window without accounting for it; + the TzAwareCrontab remaining_estimate tz bug (conversion lives only in is_due()). Sources: + backend/settings/base.py, backend/celery.py, deploy/ units, docs/redis-celery-handoff.md, + docs/ingestion-monitoring.md (last_run_at incident). + +7. DEPLOYMENT & OPS — human-docs/deploy-ops.md + The single Oracle Cloud VM, Neon Postgres, nginx + gunicorn (Unix socket), systemd units + (celery / celerybeat / broadcast-worker / scrape-worker), how deploys happen, and media + handling (MEDIA_ROOT lives OUTSIDE the checkout so git pull never touches uploads; nginx + serves /media, never Django). Sharp edges: the DJANGO_ENV prod selector (above); the + snap-uv user-slice teardown that took out the scheduler (prod incident); dev-DB isolation + via Neon branches. Sources: DEPLOY.md, deploy/, docs/dev-db-isolation.md, + docs/prod-incident-2026-07-21-scheduler-outage.md. + +8. FRONTEND (main site) — human-docs/frontend.md + Next.js 16 App Router structure, the services/hooks data layer, TanStack Query usage, + Better Auth client integration, and the routes. Sharp edge: this is pnpm-managed — `npm + install` fails on the symlinked store; type-check is `pnpm build`. Sources: theCommonsWeb/, + theCommonsWeb/AGENTS.md. + +9. DESIGN SYSTEM — human-docs/design-system.md + The digital-newspaper aesthetic as an enforceable spec: Georgia serif, cream/ink palette, + column rules, drop caps, density over whitespace; the banned list (no gradients, no rounded + pill buttons, no startup/AI-forward vibes). CSS tokens and component conventions. Sources: + CODING_STYLE.md, theCommonsWeb/src/app/globals.css, src/components/ui/. + +10. DATA MODEL REFERENCE — human-docs/data-model.md + Every core model and how they relate: Event, Town, Category, Tag; RawEvent, StagedEvent, + EventSource, SourceRun; accounts (BetterAuth mirrors, UserProfile, BusinessProfile); + NewsletterSubscriber; broadcast models. One ER-style Mermaid diagram + a table per model + (fields, key relationships, which app owns it). Call out that Event PK is uuid. Sources: + each app's models.py, ARCHITECTURE.md §Data Models. + +11. TESTING & LOCAL DEV — human-docs/testing.md + How to run everything locally (uv + pnpm, local Redis), the two backend test tiers + (--tag=fast no-DB, --tag=db Postgres) and the test settings module, and the frontend test + setup (vitest). Sharp edges: the Neon test DB is shared — concurrent test runs collide and + green results are untrustworthy; run serially. Sources: backendServer/AGENTS.md#testing, + CLAUDE.md, vitest.config.ts. + +DEFINITION OF DONE: each doc reads as standalone onboarding for a human who's never seen the +repo; every diagram reflects real code paths (not idealized); every sharp edge is verified +against current code; each doc registered in human-docs/README.md's index with a one-line +purpose; internal links resolve. Docs only — no code changes. Where the agent-facing docs/ +already cover something deeply, summarize for humans and link rather than duplicate. +``` + +--- + +## Suite 41 addendum (the refactor that prompted this) + +When writing **auth.md**, **newsletter.md**, and **data-model.md**, fold in the Suite 41 +outcomes: the `accounts` app owns identity (5 Better Auth mirrors + `UserProfile` + +`BusinessProfile`) and `newsletter` owns the digest engine; `backend/urls.py` is +`include()`-only; model moves were `SeparateDatabaseAndState` state-only (zero DDL, +`db_table` preserved, `neon_auth` never migrated). Document the deliberate +`accounts ↔ newsletter` import cycle (email-pref sync ↔ digest tag-filtering) so a reader +doesn't "fix" it by accident. Two agent-facing docs still carry stale references to fix in +the same pass: `docs/admin-backend.md` (`/admin/events/userprofile/` → +`/admin/accounts/userprofile/`) and `docs/redis-celery-handoff.md` +(`events.tasks.fan_out_*` → `newsletter.tasks.*`). diff --git a/human-docs/README.md b/human-docs/README.md new file mode 100644 index 0000000..8fc475c --- /dev/null +++ b/human-docs/README.md @@ -0,0 +1,33 @@ +# Human Docs + +This folder is for people, not agents. `docs/` is the system of record that Claude reads before +every task (`CLAUDE.md` → `AGENTS.md` → `ARCHITECTURE.md` → `CODING_STYLE.md` → `docs/*.md`) — +keep that tree lean and agent-oriented. Anything written primarily for a human reader — an +inheriting owner, a new teammate, a neighbouring team — lands here instead, so it doesn't dilute +what agents load on every task. + +Written mainly by `/handoff-report`. See that skill for structure, grounding rules, and the +publish checklist. + +**New here?** Start with [`start-here.md`](start-here.md) — it routes you to the right doc by +what you're trying to do. [`HANDOFF_PLAN.md`](HANDOFF_PLAN.md) tracks which subsystem docs are +still to be written. + +## Index + +| Doc | Purpose | Written | +|---|---|---| +| [start-here.md](start-here.md) | Task-oriented map: "I want to…" → the right doc | 2026-08-01 | +| [overview.md](overview.md) | System map: product, monorepo layout, event lifecycle, architecture diagram — read this first | 2026-08-01 | +| [auth.md](auth.md) | Better Auth (Next.js) is identity's source of truth; Django only mirrors it and verifies JWTs | 2026-08-01 | +| [ingestion.md](ingestion.md) | Source → Gemini standardize → dedupe → safety-score → publish, and how to classify a new source | 2026-08-01 | +| [data-model.md](data-model.md) | Every core model, field by field, and how they relate — the reference to keep open in another tab | 2026-08-01 | +| [broadcast.md](broadcast.md) | Syndicating events out to other towns' calendars: adapters, extension autofill, access codes | 2026-08-01 | +| [newsletter.md](newsletter.md) | Subscription lifecycle, recipient resolution, and the weekly/monthly digest engine | 2026-08-01 | +| [async-jobs.md](async-jobs.md) | Redis layout, the three Celery queues and who drains each, beat scheduling and its lag trap | 2026-08-01 | +| [deploy-ops.md](deploy-ops.md) | Production mental model: the VM, systemd units, deploys, and the sharp edges that cause outages | 2026-08-01 | +| [containerization.md](containerization.md) | **Provisional** — the in-flight Docker cutover: built and locally verified, not yet deployed | 2026-08-01 | +| [frontend.md](frontend.md) | Main site: App Router routes, the TanStack Query data layer, Better Auth on the client | 2026-08-01 | +| [design-system.md](design-system.md) | The digital-newspaper aesthetic as an enforceable spec — tokens, type scale, the banned list | 2026-08-01 | +| [testing.md](testing.md) | Local setup, the backend test tiers, the shared-test-DB hazard, and what CI runs | 2026-08-01 | +| [HANDOFF_PLAN.md](HANDOFF_PLAN.md) | The plan these docs were written from — kept for provenance; all 11 planned docs now exist | 2026-08-01 | diff --git a/human-docs/async-jobs.md b/human-docs/async-jobs.md new file mode 100644 index 0000000..45c6e33 --- /dev/null +++ b/human-docs/async-jobs.md @@ -0,0 +1,443 @@ +# Async: Redis + Celery + +Written 2026-08-01 against commit `5fe7a45`. This is the human-facing companion to +[`docs/redis-celery-handoff.md`](../docs/redis-celery-handoff.md) (the agent-facing deep +dive — stays the system of record for exact settings names, migration mechanics, and prod +provisioning commands) and touches on the same incident [`docs/ingestion-monitoring.md`](../docs/ingestion-monitoring.md) +records in its "Beat scheduler: last_run_at persistence lag" section. Where this doc and +those disagree, the code won the argument — see "Known gaps and doc drift" at the end. + +Audience: someone inheriting this codebase who needs to add a new background job or +scheduled task, or figure out why a scheduled job didn't run. + +**A note on what's current.** A parallel effort is moving this stack into Docker Compose — +there's a `docker-compose.yml` in the tree defining `celery`, `celerybeat`, +`broadcast-worker`, and `scrape-worker` services that mirror the systemd units this doc +describes, and `deploy/healthcheck.sh` has already been rewritten to shell into those +containers. None of that is cut over yet. This doc describes the systemd units +(`deploy/celery.service`, `deploy/celerybeat.service`, `deploy/broadcast-worker.service`, +`deploy/scrape-worker.service`) as they exist at this commit, because that's what's +actually running in production today. The containerized version is covered by a sibling +doc, `containerization.md`, once it exists. + +--- + +## 1. What this is and who depends on it + +The Commons runs almost everything that isn't an HTTP request/response cycle through +Celery: the nightly ingestion pipeline, weekly and monthly email digests, bulk-publishing +approved events, and pushing a submitted event out to third-party town calendars +(broadcast). Before Celery existed, this was two disconnected things — a bespoke +Postgres-polling broadcast worker, and OS cron hitting HTTP endpoints — with no shared +retry, queueing, or scheduling story. Redis + Celery gives every Django app in the repo +(`accounts`, `events`, `newsletter`, `ingestion`, `broadcast`) a single place to define +`@shared_task` functions and a single place (Postgres, via `django-celery-beat`) to define +when they run. + +Two things depend on this layer directly and visibly: the ingestion pipeline (see +[`ingestion.md`](ingestion.md) once it exists) won't pull in new events at all if +beat or the default worker is down, and the newsletter digest engine +(see [`newsletter.md`](newsletter.md) once it exists) won't send weekly/monthly email if +the same is true. Broadcast (see [`broadcast.md`](broadcast.md) once it exists, or +[`docs/broadcast.md`](../docs/broadcast.md) today) depends on it too, but less visibly — +**`ARCHITECTURE.md` currently claims broadcast "does not use Celery — it runs its own +DB-backed queue worker," and that's stale.** `broadcast/tasks.py` defines two real Celery +tasks, `process_broadcast_queue` and `recover_broadcast_orphans`, routed to their own +`broadcast` queue. What's true is narrower and more interesting than the doc's claim: the +*queue-claiming logic itself* (`broadcast/worker.py`, `SELECT ... FOR UPDATE SKIP LOCKED` +against Postgres) is bespoke and older than the Celery integration, but it now runs +*inside* a Celery task rather than its own standalone service. If this whole layer goes +down, nothing crashes anywhere in the request path — events just quietly stop flowing, +digests stop sending, and broadcast submissions sit at `status="queued"` forever. That +silence is exactly why the healthcheck section later in this doc matters as much as the +"how to add a task" section. + +--- + +## 2. How it works + +### 2.1 Queue and worker topology + +Redis holds two logically separate things on one self-hosted instance, and it's worth +saying plainly: **DB 0 is the Celery broker and result backend. DB 1 is the Django page +cache** (`events/cache.py`, wired into `CACHES["default"]` via Django's stdlib +`RedisCache` backend). They are unrelated to each other — a Redis `FLUSHALL` takes out +both, but nothing else couples them, and nothing should ever read/write DB 1 as if it were +the broker or vice versa. + +On the Celery side there are three named queues, each drained by a purpose-built worker +process (a separate systemd unit, all reading the same `REDIS_URL` broker on DB 0): + +```mermaid +flowchart LR + subgraph Triggers["What puts a task on a queue"] + Beat["celerybeat\n(DatabaseScheduler,\nreads Postgres)"] + HTTPCron["GET /api/cron/ingest"] + HTTPPublish["POST /api/events/publish-approved"] + HTTPSubmit["POST /api/events/direct-submit"] + HTTPBroadcast["POST /broadcast/submit\n/retry /submit-real"] + end + + subgraph Broker["Redis DB 0 (broker + result backend)"] + QDefault["queue: default\n(Celery's implicit 'celery' queue —\nanything not explicitly routed)"] + QScrape["queue: scrape"] + QBroadcast["queue: broadcast"] + end + + subgraph Workers["Worker processes (systemd, run in parallel)"] + WDefault["commons-default\ncelery.service\n--concurrency=2"] + WScrape["commons-scrape\nscrape-worker.service\n-Q scrape -c 1"] + WBroadcast["commons-broadcast\nbroadcast-worker.service\n-Q broadcast -c 1"] + end + + Beat -->|"ingest-events-daily,\nweekly/monthly digest"| QDefault + Beat -->|scrape-sources-daily| QScrape + Beat -->|broadcast-orphan-recovery| QBroadcast + HTTPCron --> QDefault + HTTPPublish --> QDefault + HTTPSubmit --> QDefault + HTTPBroadcast -->|"transaction.on_commit\n(process_broadcast_queue)"| QBroadcast + + QDefault --> WDefault + QScrape --> WScrape + QBroadcast --> WBroadcast +``` + +**Five things worth calling out:** + +1. **The default queue is memory-light and network-bound** (LLM calls, Brevo email, plain + HTTP fetches), so it runs at `--concurrency=2` on the same 1-OCPU/6GB VM everything + else shares. `scrape` and `broadcast` each get their own `MemoryMax=2G`-capped, `-c 1` + worker specifically because both drive headless Chromium (Playwright), which is memory- + heavy enough that letting it share the default worker risked taking down gunicorn and + Next.js alongside it. +2. **The `broadcast` queue's `-c 1` is not a tuning knob — it's a correctness + requirement.** `recover_broadcast_orphans` assumes that any `BroadcastSubmission` still + in `status="running"` at the moment it runs is necessarily orphaned (the worker that + claimed it must have died mid-job). That assumption only holds if `process_broadcast_queue` + can never be mid-drain on a *second* worker at the same time — i.e., if there is + structurally only one broadcast worker process. Scaling `broadcast-worker.service` up + (more replicas, or dropping the `-c 1`) is the obvious-looking fix for "broadcast feels + slow" and it is wrong: it reintroduces exactly the race the single-worker constraint + exists to prevent. +3. **`process_broadcast_queue` is not on a beat schedule at all.** Every submission, + retry, and promote-to-real action calls `transaction.on_commit(process_broadcast_queue.delay)` + (`broadcast/services.py`) — the queue gets drained on demand, right after the DB row + that created the work actually commits, not on a poll loop. `recover_broadcast_orphans` + *is* on a beat schedule (every 6 hours) precisely because on-demand dispatch has no + mechanism to notice a submission that got stranded when nobody triggered a new one. +4. **Scrape and ingest are two separate beat schedules on two separate queues, not one + pipeline.** `run_ingestion_pipeline` (the `ingest-events-daily` task, on the default + queue) only polls `ics`-type sources. `scrape_all_sources_task` (on the `scrape` queue, + fired 30 minutes earlier so scraped rows land before the standardizer runs) polls + `scraper`- and `http`-type sources. They are deliberately not chained — a prior version + of `run_ingestion_pipeline`'s docstring claimed it mirrored the CLI command end-to-end, + which was false, and two `http`-type sources went unpolled for weeks before anyone + noticed. `python manage.py ingest_events` with no flags is the one entrypoint that + actually covers both legs in one pass. +5. **"default" is a convenience name, not a Celery setting.** Nothing in + `backend/settings/base.py` sets `CELERY_TASK_DEFAULT_QUEUE` — the queue everything lands + on by default is Celery's own built-in `celery` queue. `CELERY_TASK_ROUTES` only lists + the two exceptions (`scrape`, `broadcast`); everything else falls through to that + implicit queue, which `commons-default`'s worker drains because it starts with no `-Q` + flag at all. + +### 2.2 Where a schedule comes from, and how to change one + +Beat schedules are **not defined in a settings file you can grep for the current cron +times.** `django-celery-beat`'s `DatabaseScheduler` reads `CrontabSchedule`/`PeriodicTask` +rows out of Postgres, and Postgres — not any `.py` file — is the live truth. Code only +gets involved once, to seed the row the first time: + +```mermaid +flowchart TD + A["Write the @shared_task\nin the owning app's tasks.py"] --> B["Write a data migration:\nCrontabSchedule.objects.get_or_create(...)\nPeriodicTask.objects.update_or_create(name=..., task='app.tasks.func')"] + B --> C["Migration runs once,\non whichever DB applies it"] + C --> D["Row now lives in Postgres\n(django_celery_beat_periodictask /\n_crontabschedule)"] + D --> E["DatabaseScheduler polls Postgres\nand picks it up automatically —\nno beat restart needed"] + E --> F{"Need to change the time,\nenable/disable, or force\na one-off run later?"} + F -->|yes| G["Edit the PeriodicTask / CrontabSchedule\nlive in the django-unfold admin\n('Periodic Tasks')"] + G --> H["Takes effect on beat's next tick —\nno deploy needed"] + F -->|"no, need a permanent\nfresh-install default too"| I["Also edit the seed migration —\nit only affects DBs that haven't\napplied it yet"] +``` + +**Three things worth calling out:** + +1. **The migration is a one-time seed, not a sync.** It runs `get_or_create`/`update_or_create` + once, on first apply. Editing the migration file after it has already run on a given + database (dev, prod) does nothing to that database's existing row — you have to either + edit the row live (admin) or write a follow-up migration, the same way + `newsletter/migrations/0002_repoint_digest_beat.py` did to repoint the digest tasks' + dotted path after they moved apps (see § Interfaces below). +2. **To see the schedule that's actually running right now**, don't read a settings file — + open the django-unfold admin's "Periodic Tasks" page (or `PeriodicTask.objects.all()` + in a shell), on the database you actually care about (dev vs. prod are different + Postgres instances with potentially different live edits). +3. **A schedule change made in the admin is real and permanent for that database**, but + invisible to anyone reading the seed migrations in the repo. If prod's actual fire time + for a task ever needs to match what's in git, someone has to check both places. + +### 2.3 One beat tick, and why `last_run_at` can lie + +This is the flow behind the sharpest edge in this whole layer, so it earns its own +diagram. `django-celery-beat`'s `DatabaseScheduler` keeps `PeriodicTask.last_run_at` in +memory while it runs, and only writes it back to Postgres when Celery's own +`Scheduler._do_sync()` fires: + +```mermaid +sequenceDiagram + autonumber + participant Beat as celerybeat process + participant Mem as Beat's in-memory schedule + participant Broker as Redis DB 0 + participant Worker as commons-default worker + participant PG as Postgres (PeriodicTask row) + + Beat->>Mem: wake at the scheduled time (e.g. 04:00 ET) + Mem->>Broker: send task message (ingest-events-daily) + Mem->>Mem: update last_run_at in memory + Broker->>Worker: deliver message + Worker->>Worker: run_ingestion_pipeline() + Beat->>PG: sync last_run_at (forced after every task send,\nCELERY_BEAT_SYNC_EVERY = 1) +``` + +**Four things worth calling out:** + +1. **`CELERY_BEAT_SYNC_EVERY = 1` (in `backend/settings/base.py`) is why step 6 happens + right after every task send instead of on a timer.** Without it, the only sync triggers + are `CELERY_BEAT_MAX_LOOP_INTERVAL` (currently `6 * 60 * 60`, i.e. six hours — this + setting caps how long beat may sleep between waking to re-poll Postgres for schedule + *changes*, not how often it fires due tasks) and a 3-minute time-based trigger baked + into Celery itself. A task that fires shortly after the last sync could sit unflushed + in memory for up to six hours. +2. **This is not hypothetical — it's a confirmed 2026-07-29 prod incident** + (`docs/ingestion-monitoring.md`, "Beat scheduler: last_run_at persistence lag"). Before + `CELERY_BEAT_SYNC_EVERY = 1` was added, `broadcast-orphan-recovery` was firing exactly + on schedule every six hours, but the healthcheck kept reporting it `STALE` because + `last_run_at` in Postgres was still showing the *previous* run — the write was buffered + in memory and only flushed when celerybeat happened to restart. The task was fine; the + bookkeeping column was lying. +3. **Never widen a staleness window to paper over this.** The tempting fix when a + monitoring check false-alarms is to make the window bigger. That hides the false alarm + but also means a *real* multi-hour outage now takes that much longer to surface. The + actual fix is making the persisted value trustworthy (`CELERY_BEAT_SYNC_EVERY = 1`), not + giving the check more slack to be wrong in. +4. **If `CELERY_BEAT_SYNC_EVERY` is ever unset or set back to a falsy value, this lag + comes back exactly as it was** — `backendServer/events/management/commands/healthcheck.py`'s + `crontab_grace_seconds()` knows this and automatically widens its own grace window to + match `CELERY_BEAT_MAX_LOOP_INTERVAL` when that happens, so the healthcheck won't + immediately start crying wolf again — but the underlying lag itself is back, silently. + +### 2.4 Diagnosing "the job didn't run" + +`manage.py healthcheck` (run hourly on the VM by `healthcheck.timer` → `healthcheck.service`, +and manually via `bash deploy/healthcheck.sh`) is the tool for this. Its periodic-task +check doesn't use a single hand-tuned staleness window for every task — a window wide +enough to never false-alarm on a *weekly* task would let a missed *daily* task ride along +silently for over a week. Instead, wherever a task has a crontab, it asks the crontab +itself what the next expected fire time was and compares that to now: + +```mermaid +flowchart TD + A["For each enabled PeriodicTask"] --> B{"last_run_at is\nnull?"} + B -->|yes| FAIL1["FAIL — enabled, never run yet"] + B -->|no| C{"Task has a\ncrontab schedule?"} + C -->|no, interval or none| D{"Task name has an entry\nin DEFAULT_STALENESS_HOURS?"} + D -->|no| WARN1["WARN — no staleness window\nconfigured for this task"] + D -->|yes| E{"now minus last_run_at\nexceeds the configured window?"} + E -->|yes| FAIL2["FAIL — STALE"] + E -->|no| OK1["OK"] + C -->|yes| F["Derive expected next fire\nfrom the crontab itself\n(remaining_estimate, tz-corrected)"] + F --> G{"Expected fire\nalready passed?"} + G -->|no| OK2["OK — reports the derived next-fire time"] + G -->|yes| H{"Overdue by more than\ncrontab_grace_seconds()?\n(~5 min if BEAT_SYNC_EVERY is set,\nelse ~7h)"} + H -->|no| WARN2["WARN — overdue but within grace,\ncould be persistence lag, not a real miss"] + H -->|yes| FAIL3["FAIL — MISSED"] +``` + +**Three things worth calling out:** + +1. **`DEFAULT_STALENESS_HOURS` (in `healthcheck.py`) plays a narrower role than its name + suggests.** It's not the primary freshness check for the four crontab-backed seeded + tasks (`ingest-events-daily`, `scrape-sources-daily`, `weekly-digest-sunday`, + `broadcast-orphan-recovery`) — those are judged against their own crontab's expected + fire time. It's used for two narrower things: flagging a seeded task that's missing + from the schedule entirely (a FAIL, regardless of staleness), and as the only signal + available for a hypothetical interval-scheduled or schedule-less task, which has no + crontab to derive an expected time from. +2. **The grace window in step H exists because of § 2.3's persistence lag, and shrinks + automatically once that lag is fixed.** With `CELERY_BEAT_SYNC_EVERY = 1` (current + prod config), the grace is ~5 minutes — just enough for scheduler jitter and the task's + own run time. If that setting is ever lost, the grace widens itself back out to match + `CELERY_BEAT_MAX_LOOP_INTERVAL` so a `FAIL` doesn't start firing on every ordinary sync + delay. +3. **A `WARN` here is not "probably fine" — it means "cannot yet distinguish a real miss + from bookkeeping lag."** Don't dismiss it as noise; if a `WARN` on the same task + persists across multiple healthcheck runs (an hour or more apart), that's no longer + plausibly persistence lag and is worth treating as a real miss. + +--- + +## 3. Data model + +There's no domain data model here in the usual sense — the state this layer owns is +scheduling and caching metadata, not business data: + +| Table (owner) | Key fields | What it means | +|---|---|---| +| `django_celery_beat_periodictask` (`django_celery_beat`) | `name` (unique, human label like `ingest-events-daily`), `task` (dotted Python path, e.g. `ingestion.tasks.run_ingestion_pipeline`), `crontab` FK, `enabled`, `last_run_at`, `total_run_count` | One row per scheduled job. `task` is a plain string, not a real import reference — nothing breaks at write time if it points at a function that no longer exists; it just fails loudly the next time beat tries to send it. This is exactly why moving a task to a new app requires a *data migration* to repoint this column (§ 2.2), not just moving the Python code. | +| `django_celery_beat_crontabschedule` (`django_celery_beat`) | `minute`/`hour`/`day_of_week`/`day_of_month`/`month_of_year`, `timezone` | The crontab a `PeriodicTask` points at. `timezone` is stored per-row (`America/New_York` for the ingest/scrape/digest tasks, `UTC` for broadcast orphan recovery) — beat uses `TzAwareCrontab` (see § 5) to honor it. | +| Redis DB 0 keys (broker/result backend) | Celery-internal — task messages, results | Not meant to be read directly; `redis-cli -n 0 KEYS '*'` works for debugging but there's no app-level schema here worth documenting. | +| Redis DB 1 keys (`events/cache.py`) | `events:list:version` (an integer, bumped on every `Event`/write); `events:list:v{N}:{sha256-prefix-of-sorted-query-params}` (TTL 60s); `events:towns`, `events:categories` (plain keys, TTL 1h) | The read-endpoint cache. **The version-in-the-key trick exists because Django's stdlib `RedisCache` backend has no `delete_pattern`** — there's no way to invalidate "every cached event-list page" by pattern, so instead every write bumps a version counter and old-version keys just age out via TTL instead of ever being explicitly deleted. `Town`/`Category` writes clear their own plain keys directly instead, since those aren't parameterized by query string. | + +Invalidation is signal-driven, not task-driven: `events/signals.py`, registered in +`EventsConfig.ready()`, listens for `post_save`/`post_delete` on `Event`, `Town`, and +`Category` and calls the corresponding `events/cache.py` invalidation function inline, +synchronously, in the request/transaction that made the write — there's no Celery task +involved in cache invalidation at all. + +--- + +## 4. Interfaces + +### Periodic (beat-scheduled) tasks — the live schedule as seeded + +| `PeriodicTask.name` | Task (dotted path) | What it does | Queue | Cadence | Worker that drains it | +|---|---|---|---|---|---| +| `ingest-events-daily` | `ingestion.tasks.run_ingestion_pipeline` | Full ICS-leg pipeline: cleanup → poll ICS sources → Gemini standardize → dedup → safety-score → auto-publish | default | 04:00 daily, `America/New_York` | `commons-default` (`celery.service`) | +| `scrape-sources-daily` | `ingestion.tasks.scrape_all_sources_task` | Polls `scraper`- and `http`-type sources (fires 30 min before the daily ingest so rows land in time) | scrape | 03:30 daily, `America/New_York` | `commons-scrape` (`scrape-worker.service`) | +| `weekly-digest-sunday` | `newsletter.tasks.fan_out_weekly_digest` | Queues one `send_one_digest` per WEEKLY subscriber/user-profile | default | Sundays 18:00, `America/New_York` | `commons-default` | +| `monthly-digest-first` | `newsletter.tasks.fan_out_monthly_digest` | Queues one `send_one_digest` per MONTHLY subscriber/user-profile | default | 1st of month, 18:00, `America/New_York` | `commons-default` | +| `broadcast-orphan-recovery` | `broadcast.tasks.recover_broadcast_orphans` | Re-queues `BroadcastSubmission`/`BroadcastTarget` rows stranded by a crashed worker | broadcast | every 6 hours on the hour, `UTC` | `commons-broadcast` (`broadcast-worker.service`) | + +The two digest rows are the ones with history worth knowing: both were originally seeded +(by `events/migrations/0015_seed_digest_beat.py` and `0020_seed_monthly_digest_beat.py`) +pointing at `events.tasks.fan_out_weekly_digest`/`fan_out_monthly_digest`. When the digest +engine moved to the `newsletter` app, a follow-up data migration +(`newsletter/migrations/0002_repoint_digest_beat.py`) updated the `task` string on those +*same* rows to `newsletter.tasks.fan_out_weekly_digest`/`fan_out_monthly_digest` — the +`PeriodicTask.name` and its `CrontabSchedule` never changed, only the dotted path it +dispatches to. `docs/redis-celery-handoff.md` referred to the pre-move path; that's been +corrected as part of this pass. + +### On-demand tasks (not on any beat schedule) + +| Task | Queue | Triggered by | +|---|---|---| +| `ingestion.tasks.publish_all_approved_task` | default | `POST /api/events/publish-approved` (API key), and the admin's "Publish approved events" docs page | +| `ingestion.tasks.ingest_direct_submission_task` | default | `POST /api/events/direct-submit` — the broadcast SPA's host-submission flow | +| `newsletter.tasks.send_one_digest` | default | Called once per recipient by `fan_out_weekly_digest`/`fan_out_monthly_digest` — never called directly from a view | +| `broadcast.tasks.process_broadcast_queue` | broadcast | `transaction.on_commit(...)` inside `broadcast/services.py`, after `POST /broadcast/submit`, `/retry`, or `/submit-real` commits | +| `events.tasks.ping` | default | Nothing in the running app calls this — it exists purely as a smoke-test task (`events/tests/test_tasks_fast.py`), not part of the healthcheck's worker-liveness probe (that uses Celery's own `control.ping()` RPC, a different mechanism with the same name) | + +### Local dev commands + +```bash +# alongside runserver +uv run celery -A backend worker -l info # default queue +uv run celery -A backend beat -l info # scheduler +uv run celery -A backend worker -Q scrape -c 1 -l info # scrape queue +uv run celery -A backend worker -Q broadcast -c 1 -l info # broadcast queue +``` + +The test suite doesn't need any of this running — `backend/settings/test.py` sets +`CELERY_TASK_ALWAYS_EAGER = True`, so every `.delay()` call executes inline, synchronously, +in the test process, and exceptions propagate immediately instead of vanishing into a +worker that isn't there. + +### Adding a new periodic task, correctly + +1. Write the `@shared_task` function in the owning app's `tasks.py`. `autodiscover_tasks()` + (called from `backend/celery.py`) finds it automatically — no registration step. +2. If it needs to stay off the default queue (heavy browser work, or anything that + shouldn't share a process with digests/ingestion), add an entry to `CELERY_TASK_ROUTES` + in `backend/settings/base.py`, and give it a dedicated worker unit if it's going to run + in prod unattended (follow the pattern in `deploy/scrape-worker.service`). +3. Write a data migration in the owning app that does the `CrontabSchedule.get_or_create` + / `PeriodicTask.update_or_create` seed — copy the shape of e.g. + `ingestion/migrations/0012_seed_scrape_beat.py`. Pick the timezone deliberately: + `America/New_York` for anything user-facing/business-hours-sensitive, `UTC` for + internal crash-recovery-style jobs (that's the actual distinction the two existing + choices encode, not an arbitrary pick). +4. Run the migration wherever the task needs to actually be live (dev DB, then prod on + deploy). Nothing else needs restarting for the schedule itself to take effect — beat + picks up new/changed rows on its own poll of Postgres. +5. If the task ever moves to a different app afterward, its dotted `task` path has to be + repointed with a *second* data migration (see the digest fan-out example in § Interfaces + above) — moving the Python file alone silently orphans the schedule, since + `PeriodicTask.task` is just a string beat doesn't validate until send time. +6. Add it to `DEFAULT_STALENESS_HOURS` in `events/management/commands/healthcheck.py` only + if it has no crontab (interval-based) — crontab-backed tasks get their freshness judged + automatically from the crontab itself (§ 2.4). + +--- + +## 5. Sharp edges + +**`CELERY_BEAT_MAX_LOOP_INTERVAL` (6 hours) and `CELERY_BEAT_SYNC_EVERY` (1) work together, +and tuning one without the other silently reopens a monitoring blind spot.** Both live in +`backend/settings/base.py`. `CELERY_BEAT_MAX_LOOP_INTERVAL = 6 * 60 * 60` caps how long +beat may sleep before re-polling Postgres for schedule *changes* — a deliberate trade-off +to reduce Neon wake-ups, and worth keeping. `CELERY_BEAT_SYNC_EVERY = 1` is the fix for the +side effect that setting has on `last_run_at` bookkeeping: without it, a fired task's +`last_run_at` can sit unflushed in memory for up to that same six hours (§ 2.3), which is +exactly what happened in the confirmed 2026-07-29 prod incident. Never change a +staleness/monitoring window that reads `last_run_at` without checking both of these +settings' current values first — as of this commit they're `6 * 60 * 60` and `1` +respectively. + +**`TzAwareCrontab.remaining_estimate()` has a timezone bug that `is_due()` on the same +class doesn't.** `TzAwareCrontab` (from `django_celery_beat.tzcrontab`, the class every +seeded crontab actually uses) overrides `is_due(last_run_at)` to convert its argument into +the schedule's own timezone (`last_run_at.astimezone(self.tz)`) before delegating to the +base Celery `crontab` class — that conversion is what makes an `America/New_York` schedule +actually fire at the right wall-clock time. It does **not** override +`remaining_estimate()`, which is inherited unmodified from `celery.schedules.crontab` and +compares whatever `datetime` it's handed directly against the crontab's fields, with no +conversion at all. In real beat operation this is invisible — beat only ever calls +`is_due()`, and its own `nowfunc()` already returns time in the schedule's tz, so both +sides of every comparison beat makes are already in the same zone. It becomes a real bug +the moment anything calls `remaining_estimate()` directly with a UTC datetime, which is +exactly what `manage.py healthcheck`'s freshness check does (§ 2.4) — without an explicit +`.astimezone(schedule.tz)` conversion on both operands first, `"0 4 * * *"`/ +`America/New_York` gets silently read as "04:00 UTC," landing the expected-fire estimate +four to five hours off **every single day**, not just around a DST transition. The +healthcheck code already carries this conversion (see the comment in +`_crontab_freshness` in `healthcheck.py`) — the edge to remember is that it's a property of +*that call site*, not of `TzAwareCrontab` in general: any new code that calls +`remaining_estimate()` directly (rather than going through beat's own `is_due()` machinery) +has to redo the same conversion by hand, or it will be wrong in exactly this way. + +**Beat schedules are database rows, not code — the file you'd grep for the current cron +time can be lying.** A seed migration only writes its `CrontabSchedule`/`PeriodicTask` once, +on first apply, and django-unfold admin edits made afterward change the row live without +ever touching a migration file. Reading `ingestion/migrations/0007_seed_ingest_beat.py` +tells you what the schedule *was* the day that migration was written and applied — it does +not tell you what's actually firing in prod right now if anyone has since edited it in the +admin. § 2.2 covers how to check the live truth. + +--- + +## 6. Known gaps and doc drift + +- **`ARCHITECTURE.md`'s Broadcast section is stale**, per § 1 above — it says broadcast + "does not use Celery," which was true before the queue-drain and orphan-recovery logic + moved into `broadcast/tasks.py`'s `@shared_task` functions. The Async section further + down the same file is accurate; only the Broadcast section wasn't updated to match. +- **`docs/redis-celery-handoff.md`'s "Testing" section says the suite runs under dev + settings with `CELERY_TASK_ALWAYS_EAGER = False`.** That's backwards from what's actually + configured: the suite runs under `backend.settings.test` (which inherits from `dev.py` + but then explicitly sets `CELERY_TASK_ALWAYS_EAGER = True`), so tasks run inline by + default and the `@override_settings` pattern shown there is only needed if a test + deliberately wants *non*-eager behavior, not the reverse. This wasn't in scope to correct + on that file for this pass (only its stale digest task-path references were) — flagging + it here so it doesn't get propagated further. +- **This doc did not independently verify `docs/redis-celery-handoff.md`'s prod + provisioning commands** (Redis install/config, systemd `cp`/`enable` sequence) — they + read as plausible and internally consistent with the service files in `deploy/`, but + weren't re-run against a live VM for this pass. Treat that doc as authoritative for the + exact commands; this doc is about the shape of the system, not a runbook. +- **No consumer of `events.tasks.ping` exists outside its own test.** It's either a leftover + smoke-test scaffold from before `manage.py healthcheck`'s `control.ping()`-based worker + check existed, or an intentionally-kept minimal example task — nothing in the codebase + says which, and it wasn't worth guessing at here. diff --git a/human-docs/auth.md b/human-docs/auth.md new file mode 100644 index 0000000..94ae72f --- /dev/null +++ b/human-docs/auth.md @@ -0,0 +1,353 @@ +# The Auth Bridge — Better Auth and Django + +*Reflects commit `5fe7a45`, 2026-08-01. Written by reading `backendServer/accounts/` (models, +admin, urls, views, migrations), `backendServer/backend/jwt_auth.py`, +`backendServer/backend/permissions.py`, `backendServer/backend/settings/` (`__init__.py`, +`base.py`, `dev.py`, `prod.py`), `theCommonsWeb/src/lib/auth.ts`, `auth-schema.ts`, +`auth-client.ts`, `db.ts`, `theCommonsWeb/src/hooks/useAuth.tsx`, the portal route group +(`src/app/(portal)/`), `src/app/reset-password/`, `src/lib/redirect-allowlist.ts`, and +`broadcastWeb/src/App.tsx`. Complements `ARCHITECTURE.md`'s Authentication section, which is +also current as of this commit; the two shouldn't drift, and if they ever do, trust the code. +Companion docs referenced below: `overview.md` (system map), `data-model.md` (full schema), +`deploy-ops.md` (VM/nginx/systemd), `frontend.md` (Next.js app structure).* + +The single fact that matters more than any other in this codebase: **Django does not own user +accounts.** There is no `django.contrib.auth.User` for app users, no Django login view, no +Django-issued session. Identity lives entirely inside the Next.js app (`theCommonsWeb`), owned +by an embedded library called **Better Auth**. Django's job is narrower and more mechanical than +a newcomer usually assumes: it keeps a read-only mirror of Better Auth's tables so it can `JOIN` +against them, and it verifies a token that Better Auth issued. It never creates a user, never +checks a password, and never has the authority to invalidate a session. If you find yourself +about to add a login endpoint to Django, or wondering why `python manage.py createsuperuser` +doesn't create an account regular users can sign in with, that's the inversion this doc exists +to correct. + +## 1. What this is and who depends on it + +Two systems share one identity: **Better Auth**, running inside `theCommonsWeb`'s Next.js +process, and Django's `accounts` app, which mirrors Better Auth's tables just enough to join +against them and verify the tokens Better Auth mints. Every other backend app that needs to know +"who is this request from" — `events`, `newsletter`, `broadcast`, `ingestion` — depends on +`accounts` for that answer; none of them talk to Better Auth directly. + +Concretely, three things depend on this bridge working: + +- **The main site** (`theCommonsWeb`) — every authenticated page (`/profile`, `/dashboard`, + `/post`) needs both a live Better Auth session (for the UI) and a valid JWT (to call Django's + API for profile data, event ownership, and business listings). +- **The broadcast SPA** (`broadcastWeb`) — a separate Vite app on its own subdomain that + authenticates against the *same* Better Auth instance via a shared cookie, so a user who signs + in on the main site is already signed in there, and vice versa. +- **Django itself** — `BearerTokenAuthentication` (`backend/permissions.py`) is the + authentication class nearly every non-public DRF view in this repo declares. If JWT + verification breaks, every one of those views starts rejecting real users, and the failure + mode looks nothing like "auth is down" — see Sharp edge 1. + +If this bridge is down, users can typically still sign in on the frontend (Better Auth doesn't +depend on Django), but every page that calls the Django API for personalized data — profile, +dashboard, event submission, business listings, all of `broadcast/` — starts failing or serving +generic/anonymous responses. + +## 2. How it works + +### 2.1 Signing in or creating an account + +Every place in the product that needs a user to authenticate — the main site's header, the +broadcast SPA's "Sign in" button, the old `/auth` legacy routes — does the same thing: a full +browser navigation to the portal (`/signin` or `/join`, on `theCommonsWeb`, at the shared auth +origin) with a `?redirect_to=` query param. Sign-up and sign-in converge on the +same downstream steps once Better Auth accepts the credentials, so this is one diagram with an +`alt` for the branch that differs — account creation also fires a database hook that inserts a +matching Django-side profile row. + +```mermaid +sequenceDiagram + autonumber + participant Browser + participant PortalUI as Portal UI (JoinForm / SignInForm, via useAuth) + participant BetterAuth as Better Auth (theCommonsWeb, /api/auth/*) + participant PG as Postgres + participant Django + + Browser->>PortalUI: GET /join or /signin?redirect_to=... + alt Creating an account + PortalUI->>BetterAuth: authClient.signUp.email(email, password, user_type) + BetterAuth->>PG: INSERT neon_auth.user, neon_auth.account (provider=credential) + BetterAuth->>PG: databaseHooks.user.create.after - INSERT public.events_userprofile + Note over BetterAuth,PG: best-effort - a failed mirror insert is caught and logged,
never rolled back, since Better Auth already committed the login-usable account + else Signing in + PortalUI->>BetterAuth: authClient.signIn.email(email, password) + BetterAuth->>PG: look up neon_auth.account (provider=credential), verify password + end + BetterAuth-->>Browser: Set-Cookie session, cross-subdomain when BETTER_AUTH_COOKIE_DOMAIN is set + PortalUI->>BetterAuth: GET /api/auth/token (cookie sent) + BetterAuth-->>PortalUI: short-lived JWT (jwt() plugin - sub=user id, email claim) + PortalUI->>Django: GET /events/me/profile, Authorization Bearer JWT + Django-->>PortalUI: profile JSON (verification internals: next diagram) + PortalUI->>Browser: window.location.href = resolveRedirect(redirect_to) + Note over Browser: full navigation, not a client route change - lands back on the calling app,
whose own useAuth re-resolves session + JWT + profile independently on mount +``` + +**Five things worth calling out about this picture:** + +1. **The JWT is fetched and re-fetched from scratch, never stored.** `useAuth.tsx` keeps the JWT + only in React state — nothing is written to `localStorage` or a non-httpOnly cookie. Every + fresh page load (including the one after the portal's full navigation back) calls + `authClient.getSession()` then `/api/auth/token` again. That's simple and safe against XSS + token theft, but it means a slow or failing `/api/auth/token` call blocks every authenticated + page load, not just sign-in. +2. **The account-creation side effect can silently fail without blocking signup.** The + `databaseHooks.user.create.after` hook in `theCommonsWeb/src/lib/auth.ts` inserts into + `public.events_userprofile` wrapped in a try/catch specifically so a schema hiccup on the + Django side can never roll back the `neon_auth.user`/`account` rows Better Auth just + committed — a user must never be left with a login that immediately breaks. The trade-off is + a `UserProfile`-less `BetterAuthUser` if that insert ever does fail, which most `accounts` + views assume can't happen (`me`, `businesses`, etc. all `.filter(user_id=...).first()` and + 404 rather than crash, so this degrades gracefully, but there's no reconciliation job that + backfills the missing row). +3. **The physical table name is a historical fossil, not a bug.** That `INSERT INTO + public.events_userprofile` targets a table whose Django model is `accounts.UserProfile` + today — the app moved in a state-only migration (§5, Sharp edge 5) but the table's physical + name never changed, and this raw-SQL insert (written before the move, unchanged since) + still says `events_userprofile` on purpose. +4. **Cross-subdomain cookies are conditional, not a repo-wide constant.** `BETTER_AUTH_COOKIE_DOMAIN` + is only set to `.thecommons.town` in production; when it's unset (local dev), Better Auth's + default `SameSite=Lax`, host-scoped cookie applies instead, so a session on `localhost:3000` + won't be visible to a broadcast dev server on `localhost:5173` — you sign in separately in + dev, and that's expected, not broken. In prod, the cookie is scoped to the whole + `.thecommons.town` domain with `SameSite=None; Secure`, which is what lets someone who signs + in on the apex site show up already signed in on `broadcast.thecommons.town` — and what lets + `broadcastWeb`'s "Sign in" button (`App.tsx`) just navigate to + `${VITE_BETTER_AUTH_URL}/signin?redirect_to=` and rely on the shared cookie to + bring the user right back in. +5. **Password sign-up is the only path today.** `emailAndPassword: { enabled: true, autoSignIn: + true }` in `auth.ts` is the entire signup surface — `signUp.email` creates the credential + account and signs the user in in the same call. Google sign-in exists in the codebase + (`socialProviders.google`) but is commented out; there's no passwordless flow left either + (see Sharp edge 4). + +### 2.2 Django verifying a Bearer JWT against JWKS + +This is the other half of the bridge, and the one that actually enforces anything: every +protected DRF view in this repo uses `BearerTokenAuthentication` +(`backend/permissions.py`), which accepts either the shared `THE_COMMONS_API_KEY` (no user +attached — for server-to-server calls like the ingestion pipeline) or a Better Auth JWT, +verified statelessly against Better Auth's published JWKS endpoint. Django never calls back into +Next.js to ask "is this session valid" — it downloads Better Auth's public signing keys once, +caches them in-process, and verifies the JWT's signature locally. + +```mermaid +sequenceDiagram + autonumber + participant Client + participant View as DRF view + participant Auth as BearerTokenAuthentication + participant JWTAuth as jwt_auth.verify_better_auth_jwt + participant JWKS as Better Auth JWKS endpoint + participant DB as neon_auth.user (Postgres) + + Client->>View: Authorization Bearer token + View->>Auth: authenticate(request) + alt token equals THE_COMMONS_API_KEY + Auth-->>View: (None, None) - authorized, no user attached + else Better Auth JWT + Auth->>JWTAuth: verify_better_auth_jwt(token) + alt JWKS client cached and under 600s old + Note over JWTAuth: reuse the in-process PyJWKClient, no network call + else cache cold, expired, or never fetched + JWTAuth->>JWKS: GET jwks.json, browser User-Agent header set explicitly + JWKS-->>JWTAuth: signing keys, cache refreshed + Note over JWTAuth: on fetch failure, reuse the last-known client for up to
1h stale-grace before returning None (fail closed) + end + JWTAuth->>JWTAuth: jwt.decode against the resolved signing key (EdDSA/RS256/ES256) + alt no client, or decode fails + JWTAuth-->>Auth: None + Auth-->>View: raise AuthenticationFailed - Invalid token + else decode succeeds + JWTAuth-->>Auth: claims (sub, email, ...) + Auth->>DB: BetterAuthUser.objects.get(id=claims.sub) + DB-->>Auth: user row + Auth-->>View: (user, claims) + end + end +``` + +**Three things worth calling out about this picture:** + +1. **The browser User-Agent header on the JWKS fetch is load-bearing, not decoration** — see + Sharp edge 1. It looks like the kind of line a cleanup pass removes. +2. **The stale-grace fallback exists so a Next.js deploy or blip doesn't cascade into Django + auth.** For up to an hour after the last successful JWKS fetch, a failing refresh reuses the + old signing keys rather than rejecting every request. Past that window, verification fails + closed (`None`), which surfaces to callers as `AuthenticationFailed`. +3. **`broadcast/` calls `verify_better_auth_jwt` directly**, skipping the `BetterAuthUser` ORM + lookup step entirely (`broadcast/access.py`) — by design, to keep `broadcast/` from importing + anything outside itself. It gets back claims, not a Django user object, and resolves an + access tier from the email claim instead. + +## 3. Data model + +`accounts` owns two kinds of tables: five **Better Auth mirrors** it never writes to, and two +**Django-owned profile tables** that hang off them. + +| Model | Schema / table | Managed by Django | What it's for | +|---|---|---|---| +| `BetterAuthUser` | `neon_auth.user` | No (`managed = False`) | The account record — id, email, name. Hardcodes `is_authenticated = True` / `is_anonymous = False` as class attributes so DRF's permission classes treat it as a real authenticated user without a database round trip for that check. | +| `BetterAuthSession` | `neon_auth.session` | No | Better Auth's server-side session record. Django never reads this for request auth — auth is JWT-based, not session-based, on the Django side. | +| `BetterAuthAccount` | `neon_auth.account` | No | One row per sign-in method (`provider_id = 'credential'` for password auth). Holds the hashed password for credential accounts. `user_id` is a `UUIDField` — see Sharp edge 3. | +| `BetterAuthVerification` | `neon_auth.verification` | No | Better Auth's internal token bookkeeping (password-reset tokens, etc). | +| `BetterAuthJwks` | `neon_auth.jwks` | No | The signing keypair(s) Better Auth's `jwt()` plugin uses. Django never reads this table directly — it fetches the public half over HTTP from the JWKS endpoint instead (§2.2), not from the database. | +| `UserProfile` | `public.events_userprofile` | Yes | `user_type` (LOCAL/BUSINESS/VENUE), `primary_city`, `address`, `email_preference`, tag interests. `OneToOneField` to `BetterAuthUser` with `db_constraint=False` (no DB-level FK across the schema boundary onto an unmanaged table). | +| `BusinessProfile` | `public.events_businessprofile` | Yes | Business-account listing data — name, description, contact info, service area, publish state. Also `OneToOneField` to `BetterAuthUser`, `db_constraint=False`. | + +Every `neon_auth` model's `db_table` uses a double-quote trick to cross a schema boundary Django +doesn't natively support in this form — `db_table = 'neon_auth"."user'` — so that when Django +wraps it in its own quoting (`"` + value + `"`), the emitted SQL comes out as +`FROM "neon_auth"."user"`, a valid Postgres cross-schema reference. It reads like a stray quote +character; it isn't one. + +The full event/newsletter/ingestion/broadcast schema (and how those tables relate to the ones +here) is `data-model.md`'s job, not this doc's — this table only covers the identity slice. + +## 4. Interfaces + +| Method | Path | Auth | Notes | +|---|---|---|---| +| GET/PATCH | `/auth/me` | Bearer JWT | Read/update the caller's own `UserProfile` — email preference, city, address, tags, and (once, LOCAL to BUSINESS/VENUE only) account type. PATCH also syncs a `NewsletterSubscriber` row to match `email_preference`. | +| GET/POST | `/businesses` | Bearer JWT | Browse published businesses (VENUE accounts only) / create a listing (BUSINESS accounts only). | +| GET | `/businesses/me` | Bearer JWT | Caller's own business listing. | +| GET/PATCH/DELETE | `/businesses/` | Bearer JWT | Business listing CRUD, owner-only for write. | +| GET/POST | `/api/auth/*` | — | Not a Django route — Better Auth's own catch-all handler in `theCommonsWeb` (`src/app/api/auth/[...all]/route.ts`). This is where `/api/auth/token` (JWT mint), `/api/auth/jwks` (public keys), and the password/reset-password endpoints actually live. | + +Every `accounts` view above uses `BearerTokenAuthentication` + DRF's `IsAuthenticated` — there is +no Django session auth and no CSRF token involved, by design (see §1). A request with no +`Authorization` header, an expired JWT, or a JWT whose `sub` doesn't resolve to a +`BetterAuthUser` all end up as `401`; a valid user hitting an endpoint gated on the wrong +`user_type` (e.g. a LOCAL account calling `POST /businesses`) gets `403`, not `401` — worth +knowing when a bug report says "I'm logged in but I get an error." + +## 5. Sharp edges + +**1. The JWKS fetch needs a browser-like User-Agent, or every JWT verification fails.** +Cloudflare sits in front of the auth origin and 403s the default `python-urllib` User-Agent that +`PyJWKClient` sends by default. `backend/jwt_auth.py` sets an explicit +`_JWKS_HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; TheCommons/1.0)"}` on both the +pre-flight `requests.get` and the `PyJWKClient` itself specifically to route around this. If +someone "cleans up" that header as unnecessary boilerplate, the JWKS fetch starts getting 403'd +by Cloudflare, `_get_jwks_client` returns `None` (once any stale-grace window expires), +`verify_better_auth_jwt` returns `None` for every token, and every Bearer-JWT-protected endpoint +in the app starts returning 401/403 for real, legitimately-signed-in users. This is exactly what +happened once in production — broadcast's tier-2 endpoints started blanket-403ing, and the +symptom (auth suddenly broken everywhere) looked nothing like the cause (one HTTP header). + +**2. `DJANGO_ENV` has to be exactly `prod` for the production settings to load — and the +failure mode for getting this wrong is still partly silent.** `backend/settings/__init__.py` +resolves `DJANGO_ENV` to `dev` or `prod` before anything else loads: + +```mermaid +flowchart TD + A[DJANGO_ENV read from the environment] --> B{Value} + B -->|unset or blank| C[dev.py loads - DEBUG True, ALLOWED_HOSTS is localhost only] + B -->|prod, case-insensitive, trimmed| D[prod.py loads - DEBUG False, ALLOWED_HOSTS from DJANGO_ALLOWED_HOSTS] + B -->|anything else, e.g. production| E[ImproperlyConfigured raised at import time, process fails to start] + C --> F[If this happens on the public host: every request 400s with DisallowedHost, site looks like it has no events] +``` + +A *typo'd* value now fails loudly (`ImproperlyConfigured` at process startup) — that part used +to be silent and has since been hardened, with a regression test +(`events/tests/test_config_fast.py::SelectSettingsEnvTests`) locking the behavior in. What's +*still* true, and still by design (every local dev setup relies on it), is that an **unset** +`DJANGO_ENV` silently resolves to `dev`. If the production VM's `.env` is ever missing that line +entirely, the app comes up as `dev.py` — `DEBUG=True`, `ALLOWED_HOSTS=["localhost", "127.0.0.1"]` +— and every request to the real public host gets rejected with `DisallowedHost` (HTTP 400), +which the frontend renders as "no events." `manage.py healthcheck --require-prod` now exists +specifically to catch this after the fact (it fails loud if `DEBUG` is `True` or `ALLOWED_HOSTS` +is localhost-only) and is wired into the periodic health check, but it's a detector, not a +guard — it doesn't stop a misconfigured deploy from going live, it just makes the misconfig +visible fast instead of only when someone notices the site is empty. + +**3. `BetterAuthAccount.user_id` is a `UUIDField`, matching the real column — this used to be +wrong.** The live `neon_auth.account."userId"` column is `uuid`. The Django mirror model +originally declared it as a `TextField`, which worked fine for simple lookups but broke any ORM +join across it — `operator does not exist: uuid = text` — because Postgres won't implicitly +compare those types. This was fixed by changing the field to `UUIDField(db_column="userId")`, +and there's now a standing regression test +(`BetterAuthAccountUserIdFieldTests` in `events/tests/test_config_fast.py`) asserting the field +type matches. If someone reverts that field back to `TextField` — plausible, since a lone +`get()`-by-id lookup on this model still works fine with either type and gives no visible signal +anything's wrong — any code doing a cross-model anti-join (exactly the shape the passwordless +account rollover, Sharp edge 4, needs) breaks again at query time. Because this model is +`managed = False`, nothing about the database enforces the Django field type matches reality; +only the test does. + +**4. Password-required accounts and the passwordless rollover: wiring is done, the actual send +looks outstanding.** Before password requirements landed, an account could exist with no +password at all — sign-in was just "enter your email." That flow is gone: `SignInForm.tsx` +requires both fields, and there is no other way to reach a session without one. Anyone whose +account predates that change is locked out with no password to type. The fix that's since landed +threads all the way through: `theCommonsWeb/src/lib/auth.ts` now wires +`emailAndPassword.sendResetPassword` (sends the reset email directly via Brevo's REST API from +the Next.js server), `/forgot-password` calls `authClient.requestPasswordReset` for real instead +of showing a stub message, and a new public `/reset-password` page consumes the token via +`authClient.resetPassword`. The identification side is also built: +`events.management.commands.rollover_passwordless_accounts` finds affected users with a raw-SQL +anti-join against `neon_auth` (deliberately raw SQL, not the ORM — the `neon_auth` mirror tables +aren't built in the test database, so there's no way to exercise an ORM version of this query in +CI) and, with `--send`, emails each one a rollover notice. What isn't confirmed from the repo: +whether that command has actually been run with `--send` against production yet. Its own +docstring, and the runbook it points to (`docs/suite-38-passwordless-rollover.md`), still +describe the reset link as a dead end and the whole effort as blocked — that prose predates the +wiring above and is now stale; don't trust it over the code in `auth.ts` and +`ForgotPasswordForm.tsx`/`ResetPasswordForm.tsx`. Before ever passing `--send`, verify the +current prerequisites for real (an unset `BREVO_API_KEY` in the frontend's production +environment makes `/forgot-password` report success while sending nothing), and treat the +management command's dry-run output as the actual list of who's still stranded. + +**5. `neon_auth` is never migrated by Django, and the app-extraction migration that moved +`UserProfile`/`BusinessProfile` into `accounts` is deliberately state-only.** Every `neon_auth` +mirror model carries `managed = False` for exactly one reason: Better Auth owns those tables +completely, and a Django `migrate` must never touch them. If that flag were ever dropped or +flipped — "cleaning up" what looks like a stray `False` — the next `makemigrations` would try to +generate real `CreateModel`/`AlterField` operations against tables that already exist with a +different owner, and `migrate` would then attempt DDL against `neon_auth.user`, +`neon_auth.account`, and the rest, which at best fails outright and at worst silently diverges +Django's idea of the schema from the one Better Auth is actually writing to — breaking +authentication for the whole app, not just the mirror. Related, and worth reading correctly: the +migration that moved `UserProfile`/`BusinessProfile` (and all five mirrors) into the `accounts` +app, `accounts/migrations/0001_initial.py`, is built entirely from +`migrations.SeparateDatabaseAndState` with `database_operations=[]` — it *looks* like it creates +seven tables (there's a `CreateModel` for every one of them, mirrors included), but zero DDL +actually runs. It only changes which Django app the ORM believes owns each model; every +`db_table` (`events_userprofile`, `events_businessprofile`, and every `neon_auth.*` mirror) stays +physically exactly where it was. A newcomer skimming that migration file and assuming it built +those tables would be wrong in a way that matters if they ever try to reason about what +`migrate` did or didn't touch. + +## 6. Known gaps + +- **The passwordless-rollover command and its guardrail test still live under `events/`, not + `accounts/`**, even though they exist entirely to work around a bug in `accounts.models`. + `events/management/commands/rollover_passwordless_accounts.py`'s own docstring still says + `BetterAuthAccount.user_id (events/models.py)` — that model has lived in `accounts/models.py` + since the app extraction (§5). Neither file was moved when `accounts` was carved out of + `events`; functionally harmless (the import path in the code itself is correct — only the + comment and the file's location are stale), but worth relocating alongside any future touch of + either file. +- **Whether the passwordless-rollover email has actually been sent to production users is not + verifiable from the repository.** See Sharp edge 4 — the wiring is confirmed done, the send is + not confirmed either way. +- **No email verification.** Signup doesn't gate on a verified email for MVP (`emailAndPassword` + has no `requireEmailVerification`, and there's no verify-email route in + `theCommonsWeb/src/app`). Confirmed by absence rather than by finding an explicit "no + verification" flag — worth a second look if this doc is read much later than its date above. +- **Google sign-in is fully wired in Better Auth's config but commented out** + (`socialProviders.google` in `auth.ts`), and the client-side popup flow that used to call it + was deleted along with the pre-portal auth UI. Re-enabling it needs a new post-OAuth + account-type step built into the portal (it used to bypass the LOCAL/BUSINESS/VENUE choice + entirely) — not just uncommenting the block. +- **Production nginx routing for the auth origin was not verified for this doc.** This repo's + deploy tooling was mid-change while this doc was written (a Docker-based nginx config was + present but not yet part of a committed state), so the exact current routing for + `auth.thecommons.town` is `deploy-ops.md`'s claim to verify, not this one's — everything above + about cookies, trusted origins, and the JWKS URL is grounded in application code and env-var + wiring, not in reading the live nginx config. diff --git a/human-docs/broadcast.md b/human-docs/broadcast.md new file mode 100644 index 0000000..95b4eaa --- /dev/null +++ b/human-docs/broadcast.md @@ -0,0 +1,346 @@ +# Broadcast — Pushing Events Out + +Written 2026-08-01 against commit `5fe7a45`. This is the human-facing companion to +[`docs/broadcast.md`](../docs/broadcast.md), which stays the system of record — the exact +model fields, every API endpoint's rate limit, the full adapter list, environment variables, +and management commands all live there in more detail than belongs here. Read this first for +orientation, then go to `docs/broadcast.md` for anything precise. Where the two disagree, the +code (and `docs/broadcast.md`, which was re-verified against it) wins — this doc calls out the +one place that matters below. + +Audience: someone inheriting this codebase who has never touched the broadcast subsystem and +needs to understand what it is before the deep-dive doc makes sense. + +--- + +## 1. What this is and who depends on it + +Ingestion (see `ingestion.md`) pulls events *in* — it polls other towns' calendars and pulls +their events onto The Commons. Broadcast does the reverse: it takes a single event and pushes +it *out* onto several other towns' calendar websites, so a partner organization — a chamber of +commerce, a venue, an event host — can list one event once and have it land on The Commons plus +half a dozen third-party sites, instead of retyping the same event into six different forms by +hand. + +The people who use it are not residents browsing the public site. They're partner +organizations and event hosts, working through a separate console (`broadcastWeb`) gated +behind an access code or a login. If broadcast is down, nothing on the main site +(`theCommonsWeb`) breaks — it's an entirely separate flow with its own frontend, its own +backend app, and its own database rows. What breaks is a partner's ability to distribute an +event listing; the event itself can still be posted directly to The Commons through the normal +submission form. + +Architecturally, broadcast is deliberately walled off from the rest of the backend. The +`broadcast` Django app is not allowed to import anything from `events` or `ingestion` — a real +test (`broadcast/tests/test_isolation.py`) walks every `.py` file in the app with Python's +`ast` module and fails the suite if it finds an import rooted at either package. This isn't +incidental tidiness. Broadcast operates on its own denormalized copy of an event's fields +(title, datetimes, venue, locality tags, category tags — all just columns on +`BroadcastSubmission`, not a foreign key to `events.Event`), on its own locality/category +vocabulary (`broadcast/routing.py` defines its own list of towns and categories rather than +reading `events.Town`/`events.Category`), and its own access-control system entirely separate +from Better Auth accounts. The one place broadcast reaches back toward the rest of the site is +one-directional and lives on the other side of the wall: when a host submits an event through +the broadcast console, the SPA also fires a request at an ingestion endpoint +(`POST /api/events/direct-submit`) that runs the event through the same LLM standardization +pipeline as a scraped event. That bridge code lives entirely inside `ingestion/` — `broadcast/` +still imports nothing from `ingestion/` or `events/`. If someone "fixes" the isolation by +having `broadcast/` reach into `events.Town` directly to reuse a vocabulary, the isolation test +fails immediately, but the deeper cost is that a future change to `events.Town` (renaming a +slug, deleting a town) would silently start breaking broadcast's routing logic in a way nobody +watching the `events` app would think to check for. + +--- + +## 2. Who operates it, and how + +Three pieces work together: + +- **`broadcastWeb`** (`broadcastWeb/`) — a Vite + React single-page app, separate from + `theCommonsWeb` entirely (different build, different deploy, different test suite). This is + where a partner fills out an event, picks which calendars to send it to, and watches the + status of each site. +- **`broadcastExtension`** (`broadcastExtension/`) — a Chrome MV3 browser extension that the + SPA talks to. It's the thing that actually opens each third-party calendar's submission form + and fills it in. +- **`broadcast`** (`backendServer/broadcast/`) — the Django app underneath both. It decides + which calendars a given event is eligible for, builds the per-site "recipe" the extension + fills in, gates all of this behind an access-tier system, and also contains a second, + currently-unused submission path described in §4. + +### Signing in and getting access + +There's no separate broadcast account system — signing in uses the same Better Auth identity +as the main site (see `auth.md`), via the shared cookie domain across `thecommons.town` +subdomains. But being logged in isn't enough by itself; broadcast additionally gates every +feature behind a **tier** (0, 1, or 2), resolved per request from either a Bearer JWT or an +access-code header. Tier 0 (logged in, no grant, or not logged in at all) can't fill or +broadcast anything. Tier 1 can fill and broadcast. Tier 2 additionally unlocks the AI-autofill +helper that turns pasted free text into a draft event. + +There are two independent ways to get above tier 0, and they're deliberately kept apart: + +- A **trial code**, entered anonymously with no account required. It's always tier 2, and + instead of being metered by number of uses it just expires — three days by default. This is + the "hand someone a code so they can try it right now" path. +- An **upgrade code**, which only works for a logged-in user (`POST /broadcast/redeem`) and + permanently sets that account's tier. Whoever enters the code last wins — there's no + downgrade protection, which is a deliberate choice so sales/support can just say "enter this + code" without worrying about someone accidentally reverting a customer's access. + +The one text field in the SPA does double duty depending on login state — logged out it reads +"Access Code" and resolves a trial code anonymously; logged in it relabels to "Upgrade Account" +and permanently redeems an upgrade code against that account. `broadcast/access.py` is where +this resolution actually happens; `docs/broadcast.md`'s Access control section has the full +tier table and every metering rule. + +--- + +## 3. How an event actually gets onto another town's calendar + +This is the live flow — the one a partner uses today. + +```mermaid +sequenceDiagram + autonumber + participant Partner + participant SPA as broadcastWeb (SPA) + participant API as Django (broadcast app) + participant Ext as broadcastExtension + participant Site as Third-party calendar site + + Partner->>SPA: Fill out event form + SPA->>API: POST /broadcast/preview (event data) + API-->>SPA: eligible sites / excluded sites + reason + SPA-->>API: POST /api/events/direct-submit (fire-and-forget) + Note over API: ingestion bridge — standardizes the event\ninto the normal pipeline in parallel, non-blocking + + Partner->>SPA: Select sites, click "Autofill with extension" + loop per selected site + SPA->>API: POST /broadcast/direct-recipe (event data, site_key) + API-->>SPA: recipe JSON (fields, selectors, values — no DB row created) + SPA->>Ext: sendFill(extensionId, recipe) + Ext->>Site: open new tab at recipe.url, fill every field but the submit button + Ext-->>SPA: fill acknowledged + end + Partner->>Site: Review prefilled form, solve captcha, click Submit +``` + +**Five things this diagram can't say on its own:** + +1. **This is the primary, and today the only reachable, submission path.** The current + `broadcastWeb` UI has no button that creates a job for the server-side headless path + described in §4 — the code that would call `POST /broadcast/submit` isn't wired to anything + in `App.tsx` anymore. A partner's event never becomes a `BroadcastSubmission`/ + `BroadcastTarget` database row under this flow; `direct-recipe` is explicitly a read-through + endpoint that computes a recipe and creates nothing. +2. **The recipe request and the actual autofill are two separate round trips per site**, + because the recipe is computed fresh from whatever's currently in the form (no server-side + state to go stale) and the extension is a different origin the SPA can only reach by + messaging. +3. **The extension never clicks Submit.** That's not a missing feature — it's the whole point. + Every recipe-driven site has either a captcha, a terms checkbox, or both, and the design + intentionally leaves the final click (and any captcha-solving) to the human who can see the + filled-in form and vouch for it. +4. **The direct-submit call in step 3 runs in parallel with everything else and can fail + silently without affecting the visible flow.** It exists so a broadcast submission also + becomes a real event on The Commons itself, going through the identical LLM + standardize/dedupe/safety-score pipeline described in `ingestion.md` — but a failure there + never blocks or surfaces as an error in the preview step. +5. **Not every eligible-looking site gets this treatment.** Four Tier-1 sites + (`fun4raleighkids`, `chapelboro`, `explore_pittsboro`, `shop_pittsboro`) require a login + before any form appears, or are otherwise not deterministically fillable — their adapters + carry no recipe at all, and `SitePicker` in the SPA greys them out with "coming soon" instead + of letting a partner select them. + +### The adapter pattern + +Each third-party calendar has its own **adapter** — a small Python class in +`broadcast/adapters/` (`abc11_community.py`, `triangle_on_the_cheap.py`, `visit_raleigh.py`, +and so on) that knows that one site's form: its field selectors, what format each field expects +a date or a price in, and which locality/category combination the site is even willing to +accept (`routing.py` uses this to compute the eligible/excluded split in step 2 of the diagram +above). An adapter is intentionally dumb by design — it never invents content, never calls an +LLM at request time, and never solves a captcha; if a required field can't be resolved from the +event data, or the site presents a login wall or a captcha, the result is "needs manual +attention," not an invented guess. + +What makes an adapter usable by the extension flow is a second, declarative layer on top of the +same field definitions: `RecipeField` objects (selector, type, how to resolve a value from the +event) that get serialized straight into the recipe JSON the extension consumes. The two layers +share the same field/selector definitions on purpose, so the imperative fill logic (used by the +disabled headless path in §4) and the recipe JSON (used by the live extension path) can't drift +apart into filling a form two different ways. Six adapters currently expose a full recipe — +`abc11_community`, `triangle_on_the_cheap`, `triangle_weekender`, `visit_raleigh`, +`chatham_arts`, `chatham_chamber` — and those six are exactly the sites listed in the +extension's `host_permissions`. The remaining Tier-1 adapters exist (so routing can still +report them as "eligible, but not automatable") but return no recipe. + +--- + +## 4. The other path: server-side headless submission (disabled today) + +The `broadcast` app also contains a complete, independent second way of doing this, built +first and still fully present in the code: a database-backed job queue, a Playwright-driven +headless Chromium worker, and the same adapters' *imperative* `fill_and_submit` methods (as +opposed to their declarative `recipe()` output). `POST /broadcast/submit` creates a +`BroadcastSubmission` plus one `BroadcastTarget` row per selected site, a Celery task drains +the queue, and a worker process launches a real (usually headless) browser per target and +drives the form itself — no human in the loop unless an adapter reports `needs_manual`. + +```mermaid +flowchart TD + Submit["POST /broadcast/submit\n(NOT called by the current SPA UI)"] --> Rows["BroadcastSubmission + one BroadcastTarget\nper site created, status=queued"] + Rows --> Dispatch["transaction.on_commit ->\nprocess_broadcast_queue.delay (Celery)"] + Dispatch --> Claim["worker.claim_next:\nSELECT FOR UPDATE SKIP LOCKED\noldest queued submission -> status=running"] + Claim --> Loop{"For each pending target,\nin site_key order"} + Loop --> Launch["Launch one Chromium session\n(no ORM calls inside sync_playwright)"] + Launch --> Fill["adapter.fill_and_submit(page, event, ctx)"] + Fill --> Outcome{"Result?"} + Outcome -->|"required field\nresolves fine"| Succeed["status=succeeded\nexternal_url recorded"] + Outcome -->|"captcha / login wall /\nmissing field"| Manual["status=needs_manual\n(legacy recipe fetch still\nworks via GET .../manual/site_key)"] + Outcome -->|"adapter raised"| Fail["status=failed, error recorded"] + Succeed --> Next{"More targets?"} + Manual --> Next + Fail --> Next + Next -->|yes| Loop + Next -->|no| Done["submission.status = done or failed"] +``` + +**This path is real, tested, and currently unreachable from the SPA.** `broadcastWeb`'s +`App.tsx` still has handlers for retrying a job, promoting a dry-run job to a real submission, +and canceling a job — but nothing in the current UI ever calls `POST /broadcast/submit` to +*create* one, so those handlers only matter for a job that already exists. It's worth stating +plainly because two of the repo's own root-level docs disagree about this: `ARCHITECTURE.md`'s +Broadcast section narrates this headless path as if it were the live flow, and separately +claims broadcast "does not use Celery." Neither is accurate as of this commit — trust +`docs/broadcast.md` and the code instead, both of which agree the extension flow in §3 is +primary and this one is dispatched over a real Celery queue (see §5). + +Why keep a whole second, disabled path around instead of deleting it? Because it's the honest +long-term answer for sites without a usable recipe, or for a future bulk/unattended mode — the +architecture supports both a human-supervised and a fully automated submission without +duplicating the per-site form logic, since both paths are adapter methods on the same class. + +--- + +## 5. The worker, and why its concurrency is fixed at one + +Both the headless path's queue-drain and its crash-recovery sweep run as real Celery tasks +(`broadcast.tasks.process_broadcast_queue` and `broadcast.tasks.recover_broadcast_orphans`), +routed to a dedicated `broadcast` Celery queue via `CELERY_TASK_ROUTES` in +`backend/settings/base.py`. In production, exactly one worker process drains that queue — the +`broadcast-worker` systemd unit runs `celery -A backend worker -Q broadcast -c 1`, concurrency +pinned to one. + +That `-c 1` is load-bearing, not a conservative default someone forgot to tune up. The +crash-recovery task assumes that any `BroadcastSubmission` still marked `running` when it runs +must have been orphaned by a worker that died mid-drain — a safe assumption only if there is +never more than one worker that could have been mid-drain in the first place. Scale that worker +to two or more processes and this assumption breaks: a submission actually being processed by +worker B could get its `running` status matched by orphan-recovery's sweep and get re-queued +out from under worker B, which would then finish and write results for a submission that a +second, freshly-spawned attempt is now also racing to process. The obvious-looking fix for "the +queue is slow" — bump the worker's concurrency — is exactly the wrong move here; the right fix +is a second single-concurrency worker on its own separate queue, not a wider pool on this one. + +This queue is not polled on an interval. `services._dispatch_worker()` calls +`transaction.on_commit(process_broadcast_queue.delay)` right when a submission is created, +retried, or promoted from dry-run to real — so a job starts draining the instant its database +transaction commits, not on the next tick of a fixed poll loop. A separate, coarser safety net — +`recover_broadcast_orphans`, seeded as a periodic task running every six hours — exists purely +to catch a submission stranded by a worker that crashed mid-drain between one partner's session +and the next; it is not how ordinary progress happens. Day-to-day stalled-target recovery is +actually driven by the SPA's own polling loop, which flags a target that's been sitting +`queued`/`in_progress` too long and asks the backend to re-queue just that target, capped at a +couple of automatic retries before surfacing a "stuck" state to the partner. `async-jobs.md` +covers the full Celery queue layout (the ingestion and digest queues too) in depth; this +section is scoped to what's specific to broadcast. + +--- + +## 6. Access codes, briefly + +Access codes are managed entirely through the database — there is no environment-variable code +list to update on deploy. Codes are generated from the Django admin (self-serve, shows the raw +code once, right after creation, then never again) or from a management command. `data-model.md` +has the full field-by-field reference for `AccessCode`, `AccessCodeUse`, and +`AccessCodeRedemption`; the one thing worth internalizing here is that **trial** and +**upgrade** codes are two genuinely separate pools distinguished by `AccessCode.kind`, not two +labels on the same mechanism — a trial code is only ever checked against the anonymous +header/body path, an upgrade code only ever against the logged-in redeem endpoint, and neither +code type will ever validate against the other's path. There's also a small evergreen layer on +top (`SalesCodeSlot`) that keeps exactly one always-current, always-visible code per tier +showing in the admin, specifically so a salesperson never has to run a CLI command to hand +someone a working code. + +--- + +## 7. Sharp edges + +1. **Isolation from `events`/`ingestion` is enforced by a real test, and it's easy to break + without noticing.** `broadcast/tests/test_isolation.py` parses every `.py` file in the app + with Python's `ast` module and fails if any file imports from `events` or `ingestion` at any + depth — not just direct top-level imports. Reaching for `events.Town` to avoid duplicating + broadcast's own locality list, or importing an `ingestion` helper to reuse some parsing + logic, will fail CI immediately if it's caught — and if it somehow isn't (a new test file, + a refactor that moves code around), the real cost lands later, as a change to `events` or + `ingestion` silently breaking broadcast in a way nobody reviewing that change would think to + check. + +2. **No Django ORM call may happen while a `sync_playwright()` block is open.** `runner.py`'s + docstring explains why: Playwright keeps its own event loop alive on the thread it runs on, + and Django's async-aware connection handling can silently start using a second database + connection inside that block instead of the one the surrounding code expects. The fix in + place is structural — `run_submission` fetches everything it needs into a plain + `CanonicalEvent` object *before* opening a browser session, and the browser is opened and + closed once per target, with every database write happening in between sessions rather than + inside one. Any change that tries to look something up from the database mid-fill (to avoid + an extra round trip, say) reintroduces this specific hazard, and it's the kind of bug that + only shows up under real concurrent load, not in a quick manual test. + +3. **The `broadcast` Celery queue's single-worker constraint is a correctness requirement, not + a performance knob** — covered in full in §5. Worth repeating here as its own edge because + it's the one most likely to look like an easy fix to someone trying to speed up a slow + broadcast queue. + +4. **The Chrome extension only autofills hosts explicitly listed in `manifest.json`'s + `host_permissions`.** Today that list is exactly the six recipe-enabled adapters' domains + plus `api.thecommons.town` (needed so the extension's service worker, which isn't bound by a + page's CORS policy, can fetch an event's self-hosted image before attaching it to a form — + see `BroadcastImage` in `data-model.md`). Add a seventh recipe-enabled adapter without adding + its domain to `host_permissions`, and nothing throws an error anywhere: the extension simply + never runs its content script on that tab, no banner appears, no field gets filled, and the + most likely diagnosis from an operator's chair is "the extension is broken" rather than "the + manifest is missing an entry." Adding a `host_permissions` entry also isn't free once the + extension is actually published to the Chrome Web Store — it triggers a mandatory re-review, + and existing installs won't pick up the new permission until each user individually + re-approves it, so there's a real rollout gap between shipping the fix and it actually + reaching partners already using the extension. + +5. **The two on-record disabled/dormant framings mean different things and it's worth not + conflating them.** The extension is "dormant" in the sense that its content script never + runs during ordinary browsing — it only activates when the SPA explicitly messages it, which + is a deliberate security property (visit any of the target calendar sites directly and + nothing happens). The headless Playwright path in §4 is "disabled" in a completely different + sense — its code is complete and tested, but nothing in the current SPA UI can trigger it. + Neither means "unfinished" or "safe to delete." + +--- + +## 8. Not independently verified + +- Whether the extension has actually been submitted to and approved on the Chrome Web Store, or + is still distributed as a load-unpacked developer build to partners. `broadcastWeb/.env` has + a real-looking published extension ID alongside a dev-unpacked one, and the extension's own + README documents the Web Store submission process as something the account owner runs + manually — this doc did not confirm whether that submission has actually happened. +- The actual production behavior of the headless path end-to-end (a real Chromium run against a + live third-party form) — its code was read, not exercised, for this doc. + +--- + +See also: `data-model.md` for the full `broadcast` model field reference, `async-jobs.md` for +the complete Celery queue and beat-schedule layout, `deploy-ops.md` for the `broadcast-worker` +systemd unit and how it's deployed, and `ingestion.md` for the pipeline the direct-submit +bridge in §3 feeds into. diff --git a/human-docs/containerization.md b/human-docs/containerization.md new file mode 100644 index 0000000..ed105e2 --- /dev/null +++ b/human-docs/containerization.md @@ -0,0 +1,305 @@ +# Containerization — status + +> **PROVISIONAL — written 2026-08-01, against commit `5fe7a45`.** Every file +> this doc describes was **uncommitted** in the working tree at the time of +> writing (tracked internally as Suite 42, status "Needs QA"): `docker-compose.yml`, +> `docker-compose.override.yml`, `backendServer/Dockerfile`, `Dockerfile.frontend`, +> both `.dockerignore` files, `deploy/nginx/` (`Dockerfile` + `thecommons.conf`), +> `docs/adr/0001-containerization.md`, and a Docker-first rewrite of `DEPLOY.md`. +> Treat anything here as accurate only up to that commit and that uncommitted +> snapshot — if the working tree has moved since, re-read the source files +> before trusting a claim in this doc over the code. +> +> **Built and verified locally end-to-end. Not deployed.** The production VM +> has no Docker Engine installed and still runs the systemd units in `deploy/` +> — nothing described below is live. This doc is a placeholder: once the +> cutover actually lands, it should either be promoted into a proper handoff +> report or folded into `human-docs/deploy-ops.md` (see the closing section). +> `human-docs/deploy-ops.md` covers what's running today, in production, right +> now — this doc covers what's coming. Read that one first if you're trying to +> fix something in prod this minute. + +## What was containerized, and why + +The Commons runs today as seven long-lived processes hand-wired as individual +systemd units on a single Oracle Cloud VM, plus a hand-edited nginx config +with no local-dev equivalent at all. `docs/adr/0001-containerization.md` is +the actual design record for the work described here — this section +summarizes it; read the ADR for the full reasoning and the incidents that +motivated it. + +Four decisions anchor the design. First, **nginx moves into a container and +becomes the single ingress** for all three subdomains, replacing the +host-installed nginx package entirely. The Cloudflare origin cert stays a +read-only bind mount rather than something baked into an image layer — a +rotated cert shouldn't require a rebuild, and a private key has no business +inside something that could be pulled or inspected off the VM. The +consequence worth remembering: gunicorn moves from a unix socket to plain +TCP, because a socket path can't cross a container boundary the way nginx and +gunicorn share one today. That's a strictly weaker isolation posture — any +container on the compose network can now reach port 8000 — accepted as the +standard shape for a containerized nginx-plus-app pair, not an oversight. + +Second, **Redis moves into a container** (`redis:7-alpine`), which is also +the only piece of async infrastructure that currently has zero local-dev +story — it's apt-install-only today. Nothing about the DB-0/DB-1 broker/cache +split changes: it was never hardcoded, it falls straight out of `REDIS_URL` +and `REDIS_CACHE_URL`, so pointing both at a containerized Redis with the +same `/0` and `/1` suffixes preserves it untouched. `requirepass` is kept, +sourced the same way it is today, and the container gets a named volume +because DB 0 holds in-flight Celery task state that an unpersisted restart +would silently drop. + +Third, **Postgres stays external, on Neon, in every environment** — it was +never a candidate for the compose stack. Neon already provides the actual +restore mechanism (branching, point-in-time recovery); a containerized +Postgres alongside that would just be a second, weaker source of truth for +local dev only. The one consequence: the CI pipeline's pre-migrate `pg_dump` +safety net, which today depends on a host-installed `postgresql-client`, +moves to a throwaway `postgres:18-alpine` container invoked for the duration +of the dump and discarded. + +Fourth, **build artifacts get baked into images; only real, unrecoverable +state gets a volume.** `collectstatic` output and the compiled broadcast SPA +bundle are both regenerated from source on every build, so baking them in is +strictly safer than a volume — a volume here would let a stale build survive +a fresh deploy. Client-uploaded media, the pre-migrate backup dumps, and the +Playwright debug screenshots/downloads are all volumes, because losing any of +them on a restart destroys something no rebuild can recreate. + +Worth noting separately: containerizing this stack also closes the exact +failure class that took the async stack down for eight days starting +2026-07-21 (`docs/prod-incident-2026-07-21-scheduler-outage.md`) — a +snap-packaged `uv run` spawning its child inside a systemd user-session scope +that a post-deploy SSH logout tore down. There's no snap, no `logind`, no +per-user systemd slice inside a container's PID namespace for a lost SSH +session to tear down, so that mitigation (exec the venv binary directly, plus +`loginctl enable-linger`) becomes moot by construction rather than by fix. +One piece of it is still carried forward for an unrelated reason: every +compose service execs its real binary directly rather than through a wrapper +shell, so the container's PID 1 receives `SIGTERM` on `docker stop` and shuts +down cleanly. + +## Service topology + +The diagram below shows what starts immediately, what waits on what, and +why — not every dependency edge is drawn individually where five services +share an identical gate; see the callouts underneath. + +```mermaid +flowchart LR + subgraph immediate["No dependencies - start immediately"] + redis[redis] + migrate[migrate - one-shot, exits 0] + nextjs[nextjs] + spabuild[broadcast-spa-build - one-shot, exits 0] + end + + subgraph gated["Wait for redis healthy AND migrate exited 0"] + backend[backend] + celery[celery] + celerybeat[celerybeat] + bworker[broadcast-worker] + sworker[scrape-worker] + end + + subgraph ingress["Waits for backend, nextjs, broadcast-spa-build to have started"] + nginx[nginx] + end + + redis -- healthy --> gated + migrate -- exit 0 --> gated + backend --> nginx + nextjs --> nginx + spabuild --> nginx +``` + +Four things the diagram can't say on its own: + +1. **`migrate` doesn't wait on Redis, deliberately.** It only ever talks to + Neon over `DATABASE_URL`, so it's grouped with `redis` in the "starts + immediately" tier rather than gated behind it — the compose file's own + comment calls this out ("not a Celery service, so it has no reason to wait + on redis"). +2. **All five gated services share the identical two-part gate**, even though + only the Celery ones use Redis as a broker in the traditional sense. + `backend` (gunicorn) is gated the same way because Django will 500 on + nearly every route without a completed migration — not because gunicorn + itself needs Redis to boot. (It does read `REDIS_CACHE_URL` for the cache + backend, but nothing in this dependency graph enforces that the cache + actually round-trips before `backend` starts serving.) +3. **`nginx`'s dependency is the weakest of the three tiers.** Compose's + default `depends_on` condition — used here with no explicit condition — is + "has started," not "is healthy" or "is ready to accept traffic." `nginx` + can come up and start proxying to a `backend` or `nextjs` container that + is still mid-initialization, unlike the `redis`/`migrate` gate ahead of + it, which explicitly waits on health and exit status. +4. **`broadcast-spa-build` and `migrate` are both `restart: "no"` one-shots + that are supposed to exit 0 immediately.** `docker compose ps` showing + them `Exited (0)` is correct behavior, not a crash — a newcomer running + `ps` for the first time and seeing two "stopped" containers among a list + of "running" ones could easily read that as a partial failure. + +### Compose service to systemd unit + +| Compose service | Replaces | Notes | +|---|---|---| +| `redis` | `redis-server.service` (apt-installed; no unit file was ever tracked in this repo) | `requirepass` moves from `/etc/redis/redis.conf` into the compose `command:` block, sourced from `REDIS_PASSWORD` in `backendServer/.env`. | +| `migrate` | New — previously a manual `manage.py migrate` step run by hand during deploy | Runs once per deploy as an exiting container instead of a step in a deploy script; also seeds the `django_celery_beat` schedule tables. | +| `backend` | `gunicorn.service` (its unit file, like `nextjs.service`, was apparently never checked into `deploy/` — only referenced by name in `DEPLOY.md` and the ADR) | Switches from a unix socket to TCP `8000`, internal to the compose network only. | +| `celery` | `deploy/celery.service` | Same `ExecStart` command, minus the exec-direct/linger workaround — see the ADR's historical note above. | +| `celerybeat` | `deploy/celerybeat.service` | Same command; still exactly one process, `DatabaseScheduler` keeps the schedule in Postgres. | +| `broadcast-worker` | `deploy/broadcast-worker.service` | Same command, `-c 1` concurrency preserved (mandatory — `recover_orphans()` assumes a single worker). | +| `scrape-worker` | `deploy/scrape-worker.service` | Same command, `-c 1` to keep headless-Chromium memory off the default worker. | +| `nextjs` | `nextjs.service` (also not tracked under `deploy/`) | Unchanged behavior: `node server.js` on port 3000, internal only. | +| `broadcast-spa-build` | New — previously a manual `pnpm run build` step whose `dist/` nginx served directly off the VM filesystem | Build-only helper; never runs as a long-lived process (see diagram callout 4). | +| `nginx` | The host-installed nginx package plus the hand-edited `/etc/nginx/sites-available/thecommons` (previously only tracked as a snippet, `deploy/nginx-broadcast.conf.snippet`) | Full config now lives at `deploy/nginx/thecommons.conf` and is baked into the image; the TLS cert stays a bind mount, never baked in. | +| — | `deploy/healthcheck.service` + `.timer` | **Not replaced.** Still a host-level systemd timer; `deploy/healthcheck.sh` now shells out to `docker compose ... exec` to reach the containers, and degrades to a `WARN` rather than crashing when Docker isn't installed on the host at all. | + +Two things stood out while building this table, worth flagging rather than +silently smoothing over: `docs/adr/0001-containerization.md` describes "seven +long-lived systemd units," but only four of them (`celery`, `celerybeat`, +`broadcast-worker`, `scrape-worker`) actually have a tracked unit file under +`deploy/` — `gunicorn.service`, `nextjs.service`, and `redis-server.service` +exist only by reference in `DEPLOY.md` and the ADR, not as files in this +repo. Not a contradiction, just a gap in what's checked in versus what runs +on the box. + +## Sharp edges already known + +**The override file is a loaded footgun on the VM.** `docker-compose.yml` +alone is the production config. A bare `docker compose` invocation with no +`-f` flag auto-loads `docker-compose.override.yml` from the same directory — +that is Compose's default two-file-merge behavior, not a bug — and that +override is explicitly local-dev-only: plain HTTP nginx with no TLS cert at +all, bind mounts remapped to repo-relative `.local-dev/` paths instead of the +real `/home/ubuntu/broadcast/...` directories, and `DJANGO_ENV` forced to +`dev`. Running that combination against the VM would not fail loudly — nginx +would happily bind port 80 in plain HTTP, Django would boot under dev +settings, and the Redis URLs would point at an unauthenticated connection +string that doesn't carry the real `REDIS_PASSWORD` even though the actual +Redis container still expects it (since `.env`'s `REDIS_PASSWORD` is +untouched by the override). The result is a mix of wrong TLS, wrong +hostnames, and a Redis auth mismatch, discovered only once someone notices +the site is serving unencrypted or Celery has gone silent. Every production +command — in `DEPLOY.md`, in CI, run by hand — passes `-f +docker-compose.yml` explicitly for exactly this reason. + +**The `.dockerignore` at the repo root exists as much for secrets as for +build speed.** The repo root holds a real private key (`oraclevps.key`) and +`broadcastWeb/` holds a committed certificate. The file's own header records +an empirically verified quirk of this Docker Desktop/buildkit version: a +pattern with no leading `**/` only matches at the exact context-root path, +even with no slash in the pattern — a bare `*.pem` did not exclude a nested +`broadcastWeb/commons-broadcast.pem` in a real test build, only `**/*.pem` +did. That's stricter than plain `.gitignore` semantics, where a no-slash +pattern matches at any depth, and it means every secret-adjacent pattern in +that file deliberately carries the `**/` form rather than the shorter one +someone might "simplify" it down to later. + +**`WORKDIR` plus `COPY --chown` does not chown a pre-existing directory.** +`backendServer/Dockerfile` sets `WORKDIR /app` as root before switching to a +non-root user, and `COPY --from=builder --chown=app:app /app /app` only sets +ownership on the files it copies in — the `/app` directory itself, created +earlier by `WORKDIR`, stays root-owned unless something says otherwise. The +Dockerfile carries an explicit `chown app:app /app` specifically to avoid +`collectstatic` failing with `EACCES` trying to create +`staticfiles_build/` under a directory it can't write to — this is prior +history in this repo, not a hypothetical, and the fix is already in place, +but the trap reappears instantly if a future edit reorders those two lines. + +**A Docker Desktop proxy can corrupt large apt fetches as "Hash Sum +mismatch."** Both stages of `backendServer/Dockerfile` that touch apt write +`Acquire::Retries "3"` and disable HTTP pipelining before installing +anything, specifically because the Playwright stage's `--with-deps` fetch is +large enough to trip this locally, and the failure reads like package +corruption on a different package each run rather than what it actually is — +a proxy mangling a large or pipelined response. Anyone debugging an +inexplicable, non-reproducible apt failure inside these images should look +here before assuming a broken mirror. + +**Building `nginx` in isolation doesn't work.** `deploy/nginx/Dockerfile` +consumes two Buildx named build contexts — `backend-static` and +`broadcast-spa`, bound in `docker-compose.yml` to the `backend` and +`broadcast-spa-build` services respectively — to pull in `collectstatic` +output and the compiled broadcast bundle. A standalone `docker build -f +deploy/nginx/Dockerfile deploy/nginx` has no way to resolve those names and +fails; `docker compose build nginx` (verified to build the two named-context +services first even when only `nginx` is requested) is the only supported +path. + +**Frontend build args are a separate mechanism from runtime env, and mixing +them up produces a build that "succeeds" with the wrong values baked in.** +`nextjs` and `broadcast-spa-build` inline their `NEXT_PUBLIC_*`/`VITE_*` +values at build time via Compose variable interpolation, which only reads +the shell environment or a root-level `.env` next to `docker-compose.yml` — +it cannot read `theCommonsWeb/.env.local` or `broadcastWeb/.env` directly, +even though those are the files that actually hold the real values. Every +build arg has a safe placeholder default so `docker compose build` always +succeeds with zero setup, which is exactly the trap: a real prod build that +skips sourcing the real files and re-exporting them under the +`NEXTJS_BUILD_*`/`BROADCAST_BUILD_*` names will build cleanly and ship a +broadcast bundle silently misrouting every API call. CI already greps the +compiled bundle for a `thecommons.town` origin as a guard, but that guard +only exists in the CI job — a manual build on the VM has no equivalent +safety net. + +## What's left before this is real + +The three blockers already known going in: `backendServer/.env`'s +`REDIS_URL` and `REDIS_CACHE_URL` still point at `localhost`/`127.0.0.1`, +which inside a container is the container itself — Celery would silently +connect to nothing and no task would ever run, while everything synchronous +stayed green, the same failure shape as the 2026-07-21 outage. Both need to +repoint at the `redis` compose service with the real password embedded. +`DJANGO_ALLOWED_HOSTS` must be present in that same file — `prod.py` does a +bare `os.environ["DJANGO_ALLOWED_HOSTS"]` with no default, so its absence +crashes every backend/Celery container at boot, not just one. And the VM +itself needs one-time prep — Docker Engine and the Compose v2 plugin +installed, and the deploy user added to the `docker` group — before the +first automated container deploy can run at all. + +Beyond those three, reading the actual compose file and CI workflow surfaced +more: + +- **The CI deploy job is already wired for containers, and the VM isn't + ready for it.** `.github/workflows/ci.yml`'s `deploy` job — gated on + `needs: [backend, frontend-commons, frontend-broadcast]` and triggered only + on a push to `main` — already runs `docker compose -f docker-compose.yml + build` and `up -d` over SSH. That means the very next merge to `main` after + this branch lands will fail that step outright (`docker: command not + found`) unless the VM prep above happens first. This fails loudly rather + than silently — the deploy job errors out — but it's still a live landmine + worth flagging before anyone merges, not something to discover from a red + CI run. +- The persistent host directories the compose file bind-mounts into + (media, broadcast screenshots/downloads, the backups directory) need to + exist and be writable by uid 1000 before the first `up`, matching the + images' non-root `app` user. +- The Cloudflare origin cert needs to already be in place at + `/etc/ssl/cloudflare/thecommons.town.{pem,key}` for the container nginx to + bind 443 at all — this isn't new (the same cert the host nginx uses today), + but the container won't start without it being reachable at that exact + path. +- The actual cutover moment — stopping the host nginx and starting the + container one — is a single step where two processes briefly cannot both + hold ports 80/443, and it's the one step in the whole plan where a mistake + produces an immediate, visible outage rather than a quiet misconfiguration. + It has not been rehearsed against the real VM; everything verified so far + has been local-only. +- Nothing here has been run against the actual Oracle VM's resource limits (1 + OCPU / 6 GB) — six long-running containers plus two Playwright-capable + images is a meaningfully different memory footprint than seven bare + systemd processes, and that has not been measured, only asserted via + `mem_limit` guesses on the Celery workers. + +## When to delete this doc + +This file is provisional by design and should not accumulate history. Once +the VM cutover has actually happened and been verified — containers running +in production, the old systemd units retired, the blockers above resolved — +do one of two things, not both: either promote this into a proper handoff +report under `human-docs/` with its own verified sharp edges from real +production operation, or fold whatever's still true into +`human-docs/deploy-ops.md` and delete this file outright. Don't leave it +sitting alongside a working `deploy-ops.md` describing the same system twice. diff --git a/human-docs/data-model.md b/human-docs/data-model.md new file mode 100644 index 0000000..f5c477c --- /dev/null +++ b/human-docs/data-model.md @@ -0,0 +1,453 @@ +# Data Model Reference + +*Reflects commit `5fe7a45`, 2026-08-01. Grounded in each app's `models.py` and `migrations/` +(`accounts`, `events`, `newsletter`, `ingestion`, `broadcast`), plus `theCommonsWeb/src/lib/auth-schema.ts` +for the Better Auth side. If anything here contradicts the code, trust the code — a couple of +places where `ARCHITECTURE.md` §Data Models has drifted are called out in §9.* + +This is the reference doc, not the narrative one — for the story of how an event moves through +the system, read `overview.md` first. This doc exists so you can keep a tab open and look up +"what does this column actually mean" without re-reading a `models.py`. + +## 1. The short version + +Five Django apps in `backendServer/` own the `public` Postgres schema between them — +`events`, `ingestion`, `accounts`, `newsletter`, `broadcast` — plus five more models that +`accounts` merely *mirrors*, read-only, out of a `neon_auth` schema that belongs to Better Auth +(the Next.js identity provider, not Django). A recent refactor (Suite 41) moved several models +between apps without moving their physical tables — §8 explains exactly what that means and why +a table name and its owning Django app no longer match for three of these models. `Town` and +`Category` are real SQL tables, not hardcoded choices — nothing in the codebase should treat them +as enums, and the ingestion pipeline silently parks (not drops) any event whose town slug doesn't +match a row. + +## 2. How the models relate + +```mermaid +erDiagram + Town ||--o{ Event : "town (SET_NULL)" + Event }o--o{ Tag : tags + Event }o--o{ Category : categories + BetterAuthUser ||--o{ Event : "created_by (SET_NULL)" + BetterAuthUser ||--o| UserProfile : "user (1:1)" + BetterAuthUser ||--o| BusinessProfile : "user (1:1)" + BetterAuthUser ||--o{ BetterAuthSession : sessions + BetterAuthUser ||--o{ BetterAuthAccount : accounts + UserProfile }o--o{ Tag : tags + BusinessProfile }o--o{ Tag : tags + BusinessProfile }o--o{ Town : service_area + + EventSource ||--o{ RawEvent : raw_events + EventSource ||--o{ SourceRun : runs + RawEvent ||--o| StagedEvent : "raw_event (1:1, nullable)" + StagedEvent ||--o| StagedEvent : "duplicate_of (self, SET_NULL)" + StagedEvent }o--o| Event : "published_event (SET_NULL)" + BetterAuthUser ||--o{ StagedEvent : "submitted_by (SET_NULL)" + + BroadcastSubmission ||--o{ BroadcastTarget : targets + AccessCode ||--o{ AccessCodeUse : uses + AccessCode ||--o{ AccessCodeRedemption : redemptions + AccessCode ||--o| SalesCodeSlot : "sales_slot (1:1, PROTECT)" +``` + +Two things this diagram deliberately leaves out, because they aren't foreign keys: + +- **`NewsletterSubscriber` has no relationship line to `UserProfile`.** There isn't one in the + database — see §5 and the sharp edge in §10. +- **`broadcast`'s models never point at `events`, `accounts`, or `ingestion` models**, and vice + versa. That's not an omission — `broadcast/routing.py` is contractually forbidden from + importing `events`, and the whole app operates on its own denormalized copy of an event's + fields (`BroadcastSubmission`'s title/datetime/venue/etc. columns) rather than a foreign key + into `Event`. `BroadcastAccess` and `BroadcastImage` are standalone rows keyed by a plain + string (email, client_label) for the same reason — no FK anywhere in the picture. + +## 3. `events` app + +Owns the genuine event/taxonomy models — the public-facing data. All four tables are physically +in `public.events_*`, and the app label matches the table prefix here (unlike `accounts` and +`newsletter` — see §8). + +### `Tag` + +| Field | Meaning | +|---|---| +| `name` | Unique, free-text. Lowercased by convention at write time (`ingestion.services` does `tag_name.strip().lower()` before `get_or_create`), not enforced by the field itself. | + +Reverse-related from `Event.tags`, `UserProfile.tags`, `BusinessProfile.tags` — one shared tag +vocabulary across events, personal interests, and business listings. + +### `Town` + +| Field | Meaning | +|---|---| +| `slug` | Unique, e.g. `carrboro`. This is the join key the ingestion pipeline computes from the LLM's free-text town guess (`town.lower().replace(" ", "-")`) — see §10. | +| `name` | Display name, e.g. `Carrboro`. | + +A real SQL table, not an enum — seeded by data migrations (`events/migrations/0016_seed_chatham_towns.py`, `0018_seed_apex_durham_towns.py`), editable in the admin. Adding a new town is a data change, not a code change. + +### `Category` + +| Field | Meaning | +|---|---| +| `slug` | Unique (`SlugField`). | +| `display_name` | Human label. | + +Same story as `Town` — a real table (`Meta.verbose_name_plural = "categories"` is the only +non-default option), not hardcoded. Unlike `Town`, an unmatched category slug does **not** stall +an event's publication — see §10. + +### `Event` + +| Field | Meaning | +|---|---| +| `uuid` | **Primary key.** Not `id` — see §10, the single most-hit trap in this codebase. | +| `title`, `venue`, `description`, `price`, `photo`, `link` | Plain content fields; `price` is nullable (no price listed, not "$0"). | +| `date` | Indexed (`db_index=True`) — this is the field every list/window query filters and sorts on. | +| `town` | FK → `Town`, `SET_NULL` — an event survives its town row being deleted, just loses the association. | +| `tags`, `categories` | M2M → `Tag`, `Category`. | +| `is_verified` | `True` only when the submitter was an authenticated user with `user_type="BUSINESS"` at publish time (`ingestion.services`) — not a manual admin toggle, not related to safety scoring. | +| `source_name` | Free text describing provenance: the originating `EventSource.name` for scraped events, `"Community Submission"` for anonymous `/events/create` posts, `"Direct submission by {organizer}"` / `"Direct submission by host"` for the broadcast direct-submit path. | +| `created_by` | FK → `accounts.BetterAuthUser`, `SET_NULL`, `db_constraint=False` (see §8 for why FKs into the mirrors never carry a DB-level constraint). Null for pipeline-ingested events — this is how you tell "someone submitted this" from "the scraper found this." | + +There is no soft-delete or unpublish concept: an `Event` row existing *is* what "published" +means. Only the owner (`created_by`) can hard-delete their own event via the API. + +## 4. `ingestion` app + +The pipeline that produces `Event` rows from external sources. Full flow narrative lives in +`overview.md` §3 and `ingestion.md`; this section is field-level reference. + +### `EventSource` + +| Field | Meaning | +|---|---| +| `source_type` | One of `ics`, `scraper`, `http`, `email`, `direct`. `direct` doesn't get polled at all — it exists so direct-submit's synthetic `RawEvent`s have a `source` FK to point at, same as every other row. | +| `active` | Poll loop skips inactive sources. | +| `last_polled` | Null = never successfully polled yet. | +| `poll_interval_hours` | Minimum gap between polls for this source, default 24. | +| `prompt_suffix` | Extra text appended to the Gemini standardization prompt for this source specifically — a per-source LLM tuning knob. | +| `scraper_key` | Looks up the corresponding Python scraper module for `source_type="scraper"`; blank for other types. | + +### `SourceRun` + +| Field | Meaning | +|---|---| +| `status` | `ok` / `failed` / `refused` / `skipped` — one row per poll attempt, for observability (this table isn't in `ARCHITECTURE.md`'s Data Models section — see §9). | +| `trigger` | `scheduled` / `probe` / `manual` — how the run was kicked off. | +| `items_fetched` / `items_new` / `items_duplicate` | Counts from that single run. | +| `finished_at` | Null while the run is still in progress (or if it crashed hard enough not to reach the finally block). | +| `error_class`, `error_message`, `traceback` | Populated only on `status="failed"`. | + +Ordered `-started_at` by default; indexed on `(source, -started_at)` for the "recent runs for +this source" query the admin/monitoring views use. + +### `RawEvent` + +| Field | Meaning | +|---|---| +| `source` | FK → `EventSource`. | +| `raw_*` fields | Exactly what was scraped/submitted, before any LLM cleanup. | +| `source_uid` | The feed's own per-item identifier (an ICS `UID`) for scraped sources; the client-generated `draft_id` string for direct-submit rows. `unique_together=(source, source_uid)` is what makes direct-submit idempotent — resubmitting the same `draft_id` upserts instead of creating a duplicate row. | +| `raw_organizer` | Only populated by direct host submissions; drives the `"Direct submission by {name}"` attribution on the eventual `Event.source_name`. Blank for everything else. | +| `processed` | Flips to `True` once `standardizer.py` has consumed it into a `StagedEvent`. | + +### `StagedEvent` + +| Field | Meaning | +|---|---| +| `raw_event` | OneToOne → `RawEvent`, **nullable** — null for events entered via the plain `/events/create` form, since those skip the raw/standardize step entirely and go straight to a pending `StagedEvent`. | +| `town`, `category` | Plain strings (an LLM's best guess, or a user's typed value), **not foreign keys** — see §10 for what happens when they don't match a real row. | +| `tags` | `JSONField`, a list of tag-name strings — converted to real `Tag` rows only at publish time via `get_or_create`. | +| `status` | `pending` / `approved` / `rejected` / `duplicate` / `published` / `skipped_no_town`. See the state diagram below — `published` is terminal but the row is **not deleted**, because it's part of the deduplicator's matching corpus (`deduplicator.CANDIDATE_STATUSES` includes `pending`, `approved`, `duplicate`, `published`, and `skipped_no_town` — everything except `rejected`). Rows are eventually reaped by `cleanup_old_events` once `start_datetime` is in the past. | +| `safety_score` | **Nullable, and the null-vs-value distinction matters.** `null` = not yet scored by Gemini. A non-null value with `status="pending"` means it *was* scored and came back above `SAFETY_SCORE_THRESHOLD` (default `0.3`) — held for manual review, not rejected. `<= SAFETY_SCORE_THRESHOLD` triggers auto-approval instead. There is no separate "held for review" status value; it's this null/non-null-plus-pending combination. | +| `duplicate_of` | Self-FK, `SET_NULL`. Set alongside `status="duplicate"` when the deduplicator (`thefuzz`) matches this row against an existing `StagedEvent`. | +| `published_event` | FK → `events.Event`, `SET_NULL`. Normally non-null only once `status="approved"` flips to `"published"` — **except** on a direct-submission re-edit that lands on `duplicate` or gets re-parked `pending`/`skipped_no_town`: in that case `published_event` still points at the *previously* published `Event`, because `Event` has no unpublish mechanism and the row must not orphan a live listing just because its latest edit didn't clear the gate. Don't assume `published_event != null` implies `status` is `approved` or `published`. | +| `submitted_by` | FK → `accounts.BetterAuthUser`, `SET_NULL`, `db_constraint=False`. Null for pipeline-ingested and anonymous direct-submit events. | + +```mermaid +stateDiagram-v2 + [*] --> pending: standardized (poll, /events/create, or direct-submit) + pending --> duplicate: dedup match found + pending --> skipped_no_town: town slug unmatched + pending --> approved: score at or below threshold (auto) or manual approval + pending --> rejected: manual rejection + approved --> published: publish_all_approved() + published --> [*]: cleanup_old_events (after start_datetime passes) +``` + +One thing the state diagram can't say: **`duplicate`, `skipped_no_town`, and even `pending` +are not necessarily dead ends** for a row created via direct-submit re-editing — the row can +carry a `published_event` pointer even while parked in one of those statuses, per the +`published_event` row above. The diagram shows the primary transitions; that pointer is a side +channel that survives them. + +## 5. `accounts` app + +Owns identity: the five read-only Better Auth mirrors, plus the two profile models that hang +off a user. A "business" is modeled as a *kind of user profile*, not a separate top-level +concept. + +### The Better Auth mirrors (`neon_auth` schema, `managed = False`) + +**Better Auth, running inside the `theCommonsWeb` Next.js app, owns writes to these five +tables — Django never migrates them.** `theCommonsWeb/src/lib/auth-schema.ts` (Drizzle) is the +actual source of truth for their shape; the Django models below exist purely so the ORM can join +against them. Each model's `db_table` uses a deliberate double-quote trick — +`'neon_auth"."user'` — so Django emits a valid cross-schema reference (`FROM "neon_auth"."user"`) +without Django having first-class multi-schema support. + +| Model | Table | Key fields | Notes | +|---|---|---|---| +| `BetterAuthUser` | `neon_auth.user` | `id` (UUID PK), `email` (unique), `email_verified`, `user_type` | Hardcodes `is_authenticated = True` / `is_anonymous = False` as class attributes so DRF permission classes treat an instance as a real authenticated user. | +| `BetterAuthSession` | `neon_auth.session` | `id` (text PK), `token` (unique), `expires_at`, `user_id` | `user_id` is a plain `TextField`, not a UUID FK column — see the sharp edge below. | +| `BetterAuthAccount` | `neon_auth.account` | `id` (text PK), `provider_id`, `user_id` (UUID), `password` | One row per sign-in method per user. For the `credential` provider (email+password), `password` holds the hashed credential — null for OAuth-provider rows. | +| `BetterAuthVerification` | `neon_auth.verification` | `identifier`, `value`, `expires_at` | Email-verification / password-reset tokens. | +| `BetterAuthJwks` | `neon_auth.jwks` | `public_key`, `private_key` | The signing keyset Django's JWKS client fetches to verify JWTs — see `auth.md`. | + +Every FK from a Django-managed model into one of these mirrors is declared with +`db_constraint=False` — there is no database-level foreign key against an unmanaged table, only +an application-level one. `BetterAuthUser.id` is a UUID, but `BetterAuthAccount.user_id` and +`BetterAuthSession.user_id` are plain `TextField`/`UUIDField` columns rather than declared FKs to +`BetterAuthUser.id` at the Django level (the join happens through matching values, mirroring +however Better Auth itself models it in Postgres). + +### `UserProfile` + +| Field | Meaning | +|---|---| +| `user` | OneToOne → `BetterAuthUser`, `db_constraint=False`. | +| `uuid` | A separate identifier from `user.id` — used in URLs/serializers where you don't want to expose the Better Auth user id directly. | +| `user_type` | `LOCAL` / `BUSINESS` / `VENUE`. Drives `Event.is_verified` at publish time (see §3) and gates the business-listing endpoints. | +| `primary_city`, `address` | Free text, both blank-allowed. | +| `email_preference` | `WEEKLY` / `MONTHLY` / `NEVER`. Writing this via `PATCH /auth/me` is what triggers the `NewsletterSubscriber` sync described in §10. | +| `tags` | M2M → `events.Tag` — this user's interest tags, read by `newsletter._build_recipients` to filter their digest. | + +`db_table = "events_userprofile"` — the physical table name still says `events`, because this +model *moved apps* without moving tables. See §8. + +### `BusinessProfile` + +| Field | Meaning | +|---|---| +| `user` | OneToOne → `BetterAuthUser`. | +| `business_name`, `description`, `contact_email`, `contact_phone` | Listing content. | +| `is_published` | Gates visibility on the public `/businesses` list — an unpublished listing is only visible to its owner. | +| `tags` | M2M → `events.Tag`. | +| `service_area` | M2M → `events.Town` — which towns this business serves; drives filtering on the business directory. | + +`db_table = "events_businessprofile"` — same historical-name situation as `UserProfile`. + +## 6. `newsletter` app + +### `NewsletterSubscriber` + +| Field | Meaning | +|---|---| +| `email` | Unique. The only identity a subscriber needs — **no FK to `UserProfile` or `BetterAuthUser`.** Anonymous subscribers (no account) and account holders alike get a row here; the two are correlated only by matching `email` string, case-insensitively, at read time in `newsletter._build_recipients` (see §10). | +| `frequency` | `WEEKLY` / `MONTHLY`. | +| `is_active` | `False` = unsubscribed. A `PATCH /newsletter/manage` with `frequency=NEVER` sets this rather than deleting the row, preserving history and the `manage_token`. | +| `manage_token` | `UUIDField`, unique, unguessable — the entire authentication mechanism for the login-free manage/unsubscribe link (`/newsletter/manage?token=...`). Anyone holding the token can read or change that one subscription; nothing else. | +| `subscribed_at` | Set once on creation (`auto_now_add`), not touched on re-subscribe. | + +`db_table = "events_newslettersubscriber"` — moved apps, table name unchanged. See §8. + +## 7. `broadcast` app + +Pushes a published event out to third-party town calendars. Its models never reference +`events`, `accounts`, or `ingestion` models — see §2. Full subsystem detail is in `broadcast.md`; +this is field-level reference for the models only. + +### `BroadcastSubmission` + +| Field | Meaning | +|---|---| +| `id` | UUID PK. | +| `client_label` | Identifies which partner/operator created this — not a FK to any user model, just a string. | +| Denormalized event fields (`title`, `start_datetime`, `venue_name`, `address_line1`, `locality` JSON, `categories` JSON, `event_url`, `price`, `organizer_name`, `contact_email`, …) | A frozen snapshot of the event's data at submission time — deliberately not a FK into `events.Event`, per the isolation contract. | +| `status` | `queued` / `running` / `done` / `failed` / `canceled`. | + +### `BroadcastTarget` + +| Field | Meaning | +|---|---| +| `submission` | FK → `BroadcastSubmission`. `UniqueConstraint(submission, site_key)` — one target row per site per submission. | +| `site_key` | Which third-party calendar adapter this target is for. | +| `status` | `pending` / `in_progress` / `succeeded` / `failed` / `needs_manual` / `skipped`. | +| `dry_run` | `True` = this target ran in preview/test mode, never actually submitted to the third-party site. | +| `screenshot_path` | Blank until a run captures one; gated behind the screenshots endpoint. | + +### `BroadcastAccess` + +| Field | Meaning | +|---|---| +| `email` | Unique, lowercased on save. | +| `tier` | `0` / `1` / `2`, default `0`. The *permanent* tier for a logged-in identity, set by redeeming an `AccessCode` of `kind="upgrade"` — resolved by `broadcast/access.py` off the JWT's email claim. | + +### `AccessCode` + +| Field | Meaning | +|---|---| +| `kind` | `trial` (anonymous, always forced to `tier=2` in `save()`, time-boxed via `expires_at` rather than metered) or `upgrade` (redeemed by a logged-in user against `POST /broadcast/redeem`, permanently sets that user's `BroadcastAccess.tier`). Two independent code pools — a trial code is never resolved through the JWT path, an upgrade code never through the anonymous header path. | +| `code` | Plaintext, nullable. **Null specifically means the code was created before this field existed** (added in migration `0008_accesscode_code`) — those older codes only ever had a hash, and there is no way to recover their plaintext. Non-null on every code created since, so operators can copy it after generation. | +| `code_hash` | SHA-256 hex of the code, unique. All validation compares against this — `code` is a convenience, never read for auth checks. | +| `max_uses` | **Null = unlimited.** Default `3` when set. Distinct from `0`, which would mean "already exhausted." | +| `is_active` | Manual kill switch, independent of `expires_at`/`max_uses`. | +| `expires_at` | Null = no expiry. | + +### `AccessCodeUse` + +| Field | Meaning | +|---|---| +| `access_code`, `draft_id` | `unique_together` — meters a **trial** code's anonymous preview usage per draft, so the same in-progress draft doesn't consume multiple uses on retry. | + +### `AccessCodeRedemption` + +| Field | Meaning | +|---|---| +| `access_code`, `email` | `unique_together` — one row per email that has redeemed an **upgrade** code; this is what `max_uses` counts against. `email` lowercased on save. | + +### `BroadcastImage` + +| Field | Meaning | +|---|---| +| `image` | Stored **re-encoded**, never as received — third-party share links often lack CORS headers or aren't direct image URLs, so uploads are self-hosted rather than linked. | +| `client_label` | Same free-text partner identifier as `BroadcastSubmission.client_label` — no FK between them. | + +### `SalesCodeSlot` + +| Field | Meaning | +|---|---| +| `slot` | One of `trial` / `tier1` / `tier2`, unique — exactly one evergreen row per slot. | +| `access_code` | OneToOne → `AccessCode`, `on_delete=PROTECT` — the code currently live in that slot; rotating the slot creates a new `AccessCode` and repoints this rather than mutating the old one. | +| `raw_code` | Plaintext, unlike `AccessCode.code_hash` elsewhere — exists so a salesperson can open the admin and see a live, copyable code with no CLI step, a deliberate convenience-over-defense-in-depth tradeoff scoped to this one low-stakes flow. | + +## 8. The Suite 41 model moves — what actually happened + +`UserProfile` and `BusinessProfile` moved from `events` to `accounts`; `NewsletterSubscriber` +moved from `events` to `newsletter`. Both moves used +`migrations.SeparateDatabaseAndState` with an **empty `database_operations` list** — meaning +**zero DDL ran**. `events/migrations/0021_move_identity_to_accounts.py` and +`0022_move_newsletter_to_newsletter.py` only delete the models from Django's *state* (so the ORM +stops thinking `events` owns them); the paired `accounts/migrations/0001_initial.py` and +`newsletter/migrations/0001_initial.py` re-create the same models in the new app's state, with +`db_table` pinned to the original name (`events_userprofile`, `events_businessprofile`, +`events_newslettersubscriber`). The physical Postgres tables never moved, were never renamed, +and never lost a row. + +The practical consequence: **the Django app label and the actual Postgres table name disagree** +for these three models. If you're debugging with `psql` directly, or reading a raw SQL log, you +will see `events_userprofile` — even though the model now lives in `accounts/models.py` and +migrates under the `accounts` app label. This is not a bug or a leftover to clean up; renaming +the table would be a real (and riskier) migration for zero functional benefit, so it was left +alone on purpose. + +`neon_auth.*` was never touched by any of this — those mirrors were already `managed=False` +before the move and remain so. + +A companion data migration, `newsletter/migrations/0002_repoint_digest_beat.py`, updates the +existing `django_celery_beat.PeriodicTask` rows' `task` dotted-path from +`events.tasks.fan_out_weekly_digest` / `fan_out_monthly_digest` to +`newsletter.tasks.fan_out_weekly_digest` / `fan_out_monthly_digest` — the beat schedule rows +themselves (created earlier by `events/migrations/0015_seed_digest_beat.py` and +`0020_seed_monthly_digest_beat.py`) were left in place and only repointed, not recreated. + +## 9. Doc drift found while writing this + +`ARCHITECTURE.md` §Data Models is mostly accurate post-Suite-41, but has fallen behind the +current models in a few places: + +- **`SourceRun` is missing entirely** from the `ingestion` app table in `ARCHITECTURE.md` — it's + a real, migrated model (`ingestion/migrations/0014_sourcerun.py`) used for per-poll + observability, documented in full in §4 above. +- **`broadcast`'s model list in `ARCHITECTURE.md` stops at `AccessCodeUse`** — it's missing + `AccessCodeRedemption`, `BroadcastImage`, and `SalesCodeSlot`, all three of which exist and are + migrated (`0006_accesscode_kind_accesscoderedemption.py`, `0010_broadcastimage.py`, + `0007_salescodeslot.py`). It also doesn't mention `AccessCode.kind` (trial vs. upgrade), which + is central to how the two code pools behave differently — see §7. +- **`StagedEvent.status`'s choice list in `ARCHITECTURE.md`** omits `published` and + `skipped_no_town`, both of which are real, reachable statuses (§4's state diagram). +- **`EventSource.source_type`'s choices in `ARCHITECTURE.md`** are described as "ics/scraper/email/direct" — the model also has `http`, and `prompt_suffix`/`scraper_key` fields aren't mentioned at all. + +None of this changes any relationship or field meaning documented elsewhere in +`ARCHITECTURE.md` — it's a coverage gap (models added after the doc was last updated), not a +factual disagreement. + +## 10. Sharp edges + +1. **`events.Event`'s primary key is `uuid`, not `id`.** There is no `id` field on `Event` at + all. `Event.objects.values("id")`, `Count("id")`, or any code that assumes a Django default + auto PK will raise `FieldError: Cannot resolve keyword 'id'`. Use `Count("pk")` or reference + `.uuid` explicitly. This is the single most-hit trap in this codebase — every other model + here uses Django's default `id` PK, which is exactly what makes `Event` easy to get wrong by + habit. + +2. **`neon_auth` mirror models must never be migrated by Django.** They're `managed=False` + specifically so `python manage.py migrate` never generates DDL for them — Better Auth + (running inside `theCommonsWeb`) owns their schema exclusively via Drizzle. If a future model + change to `BetterAuthUser`/etc. accidentally flips `managed` or drops the `Meta`, the next + migration would try to create or alter tables that already exist and are owned by a different + codebase. + +3. **`Town` and `Category` are SQL rows, not enums.** Don't hardcode a Python list of towns or + categories anywhere — new ones are added as data (seed migrations or the admin), not code. + The ingestion pipeline enforces this at the boundary: a `StagedEvent.town` string that doesn't + slugify to an existing `Town.slug` sends the row to `status="skipped_no_town"` rather than + publishing with a null/wrong town (see §4's state diagram). `Category` is looser — an + unmatched category slug doesn't block publication at all; `publish_all_approved` just skips + attaching a category and the event goes live uncategorized. + +4. **`StagedEvent.safety_score` null and `StagedEvent.safety_score` non-null-but-`pending` mean + different things.** Null = the Gemini safety scorer hasn't run on this row yet. A non-null + value with `status` still `"pending"` means it *was* scored and came back above + `SAFETY_SCORE_THRESHOLD` (default `0.3`) — held for a human to approve or reject by hand, + not an error state and not "unscored." + +5. **`StagedEvent.published_event` being non-null does not imply `status` is `"approved"` or + `"published"`.** On a direct-submission re-edit that lands on `duplicate` or gets re-parked + at `pending`/`skipped_no_town`, `published_event` is deliberately left pointing at whatever + `Event` a *previous* submission under the same `draft_id` already published — because `Event` + has no unpublish/soft-delete concept, and orphaning a live listing just because a later edit + didn't clear the safety/dedup/town gate would be worse than leaving the stale content live. + Check `status`, not just whether `published_event` is set, before assuming a row represents + the live content. + +6. **`NewsletterSubscriber` has no foreign key to `UserProfile` or `BetterAuthUser`.** The two + are joined only by a case-insensitive string match on `email`, done at read time inside + `newsletter._build_recipients` (`accounts` writes the `NewsletterSubscriber` row when a + user's `email_preference` changes; `newsletter` reads `accounts.UserProfile.tags` back out by + matching email). If a user changes their Better Auth email without the corresponding + `NewsletterSubscriber.email` being updated to match, digest tag-filtering for that address + silently stops working — there's no constraint that would catch the mismatch. + +7. **`accounts` ↔ `newsletter` is a deliberate two-way coupling, not a boundary bug.** + `accounts.views` (the `/auth/me` PATCH handler) writes/updates a `NewsletterSubscriber` row + whenever `email_preference` changes; `newsletter.email_service._build_recipients` reads + `accounts.UserProfile` back out for tag-filtered digests. Each app's `test_isolation_fast.py` + forbids reaching into `ingestion`/`broadcast`, but explicitly permits this pair reaching into + each other and into `events`. Don't "fix" this into a one-way dependency — both directions are + load-bearing. + +8. **The `accounts`/`newsletter` `db_table` values don't match their app labels.** + `UserProfile`/`BusinessProfile` physically live in `events_userprofile`/`events_businessprofile`; + `NewsletterSubscriber` lives in `events_newslettersubscriber`. All three migrated apps + *without* their tables — see §8. Grepping for `accounts_userprofile` in the database will + find nothing. + +9. **`AccessCode.max_uses = null` means unlimited, not zero.** A code with `max_uses=0` would + read as already-exhausted; `null` is the sentinel for "don't meter this code at all." The two + are easy to conflate when writing a query against this field. + +10. **FKs into the `neon_auth` mirrors always carry `db_constraint=False`.** `Event.created_by`, + `StagedEvent.submitted_by`, `UserProfile.user`, `BusinessProfile.user` all point at + `BetterAuthUser` with no database-level foreign key — only an application-level one. Postgres + will not stop you from inserting a value that doesn't correspond to a real `neon_auth.user` + row; referential integrity here is enforced by the application, not the schema. + +## 11. Not independently verified + +- Whether any code outside `ingestion.services` and `newsletter.email_service` also reads or + writes `StagedEvent.published_event` or `NewsletterSubscriber` directly — this doc is grounded + in the read/write paths found while writing it, not an exhaustive grep of every call site. + `ingestion.md` and `newsletter.md` are the deeper references for those flows. +- The exact query patterns the frontend (`theCommonsWeb`) issues against these tables — this doc + covers the Django-side model shape only; `frontend.md` is where that belongs. diff --git a/human-docs/deploy-ops.md b/human-docs/deploy-ops.md new file mode 100644 index 0000000..5a309f0 --- /dev/null +++ b/human-docs/deploy-ops.md @@ -0,0 +1,357 @@ +# Deployment & Operations + +*Written 2026-08-01 against commit `5fe7a45`. Complements [`DEPLOY.md`](../DEPLOY.md), +which stays the operational source of truth for step-by-step deploy/setup commands — this +doc is the mental model: what's actually running, why it's arranged this way, and what +fails how. Sibling docs: [`overview.md`](overview.md) (whole-system map), +[`async-jobs.md`](async-jobs.md) (Redis/Celery queue and beat-schedule detail), +[`auth.md`](auth.md) (the Better Auth bridge this doc's nginx section resolves a question +for), [`containerization.md`](containerization.md) (the Docker cutover this doc explains is +pending), [`testing.md`](testing.md) (local dev setup).* + +## 1. What's running, and the central fact to hold onto + +**Read this first: two deployment stories exist in this repository right now, and only one +of them is live.** `DEPLOY.md` was rewritten during this same work session to describe a +fully containerized stack — Docker Compose, an `nginx` container, a `backend` container, +one container per Celery role — built and verified locally end-to-end. None of it has +touched the production VM. The VM has no Docker installed. Every service a reader would +SSH in and find today is the plain **systemd-unit deployment** this doc describes. If you +take one thing from this document, take this: **as of 2026-08-01, production is systemd, +not containers.** §9 covers the pending cutover and what changes when it happens. + +The Commons runs on a single Oracle Cloud VM (Ubuntu 24.04, ARM64, 1 OCPU / 6 GB, IP +`129.80.229.41`) behind nginx, with Cloudflare in front for DNS and TLS (proxied, Full +strict). Postgres lives off-box, managed by Neon — the VM never runs a database server. +Everything else — the Django API, the Next.js site, Redis, four Celery-family processes — +runs as systemd units on that one box. There is no load balancer, no second VM, no +managed container platform. If this VM is down, the whole product is down: the public +site, the API, the broadcast operator console, ingestion, digests, everything. Anyone +touching production infrastructure, chasing a 2am page, or trying to understand why an +email didn't send depends on the picture in this document. + +## 2. How it works + +### Request routing + +nginx is the single ingress. It terminates TLS using a Cloudflare origin certificate +(`/etc/ssl/cloudflare/thecommons.town.{pem,key}`) and fans requests out to whichever +backend owns that subdomain — a plain Django app (gunicorn) for the API, a Node process +for the main site, and static files for everything else. + +```mermaid +flowchart TD + Client[Browser / API client] --> CF[Cloudflare edge - DNS + TLS proxy] + CF --> Nginx[nginx on the VM - terminates TLS again, Full strict] + + Nginx -->|thecommons.town| NextJS[Next.js process, port 3000] + Nginx -->|www.thecommons.town| Redirect1[301 to apex] + Nginx -->|auth.thecommons.town| NextJS + Nginx -->|api.thecommons.town| Gunicorn[gunicorn via Unix socket - run/gunicorn/gunicorn.sock] + Nginx -->|api.thecommons.town/static/| StaticFiles[backendServer/staticfiles - collectstatic output] + Nginx -->|api.thecommons.town/media/| MediaFiles[MEDIA_ROOT on disk - never touches Django] + Nginx -->|broadcast.thecommons.town| BroadcastSPA[static broadcastWeb build - dist/] +``` + +**Three things worth calling out.** First, `auth.thecommons.town` and the apex both land +on the *same* Next.js process — Better Auth lives inside `theCommonsWeb`, not a separate +service, so the "auth origin" is a routing decision, not a different deployable. Second, +`api.thecommons.town` reaches gunicorn over a **Unix socket** +(`unix:/run/gunicorn/gunicorn.sock`), not TCP — this matters for one sharp edge below +(django-ratelimit's IP key) and is the reason the containerized rewrite in `DEPLOY.md` +switches to TCP instead: a socket path doesn't cross a container boundary cleanly. Third, +`/media/` is nginx reading a directory directly; Django is never in that request path in +production — see §5. + +**On the `auth.thecommons.town` nginx routing question:** an earlier documentation pass +(`auth.md`) explicitly could not verify this and deferred it here. It's resolved: the +cutover runbook (`docs/runbook-auth-cutover.md`) records the exact server block added to +the VM's nginx config — `server_name auth.thecommons.town` with `proxy_pass +http://127.0.0.1:3000` and the standard `X-Real-IP`/`X-Forwarded-*` headers, TLS from the +same wildcard Cloudflare origin cert as every other subdomain — and an execution record +dated 2026-07-30 confirming it was applied and smoke-tested live (`curl +https://auth.thecommons.town/api/auth/jwks` returned 200 with a real JWKS body). One honest +caveat: that server block lives in a **hand-edited file directly on the VM** +(`/etc/nginx/sites-available/thecommons`), which is not itself checked into this +repository — only the *runbook instructions* for editing it are. The broadcast subdomain's +block is the one nginx fragment actually tracked in git +(`deploy/nginx-broadcast.conf.snippet`), meant to be pasted into that same live file. So +the routing is real, live, and verified — just not something `git grep` alone will ever +show you; you have to read the runbook or SSH in. + +### How a deploy happens + +Every push to `main` runs CI (`.github/workflows/ci.yml`): a `lint` job, then `backend` +(Django tests, Postgres 16 service container, `--tag=fast` then `--tag=db`), +`frontend-commons` and `frontend-broadcast` (pnpm build as the type-check gate, plus +`test:fast`/`test:db`) all run in parallel. Only if all three test jobs are green does a +gated `deploy` job SSH into the VM and touch anything — a failing test on `main` blocks +deployment outright, there is no way around that gate from the workflow file. + +```mermaid +sequenceDiagram + autonumber + participant GH as GitHub Actions + participant VM as Oracle VM + participant PG as Neon Postgres + + GH->>GH: lint, backend tests, frontend-commons tests, frontend-broadcast tests (parallel) + Note over GH: deploy job only starts if all three test jobs pass + GH->>VM: SSH in (appleboy/ssh-action, host key pinned via fingerprint) + VM->>VM: git pull origin main + VM->>VM: uv sync (backendServer) + VM->>VM: manage.py migrate --check + alt migrations pending + VM->>PG: pg_dump (gzip, timestamped) to /home/ubuntu/backups + VM->>VM: prune to 5 newest dumps + VM->>PG: manage.py migrate --noinput + else nothing pending + VM->>VM: skip migrate entirely + end + VM->>VM: manage.py collectstatic --noinput + VM->>VM: pnpm build (theCommonsWeb, then broadcastWeb) + VM->>VM: grep built broadcastWeb bundle for a real thecommons.town API origin + VM->>VM: sudo systemctl restart gunicorn nextjs celery celerybeat broadcast-worker scrape-worker + VM->>VM: systemctl is-active on all six (must all report active) + GH->>VM: second SSH step - post-deploy smoke test + VM->>VM: curl the three public domains, expect 200 + VM->>VM: POST an invalid broadcast request, expect 403 not 500 (Unix-socket REMOTE_ADDR regression check) + VM->>VM: GET /auth/me with no credentials, expect 401/403 not 500 +``` + +**Four things worth calling out.** First, the migration guard is genuinely conditional — +`migrate --check` exits non-zero only when there's real unapplied work, so most deploys +skip the dump-and-migrate branch entirely; a `pg_dump` is never skipped when a migration +*is* about to run, and the guard hard-fails the whole deploy if `pg_dump` isn't installed +rather than silently proceeding without a backup. Second, the broadcastWeb bundle grep +exists because a malformed `VITE_BROADCAST_API_BASE_URL` builds cleanly and only fails at +runtime, as every API call silently misroutes — this catches that class of bug before the +build goes live, not after. Third, `systemctl is-active` passing is necessary but not +sufficient — a crashing view or a misrouted SPA both restart clean and report `active`, +which is exactly why there's a separate smoke-test step hitting real URLs afterward, not +just a process-liveness check. Fourth, the smoke test's `403` check on a broadcast endpoint +is a deliberate regression probe: nginx talking to gunicorn over a Unix socket used to +leave `REMOTE_ADDR` empty, which crashed `django-ratelimit`'s IP-based rate limiting with +an unhandled 500 on every request to a rate-limited broadcast view — a `500` here means +that bug is back, a `403` means the request was correctly rejected before it ever became a +ratelimit crash. + +There is no separate `gunicorn.service` or `nextjs.service` file in this repository's +`deploy/` directory, and none exists anywhere in git history — those two units were set up +by hand directly on the VM and were never checked in, unlike the four Celery-family units +and the healthcheck unit, which are. If you need their exact unit-file contents, SSH in and +read `/etc/systemd/system/gunicorn.service` / `nextjs.service` directly, or see the last +commit of `DEPLOY.md` before its Docker rewrite (`git show 053d65b:DEPLOY.md`) for a +recorded copy of what they contained as of late July. + +## 3. The systemd units + +| Unit | What it runs | Drains / serves | How to check it | +|---|---|---|---| +| `gunicorn` | Django via a Unix socket, 3 sync workers | `api.thecommons.town` (proxied by nginx) | `systemctl status gunicorn`; not tracked in `deploy/` — hand-configured on the VM | +| `nextjs` | `node`/`npm run start` for `theCommonsWeb`, port 3000 | `thecommons.town` and `auth.thecommons.town` (both proxy to the same process) | `systemctl status nextjs`; not tracked in `deploy/` — hand-configured on the VM | +| `redis-server` | Standard `apt`-installed Redis, `/etc/redis/redis.conf` | DB 0 = Celery broker/results, DB 1 = Django cache | `systemctl status redis-server`; `redis-cli -a ping` | +| `celery` (`deploy/celery.service`) | Default worker, `.venv/bin/celery -A backend worker -n commons-default@%h --concurrency=2` | Everything not explicitly routed elsewhere — digest sends, misc tasks | `systemctl status celery`; `manage.py healthcheck`'s `celery_worker` probe | +| `celerybeat` (`deploy/celerybeat.service`) | Scheduler, `django_celery_beat`'s `DatabaseScheduler` — exactly one process, never scale this | Fires `ingest-events-daily` (04:00 ET), `weekly-digest-sunday`/`monthly-digest` (18:00 ET), `broadcast-orphan-recovery` | `systemctl status celerybeat`; `manage.py healthcheck`'s per-task `beat:` freshness probes — the check this doc's §8 incident is really about | +| `broadcast-worker` (`deploy/broadcast-worker.service`) | Playwright form-filler, `celery -A backend worker -Q broadcast -c 1` | The dedicated `broadcast` queue only — `-c 1` is load-bearing, not tuning: orphan recovery assumes a single worker | `systemctl status broadcast-worker` | +| `scrape-worker` (`deploy/scrape-worker.service`) | Headless-Chromium ingestion scraper, `celery -A backend worker -Q scrape -c 1` | The dedicated `scrape` queue, kept off the default worker so Chromium memory can't starve digests/ingestion | `systemctl status scrape-worker` | +| `healthcheck.timer` / `.service` (`deploy/healthcheck.*`) | Hourly `bash deploy/healthcheck.sh`, itself running `manage.py healthcheck --require-prod` | Nothing — read-only report | `systemctl list-timers healthcheck.timer`; `journalctl -u healthcheck.service -n 50` | + +`celery`, `celerybeat`, `broadcast-worker`, and `scrape-worker` all `Require=` and +`After=redis-server.service` and set `Restart=always` — the restart policy is +belt-and-suspenders, explained in §4, not the primary fix for anything. `deploy/`'s +`nginx-broadcast.conf.snippet` is not a systemd unit; it's an nginx server-block fragment +meant to be appended by hand into the VM's single live config file. + +## 4. The `uv run` vs. venv-binary sharp edge — a real outage, not a style rule + +Every long-lived unit in `deploy/` execs `/home/ubuntu/thecommons/backendServer/.venv/bin/celery` +directly. That specific phrasing — the venv binary, not `uv run celery`, and not a wrapper +shell script — is load-bearing, and the reason is a real production incident recorded in +full at `docs/prod-incident-2026-07-21-scheduler-outage.md`. + +```mermaid +flowchart TD + Deploy[Deploy finishes over SSH] --> Logout[SSH session ends] + Logout --> Teardown[logind tears down user-1001.slice - Linger was off] + + Teardown --> SnapPath{Unit's ExecStart} + SnapPath -->|snap uv run celery ...| SnapChild[Child process lives inside a transient snap.astral-uv scope UNDER the user slice] + SnapChild --> SnapDeath[Slice teardown kills it - clean exit, status=0/SUCCESS] + SnapDeath --> NoRestart[Restart=on-failure correctly declines to restart a clean exit] + NoRestart --> Dead[celery / celerybeat / both workers silently dead] + + SnapPath -->|.venv/bin/celery ...| VenvChild[Child process is the unit's own cgroup - never enters a user-manager scope] + VenvChild --> Survives[Slice teardown has nothing to do with this process] + Survives --> Alive[Process keeps running through the next login/logout cycle] +``` + +**What actually happened, concretely:** all four Celery-family units ran through +`/snap/bin/uv run celery …`. Snap's `uv` spawns its child inside a transient +`snap.astral-uv.uv-*.scope`, parented under `user@1001.service` — the *user* session +manager, not the systemd unit's own cgroup. With account lingering off, the moment the +deploying SSH session ended, `logind` tore down `user-1001.slice` and every snap scope +under it, taking `celery`/`celerybeat`/`broadcast-worker`/`scrape-worker` down with it — +**a clean exit**, `status=0/SUCCESS`, which `Restart=on-failure` correctly declined to +restart (it's not a failure by that policy's definition). `gunicorn` and `nextjs`, which +never touched snap, stayed up the entire time on the same VM through the same deploys. The +async stack was fully dead for 8 days before anyone noticed, because the site itself kept +serving pages — nothing about "the site is up" implied "background jobs are running." + +**What breaks if someone "simplifies" a unit back to `uv run`:** exactly this, again. The +fix — execing `.venv/bin/celery` directly — removes the snap-scope mechanism entirely, +which is the actual fix; `loginctl enable-linger ubuntu` and `Restart=always` are +defense-in-depth layered on top, not substitutes for it. A reviewer who sees `uv run` as +"more consistent with the rest of the deploy tooling" and reverts a unit to it silently +reopens this exact failure mode — it will not show up in `systemctl status` right after the +change, only after the next SSH session that started the deploy ends. + +**One deliberate exception, and it is not a contradiction:** `healthcheck.service` still +uses `/snap/bin/uv` (`Environment=UV_BIN=/snap/bin/uv`, `ExecStart=... bash +deploy/healthcheck.sh`) and that is fine. It's `Type=oneshot` — the process starts, runs +the health report to completion in a few seconds, and exits on its own, well before any SSH +session it happened to be triggered near could tear down. The failure mode above only bites +a process still running at the moment a user slice gets torn down; a oneshot that's already +finished has nothing left to kill. Don't read the healthcheck unit's `uv` line as +permission to relax the rule anywhere else — it's a narrow exception with a specific reason, +not evidence the rule is soft. + +## 5. Environment selection: `DJANGO_ENV` and how failure got narrower + +Django settings are resolved by `backend/settings/__init__.py` via a function, +`select_settings_env`, reading the `DJANGO_ENV` environment variable — not by +`DJANGO_SETTINGS_MODULE` pointing at `prod.py` directly, the way Django docs usually show +it. + +```mermaid +flowchart TD + Start[DJANGO_ENV read from environment] --> Empty{Unset or blank/whitespace?} + Empty -->|yes| Dev1[Resolve to dev - dev.py loads] + Empty -->|no| Known{Value, lowercased+stripped, is 'dev' or 'prod'?} + Known -->|prod| Prod[Resolve to prod - prod.py loads] + Known -->|dev| Dev2[Resolve to dev - dev.py loads] + Known -->|anything else, e.g. 'production'| Crash[ImproperlyConfigured raised at import time - process refuses to start] +``` + +This is a deliberately narrowed failure mode, and the history matters. The original +incident (June 2026, referenced directly in the module's own docstring) was `DJANGO_ENV` +simply **missing** on the VM: the app silently served `dev.py`, whose `ALLOWED_HOSTS` is +localhost-only, so every real request to `api.thecommons.town` came back `DisallowedHost` +(HTTP 400) — which the frontend rendered indistinguishably from "there are just no events +right now." That silent-unset-defaults-to-dev behavior is **still current and still +deliberate** — every local laptop relies on `DJANGO_ENV` being absent and getting `dev.py` +for free, and changing that would break local dev for everyone. What changed is the *other* +failure shape: a **typo'd but non-empty** value (`DJANGO_ENV=production`, a stray `PRD`, +anything not exactly `dev` or `prod` after trimming and lowercasing) now raises +`ImproperlyConfigured` immediately at import time instead of quietly falling back to +`dev.py` — the process won't boot at all, which is loud and fast rather than silent and +slow. `events/tests/test_config_fast.py` pins this exact behavior as a regression test +(unset/blank → `dev`; `prod`/`PROD `/`Dev` all normalize correctly; `production`, +`staging`, `PRD`, `true` all raise). + +The remaining gap — `DJANGO_ENV` unset in prod specifically, which the hard-error change +does nothing for, since unset is still valid input — is caught by `manage.py healthcheck +--require-prod`, run hourly via `healthcheck.timer`. That command checks `settings.DEBUG` +and whether `ALLOWED_HOSTS` is anything other than localhost-only, and reports a `FAIL` if +either looks like dev settings leaked into what's supposed to be prod. **Read `--require-prod` +correctly: it is a detector, not a guard.** It can tell you, up to an hour later, that +production is quietly running on dev settings; it cannot stop that from happening, and it +does not run on every request or every deploy — only once an hour, on the health-check +timer's own schedule. A misconfigured `.env` on the VM still means real downtime for up to +that long before anyone is told. + +## 6. Media: why it lives outside the checkout + +`MEDIA_ROOT` (client-uploaded event images) is set in production to +`/home/ubuntu/broadcast/media` — a path that sits next to the git checkout +(`/home/ubuntu/thecommons`), not inside it. `backend/settings/base.py`'s own comment on +`MEDIA_ROOT` states the reason directly: it defaults to a path *inside* the checkout for +local dev, but production overrides it in `.env` specifically so a `git pull` during deploy +can never touch uploaded files. A deploy that ran `git clean` or reset the working tree +inside the checkout would have no way to reach these files at all — they're simply not +under that directory. + +The second half of the same design: nginx serves `/media/` directly as a plain file alias, +and Django is never in that request path in production. The comment in `base.py` says this +outright ("Served by nginx in prod, never by Django"), and the pre-Docker `DEPLOY.md` +revision that documented the live nginx config confirms it as a sibling `location /media/` +block to the existing `/static/` alias, pointing at the same `MEDIA_ROOT` path. The reason +is the ordinary one for serving static assets from the ingress instead of the app server: +nginx does it faster and without spinning up a Python worker to stream a file back to +disk. There's a real cost worth knowing about, not a bug: uploaded images are kept +indefinitely — no pruning job exists anywhere in this repo — so `MEDIA_ROOT` grows without +bound. At roughly 1–3 MB per event this is currently negligible against the VM's block +volume, but it's a number worth keeping an eye on, not a problem to "fix" by inventing a +retention policy nobody asked for yet. + +## 7. Dev/prod database isolation + +Every developer's local `DATABASE_URL` should point at a **Neon branch**, not the +production database — Neon branches are copy-on-write snapshots with their own connection +string, so a branch can be migrated, seeded, and reset freely without ever touching prod +rows. `docs/dev-db-isolation.md` is the full design doc; the shape that matters here is: +the production VM's `.env` keeps the real `DATABASE_URL` pointed at Neon's main branch, and +`DJANGO_ENV` is what decides which settings module (and therefore which behavioral +guardrails) apply — it does not, by itself, decide which database gets used. Nothing in +Django enforces that a `dev`-settings process can't be pointed at the prod `DATABASE_URL`; +the isolation is a matter of which connection string ends up in which `.env` file, and +that's a human discipline, not a code guarantee. + +One extension of this worth knowing: `backend/settings/dev.py` supports an optional second +database alias, `prod_readonly`, populated only when `PROD_DATABASE_URL` is set — this lets +local devtools (the ingestion/broadcast monitor) inspect real production data without +routing writes through it and without merging prod into the primary `default` alias. It's +only actually safe if the credentials behind `PROD_DATABASE_URL` come from a Postgres role +that is read-only at the database level (a `monitor_readonly` role with `SELECT`-only +grants, per the checklist in `docs/dev-db-isolation.md`) — Django does not enforce +read-only-ness itself; a read-write DSN in that variable would happily let devtools write +to prod. If you ever set this variable locally, verify the role actually rejects writes (an +`INSERT`/`CREATE TABLE` against the `prod_readonly` connection should error with `permission +denied`) before trusting it. + +## 8. Historical incident, still worth knowing + +`docs/prod-incident-2026-07-21-scheduler-outage.md` is the full forensic record behind §4's +sharp edge — worth reading in full if you're the one debugging a "the site works but nothing +in the background is happening" report, because that is exactly the symptom this incident +produced: `gunicorn` and `nextjs` stayed up the entire 8 days, so nothing looked wrong from +the outside, while `celery`/`celerybeat`/`broadcast-worker`/`scrape-worker` were all dead. +The one lasting change from that incident that has nothing to do with the `uv run` fix: a +stale or never-fired beat schedule is now a hard `FAIL` in `manage.py healthcheck`, not a +`WARN` — a scheduler that stopped firing is treated as an outage, not a suggestion, because +that distinction is what would have caught this incident in hours instead of the 8 days it +actually took (the outage was only found by chance, during unrelated forensics against +`/devtools/monitor`, not by any monitoring that existed at the time). + +## 9. The pending cutover: containers are built, not live + +A parallel effort in this same working tree has built a complete Docker Compose +replacement for everything in this document — one container per service, described in +`docker-compose.yml`, `backendServer/Dockerfile`, `Dockerfile.frontend`, and +`deploy/nginx/`, with the full rationale in `docs/adr/0001-containerization.md`. It has been +verified locally end-to-end. **None of it is live.** The Oracle VM does not have Docker +installed, the `ubuntu` user isn't in a `docker` group, the persistent bind-mount +directories the compose file expects don't exist on the box, and the seven systemd units +this document describes have not been touched. `DEPLOY.md` was rewritten during this same +session to describe the containerized stack as the deploy target — read it as a plan for +the next cutover, not as a description of what answers a request to +`api.thecommons.town` right now. `containerization.md` (a sibling human doc, written +alongside this one) covers what changes once that cutover happens — new service names, +TCP instead of a Unix socket for gunicorn, images instead of a checked-out venv, and how +the exact sharp edges in §4–§6 above either disappear or get re-solved a different way +inside a container. Until someone runs that cutover on the actual VM, treat every fact in +§1–§7 of this document as the operative reality, and treat `DEPLOY.md`'s Docker +instructions as a runbook waiting for its day one, not a record of today. + +## 10. Known gaps + +No push notification exists for a failed health check — `systemctl --failed` and +`journalctl -u healthcheck.service` are the only read paths today; nothing pages anyone. +There is no automated rollback if a deploy's smoke test fails after the systemd restarts +already happened — the units are already running the new code by the time the smoke test +runs, so a failing smoke test currently means "go SSH in and diagnose," not "the previous +version is automatically restored." The exact live contents of `gunicorn.service` and +`nextjs.service` are not verifiable from this repository at all, for the reason noted in +§2 — they were never committed; anyone needing their precise current flags should SSH in +and read them directly rather than trust any doc's transcription, including this one's +citation of an old `DEPLOY.md` revision. diff --git a/human-docs/design-system.md b/human-docs/design-system.md new file mode 100644 index 0000000..1496680 --- /dev/null +++ b/human-docs/design-system.md @@ -0,0 +1,147 @@ +# Design System + +*Reflects commit `5fe7a45`, 2026-08-01. Every token, class, and component below was read out of `theCommonsWeb/src/app/globals.css` and `theCommonsWeb/src/components/ui/` directly — not inferred from prose. Where `CODING_STYLE.md` (the repo's canonical style statement, which this doc complements and does not replace) disagrees with the stylesheet, both are stated and the disagreement is called out. For the Next.js routing/data-fetching layer these components sit inside, see `frontend.md`; for the product as a whole, see `overview.md`.* + +## 1. What this is, in one paragraph + +The Commons is styled to look like a broadsheet newspaper's classifieds page had a baby with early Craigslist: serif type, cream newsprint, black ink, hairline and thick column rules doing the job cards and shadows do everywhere else, and a bias toward packing information in rather than giving it room to breathe. This isn't a retro skin bolted onto a normal SaaS layout — it's the whole vocabulary. A contributor who reaches for a rounded card with a soft shadow because that's what every other product looks like is not making a small stylistic choice; they're building the wrong product. The reason it matters: The Commons is a *local* events bulletin for three small towns, not a venture-backed platform, and it wants to read like a community notice board someone would trust a neighbor posted to — not like a pitch deck. Density, rules, and serifs are load-bearing for that trust, not decoration. + +## 2. Tokens + +All color and font values are CSS custom properties declared once, on `:root`, in `globals.css`. **Nothing else defines colors or fonts** — Tailwind v4 is configured with a bare `@import "tailwindcss";` at the top of that same file and no `@theme` block, no `tailwind.config.js`/`.ts` anywhere in the repo. Components consume the tokens either as Tailwind arbitrary values (`bg-[var(--color-bg)]`) or the newer Tailwind v4 shorthand (`border-(--color-border)`) — both forms are in active use side by side; neither is preferred over the other in the current code. + +| Token | Value | What it's for | +|---|---|---| +| `--color-bg` | `#f4f1eb` | Page background — the "newsprint cream." Default surface for everything. | +| `--color-bg-alt` | `#eae6dd` | A shade darker than `--color-bg`. Hover states on rows/cards, secondary surfaces (the digest box border-panel, dropdown item hover). | +| `--color-text` | `#1a1a1a` | Near-black ink. Body text, and (deliberately) also the default link color — links are not blue here. | +| `--color-text-muted` | `#555555` | Secondary text: bylines, metadata lines, captions, timestamps. | +| `--color-link` | `#1a1a1a` | Same value as `--color-text` — links read as ink, not as a distinct color, until hovered. | +| `--color-link-hover` | `#8b0000` | Dark red. Link hover state. | +| `--color-border` | `#1a1a1a` | Primary rule color — same near-black as text. Used for thick rules, card outlines, the hard "print" shadow (see §5). | +| `--color-border-light` | `#c8c3b8` | Hairline rule color. Dividers, `
`, subordinate borders (e.g., the sidebar's column rule). | +| `--color-accent` | `#8b0000` | Dark red. Same value as `--color-link-hover`. Used sparingly: active/selected states, the "Verified" stamp, kicker labels, the accent rule under section nameplates. Not a general-purpose brand color — it means "selected" or "emphasis," and overusing it dilutes that. | +| `--font-headline` | `Georgia, "Times New Roman", Times, serif` | Headings (`h1`–`h6`, applied globally in `globals.css`), and anywhere a component sets `fontFamily: 'var(--font-headline)'` inline for a display-sized headline (e.g. the masthead `

` in `Header.tsx`, the section nameplate in `EventFeed.tsx`, the footer watermark). | +| `--font-body` | `Georgia, "Times New Roman", Times, serif` | Body copy. Identical value to `--font-headline` today — there is exactly one serif stack in this system, split into two token names for future flexibility, not because they currently differ. | +| `--font-sans` | `-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif` | The *only* sanctioned escape from serif — reserved for small-caps UI chrome that needs to read as a control rather than as editorial copy: `Button.tsx`'s base class, the "Verified" stamp's inline style. Don't reach for it for anything a reader would read as content. | +| `--focus-ring` | `2px solid #1a1a1a` | Keyboard focus outline, applied globally via `:focus-visible`. | +| `--focus-ring-offset` | `2px` | Offset for the above. | + +**Georgia is real, and it's loaded for free.** There is no `next/font` call anywhere in `src/app/layout.tsx` or elsewhere in the tree, no `@font-face`, no font files in an `assets/` directory (there isn't one). Georgia is a system font on essentially every OS that ships a browser; the stack falls through to Times New Roman / Times / generic serif if it's somehow missing. This is a deliberate performance and simplicity choice, not an oversight — it means zero font network requests, ever. + +**CODING_STYLE.md drift:** its `--font-sans` snippet says `system-ui, ...` — the real value in `globals.css` starts with `-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif` and never mentions `system-ui` at all. Trust the CSS. Everything else in `CODING_STYLE.md`'s token table matches the stylesheet exactly. + +**A token that's referenced but doesn't exist:** `TimeWindowSelector.tsx` and `SectionSelector.tsx` both reference `var(--color-ink)` (e.g. `text-[var(--color-ink)]`). There is no `--color-ink` custom property defined anywhere in `globals.css` or any other stylesheet in the repo — it isn't a synonym that resolves elsewhere. An undefined CSS custom property used without a fallback makes the property using it invalid at computed-value time, which for `color` means it falls back to the inherited value rather than doing anything the author intended. In practice this makes the inactive state of the time-window and section dropdowns render in whatever color they'd inherit rather than the near-black ink the rest of the system uses. This is a live bug, not a stylistic choice — the fix is renaming both usages to `--color-text`. + +## 3. Type scale + +There is no declared type scale (no `@theme` font-size tokens, no Tailwind config extending `fontSize`). What exists is the Tailwind v4 default scale (`text-xs` through `text-6xl`) plus a lot of arbitrary pixel values and `clamp()` expressions for display headlines that need to be fluid. Reading across `Header.tsx`, `EventFeed.tsx`, `EventRow.tsx`, `Sidebar.tsx`, `Footer.tsx`, and `MiniCalendar.tsx`, the sizes actually in use settle into these bands: + +| Use | Typical value | Example | +|---|---|---| +| Masthead / site title | `clamp(2.75rem, 8vw, 6rem)`, `--font-headline`, `font-black` | `Header.tsx`'s `

` | +| Section nameplate / featured headline | `clamp(1.6rem, 3.2vw, 3.25rem)` (varies by context), `--font-headline` | `SectionNameplate` and `FeaturedCard` in `EventFeed.tsx` | +| Card headline (non-featured) | `text-base` (1rem) to `text-2xl`/`text-3xl` for featured rows, `font-bold` | `EventRow.tsx` | +| Body copy | default `15px` set on `body` in `globals.css`, `line-height: 1.6` | site-wide default | +| Metadata / byline line | `text-xs` (0.75rem) or `text-[10px]`/`text-[11px]` | venue/time lines, footer copyright | +| Kicker / eyebrow labels | `text-[9px]`–`text-[10px]`, `uppercase`, `tracking-[0.18em]`–`tracking-[0.25em]`, `font-black` | "Featured Event," "Towns:," section labels throughout `Sidebar.tsx` and `Footer.tsx` | +| Calendar grid cells | `text-[8px]`–`text-[9px]` | `MiniCalendar.tsx` | +| Drop cap first letter | `3.2rem`, `--font-headline`, `font-weight: 700` | `.drop-cap::first-letter` in `globals.css` | + +The pattern to copy: headlines are large and fluid via `clamp()` set inline (`style={{ fontSize: '...' }}`), everything else is small, uppercase, and letter-spaced when it's a label rather than content. There is no `text-4xl`/`text-5xl`/`text-6xl` Tailwind class in use anywhere — display sizes are handled by `clamp()`, not the static scale, because they need to shrink on mobile without a breakpoint ladder. + +## 4. Spacing and layout + +No custom spacing scale — Tailwind's default spacing scale (the `p-1`, `px-4`, `gap-6`, etc. system) is used directly, no `@theme` override. Two layout constants recur: + +- **Content max-width `960px`** — `PageLayout.tsx`'s `
` (`max-w-[960px]`), the reading column. +- **Chrome max-width `1200px`** (written as Tailwind's `max-w-300`, i.e. `300 × 4px = 1200px`) — `Header.tsx`, `Footer.tsx`, `TopBar.tsx`, `TagsBar.tsx` all use this wider band for the masthead and nav strips that span above/below the reading column. + +The sidebar/content split (`PageLayout.tsx`) is a 4-column CSS grid, sidebar taking 1 of 4 columns on large screens (`lg:grid-cols-4`, `lg:col-span-1` / `lg:col-span-3`), stacking to a single column below `lg`. The sidebar is separated from content by a `border-r border-[var(--color-border-light)]` hairline rule, not a gap-only whitespace split — this is the density-over-whitespace principle showing up structurally, not just typographically. + +Density in practice: `Sidebar.tsx` stacks a dozen-plus distinct blocks (post button, date, calendar, view toggle, social link, tag filters, clear-filters, count, digest box) separated only by `
` hairlines with no card wrapper around any of them. That's the intended texture — a long, rule-divided column, not a stack of padded cards. + +## 5. Rules, borders, and the one shadow that's allowed + +This is the section that replaces "cards with shadows for elevation." The system has exactly four separation devices, and reaching for anything outside this list should be treated as a smell. + +| Device | CSS | Where it shows up | +|---|---|---| +| Hairline rule | `border-*-[var(--color-border-light)]`, 1px | Dividers between list rows, sidebar `
`, footer link-column separators | +| Standard rule | `border-*-[var(--color-border)]`, 1–2px | Card/panel outlines (`EventRow.tsx` non-featured, `Modal.tsx`), the header's rule under the tagline | +| Thick rule | `.rule-thick` (`border-top: 3px solid`) / `.rule-double` (`border-top: 3px double`), both against `--color-border` | Section breaks that need more visual weight than a standard rule; declared as utility classes in `globals.css` but not currently called from any component in `src/components/` — available, underused | +| Hard "print" shadow | `shadow-[3px_3px_0px_var(--color-border)]` — a flat, zero-blur, fully-opaque offset box, not a soft/blurred elevation shadow | `Modal.tsx`, the featured variant of `EventRow.tsx` | + +That fourth row is the one to read carefully, because "no drop shadows" (the banned-list rule in §7) sounds like it should ban this too, and it doesn't — this is a hard-edged, zero-blur, fully-opaque offset that reads as a printed card stacked on newsprint (think a paper cutout with a solid ink-black edge behind it), not a soft ambient shadow implying elevation off the page. The distinguishing test: if the shadow has any blur radius, or any transparency, it's the banned kind. `shadow-[3px_3px_0px_var(--color-border)]` is the *only* shadow value used anywhere in `theCommonsWeb/src`, and it's used in exactly two places. Don't invent a second shadow value — reuse this one if a component genuinely needs the "stacked card" effect, and default to a hairline or standard rule (row 1 or 2) for everything else. + +**Decision procedure:** reaching for a shadow to create separation between a block and its background? First ask whether a rule (row 1–3) does the job — it almost always does, since most separation here is "this is a distinct row/section," not "this is floating above the page." If the block genuinely needs to read as a card sitting on top of the page (a modal, a featured/pinned item), use the exact hard-shadow value above, never a new blurred one. + +## 6. Component inventory — `src/components/ui/` + +Eight components, seven exported from `index.ts` (`Banner.tsx` exists in the directory but is not re-exported — import it directly from `../ui/Banner` if needed, or add it to the barrel if this omission isn't intentional; nothing else in the tree currently imports it that way, so it's unclear whether the omission was deliberate). + +| Component | Purpose | Notable behavior | +|---|---|---| +| `Button` | The only button styling in the system | Three variants: `primary` (filled ink-black, inverts to accent-red on hover), `secondary` (outlined, transparent, bg-alt on hover — the default), `link` (looks like an inline text link, no border/padding). Two sizes (`sm`, `md`). Uppercase, letter-spaced, bold, `--font-sans` — buttons are chrome, not editorial content, hence the one place `--font-sans` is baked into a component rather than opted into. | +| `Badge` | Small inline tag/label pill — but square, not pill-shaped | `active` boolean toggles between an inverted (filled ink, cream text) and outlined (hairline border, muted text) treatment. No `border-radius` at all. | +| `Banner` | Dismissible strip, optionally sticky with scroll-direction hide/reveal | Two variants (`default`, `accent`) that only change the border color. Not in the `ui` barrel export — see above. | +| `Input` | Labeled text input | Label is a separate `