Skip to content

feat(backend): async reel generation via submit + poll job store - #34

Merged
upgradedev merged 1 commit into
mainfrom
feat/async-generation-backend
Jul 27, 2026
Merged

feat(backend): async reel generation via submit + poll job store#34
upgradedev merged 1 commit into
mainfrom
feat/async-generation-backend

Conversation

@upgradedev

@upgradedev upgradedev commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Summary

Today POST /reels/upload* runs the full generation pipeline synchronously
(minutes), which 504s at the edge — Firebase Hosting's proxy caps a single
request at ~60s, well under a real render's ~330-350s end-to-end. This adds
an async submit + poll pair alongside the existing sync endpoints so no
single HTTP request has to stay open for the whole generation:

  • POST /reels/jobs — identical body to POST /reels/upload (same decode +
    ingest validation, same 400s). Creates a job, starts generation on a
    background daemon thread, returns 202 {job_id, status: "queued"}
    immediately.
  • GET /reels/jobs/{job_id} — polls the stored status
    (queued / running / done / failed). 404 for an unknown job id;
    an unreadable/corrupt status object degrades to the same 404, never a 500.

Job state lives in the same storage backend the app already uses
(config.build_storage()), under jobs/<job_id>/status.json — not a
separate in-process registry — so status is readable across Cloud Run
instances (multi-instance, scale-to-zero). Job id is
secrets.token_urlsafe(18) (unguessable).

Thread-safety (the part called out explicitly): the background worker
never shares mutable adapter state with a request thread.

  • Provider/stitcher: always a fresh instance per job
    (_job_pipeline) — GenblazeMediaProvider.last_manifest and
    FakeMediaProvider.calls are both mutated per-call on the instance, so
    sharing them across threads would risk cross-contamination. No downside to
    freshness here since nothing else holds a reference to them.
  • Storage: FakeStorage (offline/tests) is pure in-memory with no remote
    substrate
    — a fresh instance would be invisible to request threads
    (including this process's own poll reads), and its plain dict/list
    mutations are safe to share under the GIL, so it stays shared
    (_job_storage). B2Storage (live) keeps a mutable local index list
    plus a remote read-modify-write merge on every put (a small race already
    documented/accepted in b2_storage.py for independent writers) — sharing
    the same Python object across threads would add an avoidable extra race
    on top of that (a shared list reassigned mid-iteration by another thread =
    a lost update). So live mode builds its own B2Storage per job; its
    constructor re-reads the bucket's index.jsonl, so nothing is lost — this
    is the exact multi-writer pattern
    test_concurrent_writers_union_by_key_not_last_writer_wins already proves
    the adapter handles correctly, just applied within one process instead of
    across two.

Deploy: deploy-cloudrun.sh gains one additive flag,
--no-cpu-throttling — Cloud Run only allocates CPU while a request is in
flight by default, which would starve the background thread between polls.
CLOUDRUN.md documents why, plus the honest limitation: the worker runs
in-process on the instance that accepted the submit — polling keeps that
instance warm, but if it's scaled down mid-job the work is lost with no
retry. Deliberate demo-scale tradeoff, not a production job queue (that would
be a durable queue + a Cloud Run Job, not a request-serving instance).

Strictly additive — existing behavior unchanged

  • _run_reel gained an optional pipeline parameter defaulting to the
    module-level _pipeline — every existing call site is unaffected (verified
    by tracing: for the default case pipeline.storage is _storage, so the
    honest-degrade fallback path is byte-identical to before).
  • _generate_from_photos/upload_reel had their bodies extracted
    (_build_spec, _decode_photos) with zero logic change — same exceptions,
    same messages, same order.
  • Zero existing tests modified. Confirmed no existing test white-box-tests
    _run_reel/_generate_from_photos/upload_reel internals (grepped
    tests/ first).

Test plan

  • New unit tests (tests/unit/test_jobs.py): job-id shape, create/read
    round-trip, unknown/corrupt/non-object status → None, status
    transitions preserve created_at, mark_failed records class name
    only (never the raw exception text), defensive fallback when a
    transition is called without a prior create, hostile job-id key
    sanitisation.
  • New integration tests (tests/integration/test_api_jobs.py): submit →
    202 + job_id; poll reaches done with a real provenance result body
    (sha256, manifest_hash, steps, playback_url); bridges step-count
    matches the sync path; bad input → 400 synchronously (no job created);
    unknown job id → 404; corrupt stored status → 404 not 500; forced
    failure (broken offline provider, mirroring the existing
    test_offline_provider_failure_is_a_real_500 pattern) → status
    failed, never a 500 anywhere in the flow, raw exception text never
    reaches the response.
  • CI green (watching now)

POST /reels/upload* runs the full pipeline synchronously, which can take
minutes and 504s at the edge (Firebase Hosting proxy cap ~60s). Add
POST /reels/jobs (202 + job_id) and GET /reels/jobs/{job_id} alongside the
existing sync endpoints: submit validates + starts generation on a daemon
thread and returns immediately; poll reads status back from a small JSON
object in the same storage backend the app already uses
(jobs/<job_id>/status.json), so it resolves across Cloud Run instances.

The background worker gets its own provider/stitcher always, and its own
B2Storage instance in live mode (FakeStorage is safely shared offline) so a
job thread never mutates the same mutable adapter state a request thread
touches. deploy-cloudrun.sh gains --no-cpu-throttling so the worker actually
gets CPU between polls; CLOUDRUN.md documents the honest limitation (the
worker runs in-process on the accepting instance — no external queue).

Existing endpoints are untouched behaviorally: _run_reel/_generate_from_photos
gained pure extract-method refactors (_build_spec, _decode_photos) and an
optional pipeline parameter, all behavior-preserving for current callers.
@upgradedev
upgradedev merged commit e73823f into main Jul 27, 2026
12 checks passed
@upgradedev
upgradedev deleted the feat/async-generation-backend branch July 27, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant