feat(backend): async reel generation via submit + poll job store - #34
Merged
Conversation
POST /reels/upload* runs the full pipeline synchronously, which can take
minutes and 504s at the edge (Firebase Hosting proxy cap ~60s). Add
POST /reels/jobs (202 + job_id) and GET /reels/jobs/{job_id} alongside the
existing sync endpoints: submit validates + starts generation on a daemon
thread and returns immediately; poll reads status back from a small JSON
object in the same storage backend the app already uses
(jobs/<job_id>/status.json), so it resolves across Cloud Run instances.
The background worker gets its own provider/stitcher always, and its own
B2Storage instance in live mode (FakeStorage is safely shared offline) so a
job thread never mutates the same mutable adapter state a request thread
touches. deploy-cloudrun.sh gains --no-cpu-throttling so the worker actually
gets CPU between polls; CLOUDRUN.md documents the honest limitation (the
worker runs in-process on the accepting instance — no external queue).
Existing endpoints are untouched behaviorally: _run_reel/_generate_from_photos
gained pure extract-method refactors (_build_spec, _decode_photos) and an
optional pipeline parameter, all behavior-preserving for current callers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Today
POST /reels/upload*runs the full generation pipeline synchronously(minutes), which 504s at the edge — Firebase Hosting's proxy caps a single
request at ~60s, well under a real render's ~330-350s end-to-end. This adds
an async submit + poll pair alongside the existing sync endpoints so no
single HTTP request has to stay open for the whole generation:
POST /reels/jobs— identical body toPOST /reels/upload(same decode +ingest validation, same 400s). Creates a job, starts generation on a
background daemon thread, returns 202
{job_id, status: "queued"}immediately.
GET /reels/jobs/{job_id}— polls the stored status(
queued/running/done/failed). 404 for an unknown job id;an unreadable/corrupt status object degrades to the same 404, never a 500.
Job state lives in the same storage backend the app already uses
(
config.build_storage()), underjobs/<job_id>/status.json— not aseparate in-process registry — so status is readable across Cloud Run
instances (multi-instance, scale-to-zero). Job id is
secrets.token_urlsafe(18)(unguessable).Thread-safety (the part called out explicitly): the background worker
never shares mutable adapter state with a request thread.
(
_job_pipeline) —GenblazeMediaProvider.last_manifestandFakeMediaProvider.callsare both mutated per-call on the instance, sosharing them across threads would risk cross-contamination. No downside to
freshness here since nothing else holds a reference to them.
FakeStorage(offline/tests) is pure in-memory with no remotesubstrate — a fresh instance would be invisible to request threads
(including this process's own poll reads), and its plain dict/list
mutations are safe to share under the GIL, so it stays shared
(
_job_storage).B2Storage(live) keeps a mutable localindexlistplus a remote read-modify-write merge on every
put(a small race alreadydocumented/accepted in
b2_storage.pyfor independent writers) — sharingthe same Python object across threads would add an avoidable extra race
on top of that (a shared list reassigned mid-iteration by another thread =
a lost update). So live mode builds its own
B2Storageper job; itsconstructor re-reads the bucket's
index.jsonl, so nothing is lost — thisis the exact multi-writer pattern
test_concurrent_writers_union_by_key_not_last_writer_winsalready provesthe adapter handles correctly, just applied within one process instead of
across two.
Deploy:
deploy-cloudrun.shgains one additive flag,--no-cpu-throttling— Cloud Run only allocates CPU while a request is inflight by default, which would starve the background thread between polls.
CLOUDRUN.mddocuments why, plus the honest limitation: the worker runsin-process on the instance that accepted the submit — polling keeps that
instance warm, but if it's scaled down mid-job the work is lost with no
retry. Deliberate demo-scale tradeoff, not a production job queue (that would
be a durable queue + a Cloud Run Job, not a request-serving instance).
Strictly additive — existing behavior unchanged
_run_reelgained an optionalpipelineparameter defaulting to themodule-level
_pipeline— every existing call site is unaffected (verifiedby tracing: for the default case
pipeline.storage is _storage, so thehonest-degrade fallback path is byte-identical to before).
_generate_from_photos/upload_reelhad their bodies extracted(
_build_spec,_decode_photos) with zero logic change — same exceptions,same messages, same order.
_run_reel/_generate_from_photos/upload_reelinternals (greppedtests/first).Test plan
tests/unit/test_jobs.py): job-id shape, create/readround-trip, unknown/corrupt/non-object status →
None, statustransitions preserve
created_at,mark_failedrecords class nameonly (never the raw exception text), defensive fallback when a
transition is called without a prior
create, hostile job-id keysanitisation.
tests/integration/test_api_jobs.py): submit →202 + job_id; poll reaches
donewith a real provenance result body(sha256, manifest_hash, steps, playback_url); bridges step-count
matches the sync path; bad input → 400 synchronously (no job created);
unknown job id → 404; corrupt stored status → 404 not 500; forced
failure (broken offline provider, mirroring the existing
test_offline_provider_failure_is_a_real_500pattern) → statusfailed, never a 500 anywhere in the flow, raw exception text neverreaches the response.