[codex] Publish social backfill hardening#153
Conversation
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (122)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Superseded by cleaned replacement PR #154 after lockfile freshness correction. |
There was a problem hiding this comment.
Pull request overview
This PR hardens social backfill + scraping lanes (YouTube/Twitter/TikTok/Threads/Instagram/SocialBlade) with better surface selection, per-run safety/telemetry, improved resilience on partial failures, and accompanying Supabase migrations + tests. It also tightens operational controls (Modal maintenance ownership, SocialBlade cookie preflight) and updates dependency locks and runbooks.
Changes:
- Add/standardize Scrapling “fetcher option” env-tuning + redaction-safe metadata across multiple lanes; improve proxy handling (including budgeted public-comments proxy guardrails).
- Extend shared catalog ingestion behaviors (e.g., YouTube surfaces incl. posts; Twitter bounded-window zero semantics; TikTok/Threads per-item DB savepoints + shared catalog upserts).
- Add/adjust Supabase migrations for rollups, RLS/grants hardening, FK indexes, and index cleanup; expand test coverage and operational scripts/docs.
backend_surface: FastAPI admin routes (SocialBlade cookie health + batch preflight behavior), internal startup checks, multiple socials scraping/persistence paths, Supabase migrations, and operator scripts.
contracts_changed: Yes (e.g., get_profile_hashtags(..., assignment_status=...); new admin SocialBlade cookie health route; admin socials route-shape fixture changes).
migrations_added: Yes (multiple new migrations under supabase/migrations/*, including rollup tables/functions + RLS/grants + FK index + unused-index drops).
downstream_updates: None included here (no TRR-APP changes in this PR).
validation_run: As stated in PR description (ruff checks + pytest suites including 1086 passing tests, plus focused social backfill sets).
Reviewed changes
Copilot reviewed 119 out of 122 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| trr_backend/socials/youtube/posts_catalog/catalog.py | Add configurable requested surfaces + richer metadata |
| trr_backend/socials/twitter/posts_catalog/catalog.py | Treat explicit zero max_posts as unlimited in bounded-window |
| trr_backend/socials/tiktok/posts_scrapling/persistence.py | Per-post savepoints + optional shared catalog upsert counters |
| trr_backend/socials/tiktok/posts_scrapling/job_runner.py | Plumb pipeline_ingest_mode + catalog counters into job metadata |
| trr_backend/socials/tiktok/posts_scrapling/fetcher.py | Add Scrapling env-tuning + safe proxy/fetcher metadata |
| trr_backend/socials/threads/posts_scrapling/persistence.py | Per-post savepoints + shared catalog write path |
| trr_backend/socials/threads/posts_scrapling/fetcher.py | Add Scrapling env-tuning + safe proxy/fetcher metadata |
| trr_backend/socials/threads/posts_catalog/catalog.py | Classify empty “soft block” results as retryable |
| trr_backend/socials/socialblade/fetcher.py | Add Scrapling env-tuning + swap to browser proxy param + metadata |
| trr_backend/socials/pipelines/comments/init.py | Export new Instagram comments catalog streaming helper |
| trr_backend/socials/media_url_safety.py | Allow additional safe YouTube host + minor formatting |
| trr_backend/socials/instagram/posts_scrapling/fetcher.py | Add Scrapling env-tuning + fetcher metadata emission |
| trr_backend/socials/instagram/permalink_metadata.py | Prefer fetch__XDTMediaDict payloads; exclude noisy friendlyName payloads |
| trr_backend/socials/instagram/comments_scrapling/public_mode.py | Allow budgeted proxy opt-in while keeping cookie/auth invariants |
| trr_backend/socials/instagram/comments_scrapling/proxy.py | Budgeted public proxy behavior + per-egress fingerprinting |
| trr_backend/socials/instagram/comments_scrapling/proxy_budget.py | New: run-scoped proxy budget math + ledger writer |
| trr_backend/socials/facebook/posts_catalog/catalog.py | Classify empty “soft block” results as retryable |
| trr_backend/socials/control_plane/run_lifecycle.py | Harden deferred followup retry flow (CAS update + launch outside lock) |
| trr_backend/socials/api/handlers/profile_reads.py | Add assignment_status filter plumbing to hashtags read |
| trr_backend/repositories/reddit_refresh.py | Robust Reddit media URL extraction + sanitization |
| api/routers/admin_socialblade.py | Add cookie health endpoint + preflight cookie validation before dispatch |
| api/main.py | Fail-closed Modal maintenance ownership outside local/dev |
| scripts/socials/reddit/repair_media_urls.py | New: repair + retry Reddit media mirror URLs |
| scripts/socials/instagram/enqueue_comments_audit_cursor_retries.py | Default to public_relay + optional date bounds |
| scripts/socials/instagram/bravo_straggler_recovery.py | Default to public_relay + optional date bounds propagation |
| scripts/modal/prepare_named_secrets.py | Add SocialBlade proxy defaults + new concurrency knobs |
| scripts/db/index_advisor_social_hot_paths.py | Surface nested advisor errors as warnings |
| scripts/db/verify_socialblade_snapshot_fk_indexes.sql | New: verify FK indexes exist/valid |
| docs/workspace/scrapling-social-jobs.md | New: shared Scrapling lane conventions + env-tuning keys |
| docs/workspace/instagram-posts-scrapling.md | Link to shared Scrapling job conventions |
| docs/workspace/instagram-comments-scrapling.md | Link to shared Scrapling job conventions |
| docs/observability/modal-v439-v440-serve-backend-api-crash-loop-2026-05-28.md | Update deploy history stamp block |
| CONTEXT.md | New: domain language definitions for social ingestion |
| .env.example | Add SocialBlade business API envs |
| requirements.in | Pin numpy/sympy/opencv-python versions |
| requirements.lock.txt | Lock updates (fastapi/google-auth/firestore/etc; numpy/opencv/sympy pins) |
| requirements.modal.lean.lock.txt | Modal lock updates (fastapi/google-auth/firestore/etc) |
| requirements.modal.browser.lock.txt | Modal browser lock updates (fastapi/google-auth/firestore/etc) |
| requirements.modal.vision.lock.txt | Modal vision lock updates (fastapi/google-auth/firestore/etc) |
| supabase/migrations/20260625172000_tiktok_post_comment_rollups.sql | New: TikTok comment rollup table + refresh functions + triggers |
| supabase/migrations/20260629140000_instagram_comments_public_proxy_budget_ledger.sql | New: budget ledger table + RLS/grants |
| supabase/migrations/20260629143025_supabase_security_advisor_rpc_and_vector_hardening.sql | New: advisor-driven RPC hardening + vector extension schema |
| supabase/migrations/20260629144346_socialblade_snapshot_fk_indexes.sql | New: add missing FK indexes for SocialBlade snapshot tables |
| supabase/migrations/20260629190000_canonical_social_hashtag_assignments.sql | New: canonical hashtag assignments + conflicts table + RLS/grants |
| supabase/migrations/20260702174000_hashtag_assignments_season_fk_index.sql | New: partial index for season_id FK hot path |
| supabase/migrations/20260702181000_youtube_post_comment_rollups.sql | New: YouTube comment rollup table + refresh functions + triggers |
| supabase/migrations/20260702184229_social_rollup_function_and_rate_pace_security.sql | New: lock down rollup RPC + ig_comment_rate_pace RLS/grants |
| supabase/migrations/20260702185000_drop_social_unused_index_wave1.sql | New: drop vetted unused indexes (wave 1) |
| tests/test_startup_config.py | Add fail-closed Modal owner tests + scheduler env |
| tests/socials/youtube/test_scraper.py | Add posts-surface tests + schema.org community post parsing |
| tests/repositories/test_youtube_catalog_backfill_diagnostics.py | Ensure config profile_snapshot merge precedence |
| tests/socials/twitter/test_twitter_posts_catalog.py | Assert explicit zero max_posts -> unlimited max_pages |
| tests/socials/tiktok/posts_scrapling/test_persistence.py | Savepoint + shared catalog mode tests |
| tests/socials/tiktok/posts_scrapling/test_job_runner.py | pipeline_ingest_mode passthrough + counters tests |
| tests/socials/threads/test_threads_posts_catalog.py | Empty soft-block classification test |
| tests/socials/threads/posts_scrapling/test_persistence.py | Savepoint + shared catalog mode tests |
| tests/socials/test_scrapling_transport.py | New tests for option resolver + proxy metadata redaction + JSON safety |
| tests/socials/test_socialblade_fetcher.py | Update to browser proxy parameter expectations |
| tests/socials/test_socialblade_business_api.py | New: business API mapping tests |
| tests/socials/test_socialblade_auth.py | Cookie health report redaction + loader behavior tests |
| tests/socials/test_repair_socialblade_auth.py | New: CLI repair flow tests (1020 blocked validation contract) |
| tests/socials/test_profile_dashboard.py | Catalog progress/launch edge-case tests |
| tests/socials/test_instagram_permalink_metadata.py | New fixtures + XDTMediaDict parsing + excluded payload tests |
| tests/socials/instagram/test_instagram_direct_catalog.py | New: direct catalog soft-block + db error behavior tests |
| tests/socials/instagram/comments_scrapling/test_public_relay_env_knobs.py | Expand env knob tests incl. rate scoping |
| tests/socials/instagram/comments_scrapling/test_public_decodo_isolation.py | Public proxy opt-in + invariants tests |
| tests/socials/instagram/comments_scrapling/test_public_blocked_pause.py | Update pause threshold test expectations |
| tests/socials/instagram/comments_scrapling/test_proxy.py | Public proxy flag behavior + per-egress fingerprint test |
| tests/socials/instagram/comments_scrapling/test_proxy_budget.py | New: budget math tests |
| tests/socials/instagram/comments_scrapling/test_media_direct_guard.py | New: budget kill-switch + CDN leak tripwire tests |
| tests/socials/instagram/comments_scrapling/test_job_runner_reply_only.py | Pin cursor strategy for reply-only retry test |
| tests/socials/instagram/comments_scrapling/test_job_runner_cancellation.py | Stub new audit metadata preload for cancellation invariant |
| tests/socials/instagram/comments_scrapling/test_comments_lane_deploy_health.py | New: import-time drift guard for comments lane deploy image |
| tests/socials/instagram/comments_scrapling/test_async_http_client.py | New: async transport factory (httpx/curl_cffi) tests |
| tests/socials/facebook/test_facebook_posts_catalog.py | Empty soft-block classification test |
| tests/scripts/test_prepare_named_secrets.py | Validate new default env injections |
| tests/scripts/test_enqueue_comments_audit_cursor_retries.py | Assert new CLI defaults + date args |
| tests/scripts/test_bravo_straggler_recovery.py | Assert delegated argv includes date args + public_relay |
| tests/scripts/test_classify_approval_blocked_comments.py | New: payload and apply confirmation tests |
| tests/repositories/test_social_run_lifecycle_repository.py | Lock/CAS + launch-outside-lock followup retry tests |
| tests/repositories/test_social_dispatch_metadata.py | New: dispatch metadata touch preserves unrelated metadata |
| tests/repositories/test_social_account_catalog_backfill_integration.py | Comments streaming behavior in launch integration |
| tests/repositories/test_reddit_refresh.py | Media URL sanitize/extract tests |
| tests/db/test_index_advisor_social_hot_paths.py | Ensure nested advisor errors surface in report |
| tests/db/test_advisor_remediation_sql.py | Assert new security migrations contain expected remediation SQL |
| tests/api/test_admin_socialblade.py | Require cookie health preflight before dispatch; test unhealthy case |
| tests/api/test_admin_social_comments.py | New: dry-run comments scrape uses preview only |
| tests/fixtures/admin_socials_route_shape.json | Add new admin socials routes |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict.json | New: XDT media fixture |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict_null_fb.json | New: XDT null-FB fixture |
| tests/fixtures/instagram/scrapling/ig_direct_badge_count_off_msys.json | New: excluded friendlyName fixture |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| elif not any( | ||
| _normalize_non_negative_int(first_page_counts.get(surface)) > 0 for surface in ("videos", "shorts", "posts") | ||
| ): |
| pending_dispatch.append((item, safe_handle)) | ||
|
|
||
| if pending_dispatch: | ||
| validation_handle = pending_dispatch[0][1] |
|
|
||
| For shared Scrapling process rules, optional browser tuning envs, and | ||
| redaction-safe metadata expectations across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
|
|
||
| For shared Scrapling process rules and redaction-safe metadata expectations | ||
| across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public |
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public |
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public |
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public |
Codex Exhaustive Code ReviewFindings
Validation Reviewed only the requested diff range. I also ran a read-only AST parse over changed Python files; it passed. I did not run the test suite, and I did not modify files. |
Summary
.plan-worktool-finder dumps from this replacement PR scope.Validation
git diff --check origin/main...HEADgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 checkgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 format --check.venv/bin/python -m pytest tests/api -q(1086 passed).venv/bin/python -m pytest tests/test_modal_jobs.py tests/test_startup_config.py tests/repositories/test_social_dispatch_metadata.py -qRuntime follow-through
trr-backend-jobsdeploy/readiness follow-through was completed before PR prep.Notes