[codex] Publish social backfill hardening#152
Closed
therealityreport wants to merge 2 commits into
Closed
Conversation
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (122)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
Superseded by cleaned replacement PR #153 after final Ruff format correction. |
There was a problem hiding this comment.
Pull request overview
This PR hardens multiple “Publish social backfill” paths across social scraping/catalog pipelines, adds additional operational guardrails (proxy budgeting, savepoint-based persistence, preflight checks), and introduces/updates Supabase migrations and test coverage to lock in the new behavior.
Changes:
- Extend shared catalog scraping + diagnostics across platforms (YouTube surfaces incl. posts, Twitter bounded-window paging semantics, Threads/Facebook empty soft-block classification).
- Add resilience/operational controls (per-row DB savepoints for TikTok/Threads persistence, Scrapling fetcher option/metadata plumbing, Instagram public-lane proxy opt-in + run-scoped proxy budget ledger).
- Add migrations + admin/API/test coverage for SocialBlade, hashtag assignments, comment rollups/security, and various run lifecycle behaviors.
Reviewed changes
Copilot reviewed 119 out of 122 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| trr_backend/socials/youtube/posts_catalog/catalog.py | Adds surface selection + metadata improvements for YouTube shared catalog scraping. |
| trr_backend/socials/twitter/posts_catalog/catalog.py | Adjusts max page resolution semantics for bounded-window vs full-history runs. |
| trr_backend/socials/tiktok/posts_scrapling/persistence.py | Adds per-post savepoints + shared catalog upsert accounting. |
| trr_backend/socials/tiktok/posts_scrapling/job_runner.py | Threads ingest mode through job runner + emits catalog upsert counters. |
| trr_backend/socials/tiktok/posts_scrapling/fetcher.py | Adds optional Scrapling fetcher options + redaction-safe metadata plumbing. |
| trr_backend/socials/threads/posts_scrapling/persistence.py | Adds per-post savepoints and shared catalog upsert tracking. |
| trr_backend/socials/threads/posts_scrapling/fetcher.py | Adds optional Scrapling fetcher options + redaction-safe metadata plumbing. |
| trr_backend/socials/threads/posts_catalog/catalog.py | Classifies empty “soft block” results as retryable with structured metadata. |
| trr_backend/socials/socialblade/fetcher.py | Aligns proxy wiring + adds Scrapling fetcher options/metadata. |
| trr_backend/socials/pipelines/comments/init.py | Exposes an additional Instagram comments pipeline helper via all. |
| trr_backend/socials/media_url_safety.py | Expands safe host allowlist and minor formatting cleanup. |
| trr_backend/socials/instagram/posts_scrapling/fetcher.py | Adds Scrapling fetcher options + metadata into runtime instrumentation. |
| trr_backend/socials/instagram/permalink_metadata.py | Improves parsing by skipping excluded payloads and supporting XDTMediaDict inline payloads. |
| trr_backend/socials/instagram/comments_scrapling/public_mode.py | Introduces explicit public-proxy opt-in and splits public isolation invariants. |
| trr_backend/socials/instagram/comments_scrapling/proxy.py | Enables budgeted public proxy path + per-egress fingerprinting for pacing. |
| trr_backend/socials/instagram/comments_scrapling/proxy_budget.py | Adds run-scoped proxy budget config/math + ledger writer. |
| trr_backend/socials/facebook/posts_catalog/catalog.py | Adds empty “soft block” classification for shared Facebook catalog scraping. |
| trr_backend/socials/control_plane/run_lifecycle.py | Hardens deferred comments followup retry flow using CAS-style updates and fail-closed restore behavior. |
| trr_backend/socials/api/handlers/profile_reads.py | Extends profile hashtag reads with assignment_status filtering. |
| trr_backend/repositories/reddit_refresh.py | Adds robust extraction/sanitization for Reddit media URLs. |
| tests/test_startup_config.py | Adds tests to ensure Modal maintenance ownership validation fails closed appropriately. |
| tests/socials/youtube/test_scraper.py | Adds YouTube posts surface coverage + schema.org community post metadata tests. |
| tests/socials/twitter/test_twitter_posts_catalog.py | Locks in explicit 0 max_posts behavior as “unlimited” in bounded-window mode. |
| tests/socials/tiktok/posts_scrapling/test_persistence.py | Adds savepoint behavior tests + shared catalog mode persistence tests. |
| tests/socials/tiktok/posts_scrapling/test_job_runner.py | Verifies ingest mode propagation + catalog counters in job metadata. |
| tests/socials/threads/test_threads_posts_catalog.py | Adds test for retryable empty soft-block classification. |
| tests/socials/threads/posts_scrapling/test_persistence.py | Adds savepoint behavior tests for Threads persistence. |
| tests/socials/test_socialblade_fetcher.py | Updates SocialBlade fetcher proxy expectations (proxy vs proxy_rotator). |
| tests/socials/test_socialblade_business_api.py | Adds SocialBlade business API unit tests for mapping + auth headers. |
| tests/socials/test_socialblade_auth.py | Adds cookie health report redaction and loader behavior tests. |
| tests/socials/test_scrapling_transport.py | Adds tests for fetcher option parsing + proxy metadata redaction + JSON-safe metadata. |
| tests/socials/test_repair_socialblade_auth.py | Adds CLI control-flow tests for blocked validation and proxy-default behavior. |
| tests/socials/test_profile_dashboard.py | Hardens catalog progress/launch state interpretation and skip reason precedence. |
| tests/socials/test_instagram_permalink_metadata.py | Adds fixtures/tests for XDTMediaDict parsing and excluded payload behavior. |
| tests/socials/instagram/test_instagram_direct_catalog.py | Adds direct Instagram catalog soft-block and DB error handling tests. |
| tests/socials/instagram/comments_scrapling/test_public_relay_env_knobs.py | Adds env knob tests for retry limits + rate-scope hashing behavior. |
| tests/socials/instagram/comments_scrapling/test_public_decodo_isolation.py | Verifies public proxy opt-in and invariant enforcement around cookies/auth. |
| tests/socials/instagram/comments_scrapling/test_public_blocked_pause.py | Updates pause threshold test to match raised thresholds. |
| tests/socials/instagram/comments_scrapling/test_proxy.py | Tests new public proxy flag behavior and per-egress fingerprinting. |
| tests/socials/instagram/comments_scrapling/test_proxy_budget.py | Adds DB-free tests for proxy budget math/config resolution. |
| tests/socials/instagram/comments_scrapling/test_media_direct_guard.py | Tests fetcher budget kill-switch + CDN leak tripwire semantics. |
| tests/socials/instagram/comments_scrapling/test_job_runner_reply_only.py | Pins cursor strategy in reply-only job runner test to hit intended fast path. |
| tests/socials/instagram/comments_scrapling/test_job_runner_cancellation.py | Updates cancellation test dependencies for new setup behavior. |
| tests/socials/instagram/comments_scrapling/test_comments_lane_deploy_health.py | Adds import-time deploy drift guard for comments lane modules/entrypoints. |
| tests/socials/instagram/comments_scrapling/test_async_http_client.py | Adds async client factory tests (httpx default + curl_cffi opt-in/fallback). |
| tests/socials/facebook/test_facebook_posts_catalog.py | Adds test for retryable Facebook empty soft-block classification. |
| tests/scripts/test_prepare_named_secrets.py | Updates expected Modal secret defaults for new social knobs (SocialBlade, concurrency, mirror caps). |
| tests/scripts/test_enqueue_comments_audit_cursor_retries.py | Ensures CLI payload includes date bounds + load strategy defaults. |
| tests/scripts/test_classify_approval_blocked_comments.py | Adds script tests for selecting/classifying approval-blocked comment gaps. |
| tests/scripts/test_bravo_straggler_recovery.py | Updates delegated argv to include strategy/date filters and public relay defaults. |
| tests/repositories/test_youtube_catalog_backfill_diagnostics.py | Verifies profile_snapshot merge precedence in YouTube shared catalog metadata. |
| tests/repositories/test_social_run_lifecycle_repository.py | Adds tests ensuring advisory lock release before launch + restore-to-failed behavior on skipped retry. |
| tests/repositories/test_social_dispatch_metadata.py | Tests dispatch metadata touch updates only dispatch key and preserves other metadata. |
| tests/repositories/test_social_account_catalog_backfill_integration.py | Updates integration expectations around comments streaming enablement during catalog bootstrap. |
| tests/repositories/test_reddit_refresh.py | Adds tests for Reddit media URL sanitization + extraction. |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict.json | Adds fixture for inline fetch__XDTMediaDict post detail payload. |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict_null_fb.json | Adds fixture for inline payload with null FB count fields. |
| tests/fixtures/instagram/scrapling/ig_direct_badge_count_off_msys.json | Adds fixture for excluded IG direct badge query payload. |
| tests/fixtures/admin_socials_route_shape.json | Updates route-shape fixture for new admin endpoints. |
| tests/db/test_index_advisor_social_hot_paths.py | Adds test for nested advisor recommendation error surfacing. |
| tests/db/test_advisor_remediation_sql.py | Adds assertions for new security/RLS migration behavior. |
| tests/api/test_admin_socialblade.py | Adds SocialBlade cookie preflight checks before dispatch and error handling tests. |
| tests/api/test_admin_social_comments.py | Adds API test ensuring dry-run preview does not enqueue runs. |
| supabase/migrations/20260702185000_drop_social_unused_index_wave1.sql | Drops reviewed unused social indexes for fresh DB replays. |
| supabase/migrations/20260702184229_social_rollup_function_and_rate_pace_security.sql | Locks down rollup functions + rate pace table with RLS/grants/policies. |
| supabase/migrations/20260702181000_youtube_post_comment_rollups.sql | Introduces YouTube comment rollup table/functions/triggers and backfill. |
| supabase/migrations/20260702174000_hashtag_assignments_season_fk_index.sql | Adds missing FK index on social.hashtag_assignments(season_id). |
| supabase/migrations/20260629190000_canonical_social_hashtag_assignments.sql | Introduces canonical hashtag assignments + conflict capture with RLS/grants. |
| supabase/migrations/20260629144346_socialblade_snapshot_fk_indexes.sql | Adds FK indexes for SocialBlade snapshot tables. |
| supabase/migrations/20260629143025_supabase_security_advisor_rpc_and_vector_hardening.sql | Applies security advisor remediations (search_path, grants, extension schema). |
| supabase/migrations/20260629140000_instagram_comments_public_proxy_budget_ledger.sql | Codifies proxy budget ledger table with RLS/grants/indexes. |
| supabase/migrations/20260625172000_tiktok_post_comment_rollups.sql | Introduces TikTok comment rollup table/functions/triggers and backfill. |
| scripts/socials/reddit/repair_media_urls.py | Adds a CLI tool to repair/normalize Reddit mirror source URLs and statuses. |
| scripts/socials/instagram/enqueue_comments_audit_cursor_retries.py | Updates CLI defaults/args to include date bounds and public relay strategy. |
| scripts/socials/instagram/bravo_straggler_recovery.py | Updates recovery CLI defaults/args to public relay + optional date bounds. |
| scripts/modal/prepare_named_secrets.py | Adds/updates Modal named secret defaults for SocialBlade + concurrency caps. |
| scripts/db/verify_socialblade_snapshot_fk_indexes.sql | Adds verification SQL for SocialBlade snapshot FK/index alignment. |
| scripts/db/index_advisor_social_hot_paths.py | Surfaces nested advisor errors and marks warnings in generated reports. |
| requirements.modal.vision.lock.txt | Updates lockfile versions for Modal vision image. |
| requirements.modal.lean.lock.txt | Updates lockfile versions for Modal lean image. |
| requirements.modal.browser.lock.txt | Updates lockfile versions for Modal browser image. |
| requirements.lock.txt | Updates main lockfile versions and pins numpy/sympy/opencv to stable versions. |
| requirements.in | Pins numpy/sympy/opencv for consistent dependency resolution. |
| docs/workspace/scrapling-social-jobs.md | Adds shared Scrapling lane guidance and metadata redaction expectations. |
| docs/workspace/instagram-posts-scrapling.md | Links to shared Scrapling job guidance. |
| docs/workspace/instagram-comments-scrapling.md | Links to shared Scrapling job guidance. |
| docs/observability/modal-v439-v440-serve-backend-api-crash-loop-2026-05-28.md | Updates Modal deploy history stamp. |
| CONTEXT.md | Adds domain-language definitions for social ingestion and completeness semantics. |
| api/routers/admin_socialblade.py | Adds SocialBlade cookie health endpoint + dispatch preflight checks. |
| api/main.py | Makes Modal maintenance owner requirement fail-closed outside local/dev. |
| .env.example | Documents new SocialBlade business API env vars. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+5
to
+7
| For shared Scrapling process rules, optional browser tuning envs, and | ||
| redaction-safe metadata expectations across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
Comment on lines
+11
to
+13
| For shared Scrapling process rules and redaction-safe metadata expectations | ||
| across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
Comment on lines
+28
to
+30
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+62
to
+64
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+107
to
+108
| grant execute on function social.refresh_youtube_post_comment_rollup(uuid) to service_role; | ||
| grant execute on function social.refresh_youtube_post_comment_rollup_tg() to service_role; |
Comment on lines
+28
to
+30
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+62
to
+64
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+107
to
+108
| grant execute on function social.refresh_tiktok_post_comment_rollup(uuid) to service_role; | ||
| grant execute on function social.refresh_tiktok_post_comment_rollup_tg() to service_role; |
Contributor
Codex Exhaustive Code ReviewFindings
Validation AST parse passed for 96 changed Python files. I could not run pytest because |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.plan-worktool-finder dumps from this replacement PR scope.Validation
git diff --check origin/main...HEADgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 checkgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 format --check.venv/bin/python -m pytest tests/api -q(1086 passed).venv/bin/python -m pytest tests/test_modal_jobs.py tests/test_startup_config.py tests/repositories/test_social_dispatch_metadata.py -qRuntime follow-through
trr-backend-jobsdeploy/readiness follow-through was completed before PR prep.Notes