[codex] Publish social backfill hardening#154
Merged
therealityreport merged 5 commits intoJul 5, 2026
Conversation
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (143)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR hardens multiple “social backfill” lanes (Instagram comments/public proxy budget controls, shared catalog ingestion, SocialBlade repair + preflight, and rollup/security migrations), while expanding YouTube catalog scraping to include community posts and tightening several operational defaults and diagnostics.
Changes:
- Add proxy-budgeted public-lane controls for Instagram comments (opt-in proxy enablement, spend guardrails/ledger, and stronger isolation proofing) plus related job/metadata/test coverage.
- Expand shared-account catalog ingestion/hardening across platforms (YouTube surfaces incl. posts, TikTok/Threads per-post savepoints + shared catalog writes, soft-block empty-result classification).
- Add/refresh Supabase migrations and operational tooling (rollup tables/functions, RLS/grants hardening, FK indexes, unused-index drop wave, lockfile refresh, and verification scripts).
Skill completion contract (senior-backend):
- backend_surface: FastAPI routes/handlers, scraping workers/fetchers, repositories, scripts, and Supabase migrations.
- contracts_changed: Yes (new admin SocialBlade cookie health endpoint; hashtags read handler adds
assignment_status; additional admin socials route shapes/tests). - migrations_added: Yes (multiple new
supabase/migrations/*.sql). - downstream_updates: Not included in this PR (no TRR-APP/screenalytics changes visible here).
- validation_run: Per PR description: ruff check/format, pytest (tests/api suite + targeted tests), lockfile regen diff, plus focused social backfill test set.
Reviewed changes
Copilot reviewed 119 out of 122 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| trr_backend/socials/youtube/posts_catalog/catalog.py | Add requested surfaces incl. posts |
| trr_backend/socials/twitter/posts_catalog/catalog.py | Treat explicit 0 limits as unlimited |
| trr_backend/socials/tiktok/posts_scrapling/persistence.py | Per-post savepoints + shared catalog upsert |
| trr_backend/socials/tiktok/posts_scrapling/job_runner.py | Track/persist catalog upsert counters |
| trr_backend/socials/tiktok/posts_scrapling/fetcher.py | Scrapling tuning opts + proxy metadata |
| trr_backend/socials/threads/posts_scrapling/persistence.py | Per-post savepoints + shared catalog upsert |
| trr_backend/socials/threads/posts_scrapling/fetcher.py | Scrapling tuning opts + proxy metadata |
| trr_backend/socials/threads/posts_catalog/catalog.py | Classify empty soft-block results |
| trr_backend/socials/socialblade/fetcher.py | Scrapling tuning opts + proxy handling |
| trr_backend/socials/pipelines/comments/init.py | Export new pipeline helper |
| trr_backend/socials/media_url_safety.py | Allow ggpht.com; formatting |
| trr_backend/socials/instagram/posts_scrapling/fetcher.py | Scrapling tuning opts + metadata |
| trr_backend/socials/instagram/permalink_metadata.py | Parse fetch__XDTMediaDict + exclude payloads |
| trr_backend/socials/instagram/comments_scrapling/public_mode.py | Public lane proxy opt-in + isolation proof |
| trr_backend/socials/instagram/comments_scrapling/proxy.py | Budgeted public proxy fingerprinting |
| trr_backend/socials/instagram/comments_scrapling/proxy_budget.py | New per-run proxy budget math + ledger write |
| trr_backend/socials/facebook/posts_catalog/catalog.py | Classify empty soft-block results |
| trr_backend/socials/control_plane/run_lifecycle.py | CAS followup state + safer retry launch |
| trr_backend/socials/api/handlers/profile_reads.py | Add hashtag assignment_status filter |
| trr_backend/repositories/reddit_refresh.py | Extract/sanitize Reddit media URLs |
| tests/test_startup_config.py | Modal maintenance owner fail-closed tests |
| tests/socials/youtube/test_scraper.py | Add posts surface + schema.org tests |
| tests/socials/twitter/test_twitter_posts_catalog.py | Explicit 0 max_posts => unlimited pages |
| tests/socials/tiktok/posts_scrapling/test_job_runner.py | Assert shared mode passes ingest mode |
| tests/socials/threads/test_threads_posts_catalog.py | Empty soft-block retryable classification |
| tests/socials/threads/posts_scrapling/test_persistence.py | Savepoint behavior tests |
| tests/socials/test_socialblade_fetcher.py | Proxy plumbing expectation update |
| tests/socials/test_socialblade_business_api.py | New business API unit tests |
| tests/socials/test_socialblade_auth.py | Cookie health report redaction/loader tests |
| tests/socials/test_scrapling_transport.py | New helpers: tuning options + proxy metadata |
| tests/socials/test_repair_socialblade_auth.py | New cookie repair CLI control-flow tests |
| tests/socials/test_profile_dashboard.py | Catalog progress/skip reason semantics tests |
| tests/socials/test_instagram_permalink_metadata.py | Fixture-based XDT + exclude payload tests |
| tests/socials/instagram/test_instagram_direct_catalog.py | New direct catalog backfill tests |
| tests/socials/instagram/comments_scrapling/test_public_relay_env_knobs.py | New env knob + rate scope tests |
| tests/socials/instagram/comments_scrapling/test_public_decodo_isolation.py | Public proxy flag + allow_proxy tests |
| tests/socials/instagram/comments_scrapling/test_public_blocked_pause.py | Update pause threshold assertions |
| tests/socials/instagram/comments_scrapling/test_proxy.py | Public proxy flag + per-egress fingerprint test |
| tests/socials/instagram/comments_scrapling/test_proxy_budget.py | New proxy budget math tests |
| tests/socials/instagram/comments_scrapling/test_media_direct_guard.py | New CDN leak + budget kill-switch tests |
| tests/socials/instagram/comments_scrapling/test_job_runner_reply_only.py | Pin authenticated cursor strategy in test |
| tests/socials/instagram/comments_scrapling/test_job_runner_cancellation.py | Stub now_utc + audit-cursor read conn |
| tests/socials/instagram/comments_scrapling/test_comments_lane_deploy_health.py | Import-time deploy drift guard |
| tests/socials/instagram/comments_scrapling/test_async_http_client.py | Async transport factory tests |
| tests/socials/facebook/test_facebook_posts_catalog.py | Empty soft-block retryable classification |
| tests/scripts/test_prepare_named_secrets.py | New modal/social defaults assertions |
| tests/scripts/test_enqueue_comments_audit_cursor_retries.py | Add date bounds + strategy payload asserts |
| tests/scripts/test_classify_approval_blocked_comments.py | New approval-gap classification tests |
| tests/scripts/test_bravo_straggler_recovery.py | Delegate argv carries date bounds + strategy |
| tests/repositories/test_youtube_catalog_backfill_diagnostics.py | Profile snapshot merge precedence test |
| tests/repositories/test_social_run_lifecycle_repository.py | CAS retry semantics + lock release behavior |
| tests/repositories/test_social_dispatch_metadata.py | New dispatch metadata touch test |
| tests/repositories/test_social_account_catalog_backfill_integration.py | Streaming comments enablement assertions |
| tests/repositories/test_reddit_refresh.py | Reddit media URL sanitization tests |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict.json | New Instagram fixture |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict_null_fb.json | New Instagram fixture |
| tests/fixtures/instagram/scrapling/ig_direct_badge_count_off_msys.json | New IG excluded payload fixture |
| tests/fixtures/admin_socials_route_shape.json | Add new admin socials routes |
| tests/db/test_index_advisor_social_hot_paths.py | Surface nested advisor errors in report |
| tests/db/test_advisor_remediation_sql.py | Assert new security migrations content |
| tests/api/test_admin_socialblade.py | Batch refresh preflight health gating tests |
| tests/api/test_admin_social_comments.py | New dry-run comments scrape handler test |
| supabase/migrations/20260702185000_drop_social_unused_index_wave1.sql | Drop unused indexes (wave 1) |
| supabase/migrations/20260702184229_social_rollup_function_and_rate_pace_security.sql | Rollup function hardening + RLS |
| supabase/migrations/20260702181000_youtube_post_comment_rollups.sql | YouTube comment rollups + triggers |
| supabase/migrations/20260702174000_hashtag_assignments_season_fk_index.sql | Add FK index for season_id |
| supabase/migrations/20260629190000_canonical_social_hashtag_assignments.sql | New canonical hashtag assignments tables |
| supabase/migrations/20260629144346_socialblade_snapshot_fk_indexes.sql | Add SocialBlade snapshot FK indexes |
| supabase/migrations/20260629143025_supabase_security_advisor_rpc_and_vector_hardening.sql | Security advisor RPC/vector hardening |
| supabase/migrations/20260629140000_instagram_comments_public_proxy_budget_ledger.sql | Proxy budget ledger table + RLS |
| supabase/migrations/20260625172000_tiktok_post_comment_rollups.sql | TikTok comment rollups + triggers |
| scripts/socials/reddit/repair_media_urls.py | New Reddit media URL repair tool |
| scripts/socials/instagram/enqueue_comments_audit_cursor_retries.py | Add date bounds + default strategy |
| scripts/socials/instagram/bravo_straggler_recovery.py | Add date bounds + default strategy |
| scripts/modal/prepare_named_secrets.py | Add SocialBlade + concurrency defaults |
| scripts/db/verify_socialblade_snapshot_fk_indexes.sql | New index verification SQL |
| scripts/db/index_advisor_social_hot_paths.py | Capture advisor recommendation errors |
| requirements.modal.vision.lock.txt | Refresh pinned deps (modal vision) |
| requirements.modal.lean.lock.txt | Refresh pinned deps (modal lean) |
| requirements.modal.browser.lock.txt | Refresh pinned deps (modal browser) |
| requirements.lock.txt | Refresh lockfile (py 3.11 compile) |
| requirements.in | Pin numpy/sympy/opencv |
| docs/workspace/scrapling-social-jobs.md | New shared Scrapling practices doc |
| docs/workspace/instagram-posts-scrapling.md | Link to shared Scrapling doc |
| docs/workspace/instagram-comments-scrapling.md | Link to shared Scrapling doc |
| docs/observability/modal-v439-v440-serve-backend-api-crash-loop-2026-05-28.md | Update deploy history stamp |
| CONTEXT.md | New domain language glossary |
| api/routers/admin_socialblade.py | Cookie health endpoint + batch preflight |
| api/main.py | Fail-closed Modal owner requirement |
| .env.example | Add SocialBlade business API envs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+5
to
+7
| For shared Scrapling process rules, optional browser tuning envs, and | ||
| redaction-safe metadata expectations across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
Comment on lines
+11
to
+13
| For shared Scrapling process rules and redaction-safe metadata expectations | ||
| across social lanes, see | ||
| [Scrapling Social Jobs](/Users/thomashulihan/Projects/TRR/TRR-Backend/docs/workspace/scrapling-social-jobs.md). |
Comment on lines
+25
to
+30
| create or replace function social.refresh_youtube_post_comment_rollup(target_video_id uuid) | ||
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+59
to
+64
| create or replace function social.refresh_youtube_post_comment_rollup_tg() | ||
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+25
to
+30
| create or replace function social.refresh_tiktok_post_comment_rollup(target_post_id uuid) | ||
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+59
to
+64
| create or replace function social.refresh_tiktok_post_comment_rollup_tg() | ||
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Contributor
Codex Exhaustive Code ReviewFindings
Residual Risks
Validation
|
therealityreport
deleted the
codex/publish/trr-backend/20260705-174033-clean3
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
requirements.lock.txtwith the CI--python-version 3.11command and remove large generated.plan-worktool-finder dumps.Validation
git diff --check origin/main...HEADgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 checkgit diff --name-only --diff-filter=ACMR origin/main...HEAD -- '*.py' | xargs -r uvx ruff@0.14.4 format --checkcp requirements.lock.txt /tmp/requirements.lock.before-ci-check.txt && uv pip compile requirements.in --python-version 3.11 -o requirements.lock.txt && diff -u /tmp/requirements.lock.before-ci-check.txt requirements.lock.txt.venv/bin/python -m pytest tests/api -q(1086 passed).venv/bin/python -m pytest tests/test_modal_jobs.py tests/test_startup_config.py tests/repositories/test_social_dispatch_metadata.py -qRuntime follow-through
trr-backend-jobsdeploy/readiness follow-through was completed before PR prep.Notes