[codex] Publish social backfill hardening#151
Closed
therealityreport wants to merge 2 commits into
Closed
Conversation
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (144)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Hardens multiple social backfill/scraping lanes (Instagram comments/public relay + proxy budgeting, shared catalog ingestion across platforms, SocialBlade repair/health checks, and Scrapling runtime diagnostics) and ships supporting migrations, scripts, docs, and tests to lock in the operational contracts.
Changes:
- Add/standardize Scrapling browser tuning + redaction-safe runtime metadata across several lanes (Instagram posts, TikTok posts, Threads posts, SocialBlade).
- Harden ingestion/backfill behavior and diagnostics (shared catalog surfaces, bounded-window semantics, soft-block classification, per-post savepoints, pipeline ingest-mode propagation).
- Add Supabase migrations for rollups/indexing/security posture + operational scripts/docs/tests for SocialBlade and Instagram comments throughput/proxy-budget controls.
Reviewed changes
Copilot reviewed 128 out of 144 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| trr_backend/socials/youtube/posts_catalog/catalog.py | Add requested surfaces + richer retrieval/profile metadata for shared YouTube catalog scraping. |
| trr_backend/socials/twitter/posts_catalog/catalog.py | Fix max-pages resolution so explicit 0 means “unlimited” in bounded windows. |
| trr_backend/socials/tiktok/posts_scrapling/job_runner.py | Thread pipeline ingest mode through persistence + expose catalog upsert counters. |
| trr_backend/socials/tiktok/posts_scrapling/fetcher.py | Add Scrapling fetcher options + redaction-safe metadata wiring. |
| trr_backend/socials/threads/posts_scrapling/persistence.py | Add per-post DB savepoints to isolate failures during batch upserts. |
| trr_backend/socials/threads/posts_scrapling/fetcher.py | Add Scrapling fetcher options + runtime metadata. |
| trr_backend/socials/threads/posts_catalog/catalog.py | Classify empty “soft block” results as retryable with explicit error codes. |
| trr_backend/socials/socialblade/fetcher.py | Add Scrapling fetcher tuning/metadata and switch to browser proxy param. |
| trr_backend/socials/pipelines/comments/init.py | Export new Instagram comments catalog-stream helper. |
| trr_backend/socials/media_url_safety.py | Allow YouTube image host domain. |
| trr_backend/socials/instagram/posts_scrapling/fetcher.py | Add Scrapling fetcher tuning/metadata and centralize async_fetch kwargs. |
| trr_backend/socials/instagram/permalink_metadata.py | Improve post detail parsing with fetch__XDTMediaDict + excluded payload filtering. |
| trr_backend/socials/instagram/comments_scrapling/public_mode.py | Add explicit opt-in flag for budgeted public proxy while preserving public lane invariants. |
| trr_backend/socials/instagram/comments_scrapling/proxy.py | Enable budgeted sticky-per-egress Decodo proxy behavior for public lane behind flag. |
| trr_backend/socials/facebook/posts_catalog/catalog.py | Add soft-block empty-result classification for shared Facebook catalog scraping. |
| trr_backend/socials/api/handlers/profile_reads.py | Forward assignment_status to hashtags read path. |
| trr_backend/repositories/reddit_refresh.py | Harden Reddit media URL extraction/sanitization for mirroring. |
| api/routers/admin_socialblade.py | Add cookie health endpoint + batch preflight health check before Modal dispatch. |
| scripts/socials/instagram/enqueue_comments_audit_cursor_retries.py | Default to public_relay strategy and support date window args. |
| scripts/socials/instagram/bravo_straggler_recovery.py | Default to public_relay strategy and support date window args in delegated CLI. |
| scripts/modal/prepare_named_secrets.py | Add SocialBlade proxy defaults and new Modal concurrency defaults. |
| scripts/db/index_advisor_social_hot_paths.py | Surface nested advisor errors and mark warnings in report output. |
| requirements.in | Pin numpy/sympy/opencv versions to known-good set. |
| requirements.lock.txt | Refresh lockfile (notably numpy/opencv/sympy/mcp/fastapi/etc). |
| requirements.modal.lean.lock.txt | Refresh Modal lean lockfile versions. |
| requirements.modal.browser.lock.txt | Refresh Modal browser lockfile versions. |
| requirements.modal.vision.lock.txt | Refresh Modal vision lockfile versions. |
| .env.example | Document SocialBlade business API env vars. |
| docs/workspace/scrapling-social-jobs.md | New shared Scrapling lane practices/runbook. |
| docs/workspace/instagram-posts-scrapling.md | Link to shared Scrapling runbook. |
| docs/workspace/instagram-comments-scrapling.md | Link to shared Scrapling runbook. |
| docs/observability/modal-v439-v440-serve-backend-api-crash-loop-2026-05-28.md | Update deploy history stamp. |
| CONTEXT.md | Add TRR backend domain language glossary for social ingestion. |
| supabase/migrations/20260625172000_tiktok_post_comment_rollups.sql | Add TikTok comment rollup table + refresh trigger/functions. |
| supabase/migrations/20260629140000_instagram_comments_public_proxy_budget_ledger.sql | Codify public-comments proxy budget ledger table + RLS. |
| supabase/migrations/20260629143025_supabase_security_advisor_rpc_and_vector_hardening.sql | Harden function search_path/exec grants + vector extension schema. |
| supabase/migrations/20260629144346_socialblade_snapshot_fk_indexes.sql | Add FK indexes for SocialBlade snapshot tables. |
| supabase/migrations/20260629190000_canonical_social_hashtag_assignments.sql | Add canonical hashtag assignments + conflicts table with RLS. |
| supabase/migrations/20260702174000_hashtag_assignments_season_fk_index.sql | Add partial index for season_id FK on hashtag assignments. |
| supabase/migrations/20260702181000_youtube_post_comment_rollups.sql | Add YouTube comment rollup table + refresh trigger/functions. |
| supabase/migrations/20260702184229_social_rollup_function_and_rate_pace_security.sql | Lock down rollup function privileges + RLS for rate pace table. |
| supabase/migrations/20260702185000_drop_social_unused_index_wave1.sql | Drop selected unused social indexes (replay-focused migration). |
| tests/socials/twitter/test_twitter_posts_catalog.py | Add regression for explicit zero max_posts => unlimited pages. |
| tests/socials/tiktok/posts_scrapling/test_job_runner.py | Assert catalog upsert counters + pipeline ingest mode propagation. |
| tests/socials/threads/test_threads_posts_catalog.py | Assert Threads soft-block empty-result classification. |
| tests/socials/threads/posts_scrapling/test_persistence.py | Test per-post savepoints and post-error recovery behavior. |
| tests/socials/test_socialblade_fetcher.py | Update proxy expectations for SocialBlade fetcher. |
| tests/socials/test_socialblade_business_api.py | New tests for SocialBlade business API integration. |
| tests/socials/test_socialblade_auth.py | Add SocialBlade cookie health + loader behavior tests. |
| tests/socials/test_scrapling_transport.py | Add tests for fetcher options parsing and proxy metadata redaction. |
| tests/socials/test_repair_socialblade_auth.py | New tests for cookie-repair CLI blocked-validation contract. |
| tests/socials/test_profile_dashboard.py | Add catalog progress edge-case tests (cancelled/failed launch/deferred comments). |
| tests/socials/test_instagram_permalink_metadata.py | Add fixtures + tests for fetch__XDTMediaDict and excluded payloads. |
| tests/socials/instagram/test_instagram_direct_catalog.py | New tests for direct IG catalog backfill soft-block + DB error behavior. |
| tests/socials/instagram/comments_scrapling/test_public_relay_env_knobs.py | Add env knob tests (retry limits + rate scope hashing). |
| tests/socials/instagram/comments_scrapling/test_public_decodo_isolation.py | Add tests for public proxy opt-in and invariant enforcement. |
| tests/socials/instagram/comments_scrapling/test_public_blocked_pause.py | Update pause-threshold test expectations. |
| tests/socials/instagram/comments_scrapling/test_proxy.py | Add tests for public lane proxy flag behavior + per-egress fingerprinting. |
| tests/socials/instagram/comments_scrapling/test_proxy_budget.py | New proxy budget math tests. |
| tests/socials/instagram/comments_scrapling/test_media_direct_guard.py | New tests for run-budget kill switch + media-direct CDN tripwire. |
| tests/socials/instagram/comments_scrapling/test_job_runner_reply_only.py | Pin reply-only lane strategy to avoid public-first default. |
| tests/socials/instagram/comments_scrapling/test_job_runner_cancellation.py | Adjust stubs for new lifecycle timing + audit cursor resume load. |
| tests/socials/instagram/comments_scrapling/test_comments_lane_deploy_health.py | New import-time drift guard for Modal deploy image. |
| tests/socials/instagram/comments_scrapling/test_async_http_client.py | New tests for async client factory (httpx vs curl_cffi). |
| tests/socials/facebook/test_facebook_posts_catalog.py | Assert Facebook soft-block empty-result classification. |
| tests/scripts/test_prepare_named_secrets.py | Validate new secret defaults (SocialBlade + concurrency). |
| tests/scripts/test_enqueue_comments_audit_cursor_retries.py | Assert new args/defaults (public_relay + date window). |
| tests/scripts/test_classify_approval_blocked_comments.py | New tests for approval-blocked comment classification script. |
| tests/scripts/test_bravo_straggler_recovery.py | Assert delegated argv includes new strategy/date args. |
| tests/repositories/test_youtube_catalog_backfill_diagnostics.py | Assert config profile_snapshot precedence for YouTube catalog. |
| tests/repositories/test_social_run_lifecycle_repository.py | Harden deferred-comments followup lock/write/launch sequencing tests. |
| tests/repositories/test_social_account_catalog_backfill_integration.py | Update integration expectations for comments streaming vs deferred followup. |
| tests/repositories/test_reddit_refresh.py | Add tests for Reddit media URL sanitization/extraction. |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict.json | New fixture for IG XDT media dict payload. |
| tests/fixtures/instagram/scrapling/post_fetch_xdt_media_dict_null_fb.json | New fixture for null Facebook count edge-case. |
| tests/fixtures/instagram/scrapling/ig_direct_badge_count_off_msys.json | New fixture for excluded IG Direct badge payload. |
| tests/db/test_index_advisor_social_hot_paths.py | Assert report surfaces nested recommendation errors. |
| tests/db/test_advisor_remediation_sql.py | Add assertions for new security migrations. |
| tests/api/test_admin_socialblade.py | Add cookie-health preflight gating behavior tests. |
| tests/api/test_admin_social_comments.py | New API test for dry-run comments scrape preview. |
| tests/api/routers/test_socials_season_analytics.py | Add tests for new query params + dry-run followup/recovery endpoints. |
| .plan-work/plan-architect/instagram-scraper-improvement/VALIDATION.v3.md | Plan artifact: validation evidence v3. |
| .plan-work/plan-architect/instagram-scraper-improvement/VALIDATION.v2.md | Plan artifact: validation evidence v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/VALIDATION.md | Plan artifact: initial validation evidence. |
| .plan-work/plan-architect/instagram-scraper-improvement/TOOLS.md | Plan artifact: Tool Finder results summary. |
| .plan-work/plan-architect/instagram-scraper-improvement/SCORECARD.v3.md | Plan artifact: scorecard v3. |
| .plan-work/plan-architect/instagram-scraper-improvement/SCORECARD.v2.md | Plan artifact: scorecard v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/SCORECARD.md | Plan artifact: initial scorecard. |
| .plan-work/plan-architect/instagram-scraper-improvement/result.json | Plan artifact: run metadata/result bundle. |
| .plan-work/plan-architect/instagram-scraper-improvement/PATCHES.v3.md | Plan artifact: patch log v3. |
| .plan-work/plan-architect/instagram-scraper-improvement/LEDGER.v2.md | Plan artifact: issue ledger v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/LEDGER.md | Plan artifact: initial issue ledger. |
| .plan-work/plan-architect/instagram-scraper-improvement/INITIAL_PLAN.v2.md | Plan artifact: initial plan v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/INITIAL_PLAN.md | Plan artifact: initial plan. |
| .plan-work/plan-architect/instagram-scraper-improvement/IMPROVEMENTS.v2.md | Plan artifact: improvement log v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/IMPROVEMENTS.md | Plan artifact: improvement log. |
| .plan-work/plan-architect/instagram-scraper-improvement/HANDOFF.v3.md | Plan artifact: handoff v3. |
| .plan-work/plan-architect/instagram-scraper-improvement/HANDOFF.v2.md | Plan artifact: handoff v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/HANDOFF.md | Plan artifact: initial handoff. |
| .plan-work/plan-architect/instagram-scraper-improvement/AUDIT.v2.md | Plan artifact: audit v2. |
| .plan-work/plan-architect/instagram-scraper-improvement/AUDIT.md | Plan artifact: initial audit. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+247
to
+252
| @router.get("/socialblade/cookies/health") | ||
| async def get_socialblade_cookie_health( | ||
| validate: bool = Query(default=True), | ||
| handle: str | None = Query(default=None), | ||
| _admin: InternalAdminUser = None, | ||
| ) -> dict[str, Any]: |
Comment on lines
+25
to
+30
| create or replace function social.refresh_youtube_post_comment_rollup(target_video_id uuid) | ||
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+59
to
+64
| create or replace function social.refresh_youtube_post_comment_rollup_tg() | ||
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+107
to
+108
| grant execute on function social.refresh_youtube_post_comment_rollup(uuid) to service_role; | ||
| grant execute on function social.refresh_youtube_post_comment_rollup_tg() to service_role; |
Comment on lines
+25
to
+30
| create or replace function social.refresh_tiktok_post_comment_rollup(target_post_id uuid) | ||
| returns void | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+59
to
+64
| create or replace function social.refresh_tiktok_post_comment_rollup_tg() | ||
| returns trigger | ||
| language plpgsql | ||
| security definer | ||
| set search_path = social, public | ||
| as $$ |
Comment on lines
+107
to
+108
| grant execute on function social.refresh_tiktok_post_comment_rollup(uuid) to service_role; | ||
| grant execute on function social.refresh_tiktok_post_comment_rollup_tg() to service_role; |
Comment on lines
+9
to
+10
| -- Production apply must use DROP INDEX CONCURRENTLY outside a transaction. | ||
| -- This migration keeps plain DROP INDEX IF EXISTS for fresh database replays. |
Contributor
Codex Exhaustive Code ReviewFindings
Validation
|
Owner
Author
|
Superseded by cleaned replacement PR #152 after subagent review cleanup. |
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
git diff --check origin/main...HEAD.venv/bin/python -m pytest tests/api/test_admin_social_comments.py tests/scripts/test_classify_approval_blocked_comments.py tests/socials/instagram/comments_scrapling/test_async_http_client.py tests/socials/instagram/comments_scrapling/test_comments_lane_deploy_health.py tests/socials/instagram/comments_scrapling/test_media_direct_guard.py tests/socials/instagram/comments_scrapling/test_proxy_budget.py tests/socials/instagram/test_catalog_ingest_worker_b.py tests/socials/instagram/test_instagram_direct_catalog.py tests/socials/test_repair_socialblade_auth.py tests/socials/test_socialblade_business_api.py -qRuntime follow-through
trr-backend-jobsdeploy/readiness follow-through was completed before PR prep.