W2-1: sync/index config — ingested sortAt + model3dId, per-image Post fan-out - #326
Merged
Conversation
… fan-out Wires the scheduled-publish sync/index config to the W1-1 (#325) fan_out_per_row + shared_ops_fields trigger types. sync-config-civitai.yaml: - Post trigger: type fan_out -> fan_out_per_row (rows_from Image/postId), RETAINING track_fields {publishedAt, availability, postedToId} [PR-M1]. - Image trigger: + sortAtUnix track_field via shared_ops_fields (ms, `* 1000`, COALESCE(GREATEST...) belt — byte-identical to W1-1 SORTATUNIX_EXPR); + model3dId computed_field (INSERT-read Post subselect, NOT on the Post fan-out — set at post-create, INSERT-read suffices; documented inline). - Dump images phase: copy_query gains a type-correct `sortAt` epoch-SECONDS column [PR-B3] (all-timestamp COALESCE/GREATEST, extract epoch ::bigint outside); posts enrichment + model3dId column + field mapping. civitai-index.yaml: - sortAt sort field: drop `computed:` block ([PR-M5], same-deploy as emission flip); + model3dId filter field (single_value, per_value_lazy) + data_schema entry; verified sortAtUnix->sortAt mapping (ms_to_seconds + fallback:sortAt). - [PR-B4] pinned unit statement in both configs. Tests (src/pg_sync/sync_config.rs, load the REAL deploy configs): - test_post_fanout_disjoint_from_sortatunix: Post∩Image emitted fields == the intentional Mode-B set {publishedAt, availability, postedToId}; sortAtUnix and model3dId strictly disjoint (op_dedup safety). - test_image_sortatunix_emitted_ms_on_deploy_config: ms units + shared fn call. - test_index_config_sortat_ingested_and_model3did: no computed, mapping + model3dId. Also: delete stale deploy/configs/prod-sync-config-civitai.yaml (behind the canonical sync-config-civitai.yaml since the 2026-07-07 Mode B fix; prod-sync.toml mounts sync-config-civitai.yaml, no code/operational reference to the stale copy). Regenerated docs/design/trigger-sql-review.sql. cargo test --lib --features pg-sync: 1249 passed, 8 ignored (test_min_tracked_value_after_expansion skipped — known deadlock). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JustMaier
force-pushed
the
feat/sync-config-sortat
branch
from
July 17, 2026 01:18
d567bd9 to
7bff7be
Compare
This was referenced Jul 17, 2026
JustMaier
added a commit
that referenced
this pull request
Jul 17, 2026
…ates PASS) (#329) * test(w3): scheduled-publish + ingested-sortAt E2E + full-scale proof harness Pins the W3 critical cases (scheduled-publish-design §2.D, #325+#326): - tests/e2e/e2e-scheduled-publish-sortat.mjs — self-contained (no PG) Node E2E driving /ops with the exact #326 trigger payloads: ingested sortAtUnix(ms)->sortAt(s), scheduled->deferred, the CRITICAL no-op sweep activation at Tf (ingested future sortAt preserved through activation), unpublish/republish. 14/14 PASS. - tests/w3/ — full-scale reproducible harness (prep/dedup/dump-driver/assert, local-PG schema + model-share sortAt triggers, minimal poller, lifecycle, latency, re-emitter) + README with the gate evidence (all gates PASS: 114.8M dump 23m13s no-regression, sortAt invariant 600/600, model3dId 108,567, lifecycle incl. no-op sweep, heal-path, latency, PR-B3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(w3): post-#328 amendments — gate1 valid vs pure-GREATEST, stale-column self-heal, widened [PR-M2] - README: gate 1 remains valid without re-dump (prep computes pure GREATEST, never reads Image.sortAt — identical to #328's amended copy_query). Emptied-originals + stage_dir isolation warnings. - w3-stale-column.sql: proves the widened BEFORE trigger heals a stale sortAt column — an nsfwLevel-only update emits the CORRECT recomputed sortAtUnix (verified: stale 2020 -> correct). - w3-m2-latency.sql: widened-trigger per-update overhead +47us P50 / +390us P99 (sub-ms). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
JustMaier
added a commit
that referenced
this pull request
Jul 17, 2026
…ortAt, model3dId (#332) * docs: scheduled-publish wrap-up — per-image fan-out, ingested sortAt, model3dId (#13) Documents the shipped scheduled-publish effort (PRs #325/#326/#328/#329/#330) across the doc set: - pg-sync-v2-final.md: Post fan-out is now per-image materialized (type: fan_out_per_row) not queryOpSet; shared PG functions bitdex_post_fanout_ops / bitdex_image_sortat_ops; sortAt ingested (computed: dropped), dump never trusts the column; ModelVersion/Model stay queryOpSet. - CLAUDE.md: Sync V2 section reflects new fan-out shape + ingested sortAt + model3dId. - scheduled-publish-design.md: §1 history corrected (2024 migration triggers, Post-side neutered by name-collision, new_image_sort_at absent from prod pg_trigger per zuri's live verification); §2.D marked EXECUTED with the no-backfill decision and W1-4 dominant-class (visible-but-misordered) finding. - execution-plan: W4 step 2 no-backfill correction note. - FOLLOWUP.md: 7 residuals (over-cap fan-out skip, flaky flush tests, retired combinedNsfwLevel, frankenstein docs, MV cascade cost, stage_dir isolation, E2E population gap). - HANDOFF.md: scheduled-publish design chain pointer + W4 rollout state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(reemitter): correct Tf-activation model to sweep-driven shadow flip Prod evidence (sortat-divergence-specimens-2026-07-16.md PR-m5 + execution plan corrected [PR-B2]) established that scheduled slots are ALIVE with isPublished=false (quarantine branch), NOT deferred-map-resident, so Tf activation is an overdue-sweep shadow flip, not activate_due. The re-emitter is therefore a co-driver of activation, not a pure safety net. W3 confirmed the no-op case (sweep alone flipped isPublished + preserved ingested sortAt, 14/14 E2E). Fixes fact #1 + the §7 'does not drive activation' bullet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
Stacked on #325 (
feat/per-image-fanout) — this PR targetsfeat/per-image-fanout, notmain. It depends on #325'sfan_out_per_row/shared_ops_fieldstrigger_gen types and will need to be re-targeted atmain(or merged after) once #325 lands. Do not merge before #325.Implements W2-1 (#8) of
docs/design/scheduled-publish-execution-plan.md.Summary
deploy/configs/sync-config-civitai.yaml:type: fan_out→fan_out_per_row(rows_from: {table: Image, fk: postId}), retainingtrack_fields: {publishedAt, availability, postedToId}([PR-M1] — publishedAt drives both deferred-alive activation stamping and the isPublished shadow; only Phase 2 ever removes availability/postedToId).+ sortAtUnixviashared_ops_fields(milliseconds,* 1000,COALESCE(GREATEST(...))belt — byte-identical to W1-1'sSORTATUNIX_EXPR),+ model3dIdcomputed_field (INSERT-read Post subselect; not carried on the Post fan-out — model3dId is set at post-create before images exist, so INSERT-read alone suffices; reasoning documented inline in the config).imagesphase:copy_querygains a type-correctsortAtepoch-seconds column ([PR-B3] — all GREATEST/COALESCE operands are TIMESTAMPs,extract(epoch from ...)::bigintwraps the whole expression, neverCOALESCE(timestamp, bigint)); posts enrichment gains themodel3dIdcolumn + field mapping.deploy/configs/civitai-index.yaml:sortAtsort field: removed thecomputed: {op: greatest, source_fields: [existedAt, publishedAt]}block ([PR-M5] — same-deploy as the emission flip; computed and ingested must never coexist).+ model3dIdfilter field (single_value,per_value_lazy) +data_schemaentry.sortAtUnix → sortAtdata_schemamapping intact (ms_to_seconds: true,fallback: sortAt).deploy/configs/prod-sync-config-civitai.yaml— behind the canonicalsync-config-civitai.yamlsince the 2026-07-07 Mode B fix;prod-sync.tomlmountssync-config-civitai.yaml, and grep found no code/operational reference to the stale copy.docs/design/trigger-sql-review.sql(test-generated artifact).[PR-B4] Units verification (must-verify before cutover)
Pinned statement (also written into both config files as comments):
Evidence (file:line):
src/executor.rs:668-690(Gt/Gte bucket-snap gated onfield == tb.field_name()),src/cache_worker.rs:575(bucket membership derived fromtb.sort_field_name() = "sortAt").src/time_buckets.rs:189-196(cutoff/now arithmetic in seconds), threshold normalized atsrc/query.rs:316(ts_secs_u64).sortAtsort field (today: engine-computed viaGREATEST), thesortAtUnixname is only the clause label matched againsttb.field_name(). WhethersortAtUnixis ingested is irrelevant to the query path.sortAtseconds still feed the layers (now ingested instead of computed), seconds-Gteremains correct.civitai-index.yamltime_bucketsblock (filter_field: sortAtUnix,sort_field: sortAt) +data_schemasortAtUnix → sortAtentry.Caveat (pre-existing, unchanged by this PR): a non-snappable
sortAtUnix Gte(duration outside the 10% tolerance of every configured bucket) falls through torange_scan("sortAtUnix"), which returnsFieldNotFoundsince no such filter field is ingested. In practice the Civitai builder only sends period thresholds aligned to the 24h/7d/30d/1y buckets, so this hasn't surfaced — flagging for W3/W4 awareness, not fixing here (out of W2-1 scope).Tests
src/pg_sync/sync_config.rs, all loading the real deploy configs:test_post_fanout_disjoint_from_sortatunix— Post-fan-out ∩ Image-trigger emitted fields == exactly the intentional Mode-B overlap{publishedAt, availability, postedToId};sortAtUnix/model3dIdstrictly disjoint (the actual op_dedup LIFO hazard named in [PR-m2]).test_image_sortatunix_emitted_ms_on_deploy_config— ms units + shared-function call present on the real config.test_index_config_sortat_ingested_and_model3did— nocomputed:block, mapping semantics, model3dId wiring.cargo test --lib --features pg-sync(skippingtest_min_tracked_value_after_expansion, known deadlock): 1249 passed, 8 ignored, 0 failed.Notes for reviewers
model3dIdas aPostcolumn confirmed by the coordinator againstprisma/schema.prismain model-share (model3d-visible-ids-batch.test.ts:6).🤖 Generated with Claude Code