Refactor dlw and aux load

.cg-docs/BRAIN-01.md

@@ -1,10 +1,10 @@
 # 🧠 Project Brain — Part 1
 
-_Generated 2026-08-04_
+_Generated 2026-08-07_
 
-## Reporting_Level / Aaa.R / Pd_Deflation.R
+## Reporting_Level / Aaa.R / Valid_Dlw_Load\(\)
 
-_Keywords: `reporting_level` · `aaa.r` · `pd_deflation.r`_ · 75 entities
+_Keywords: `reporting_level` · `aaa.r` · `valid_dlw_load()`_ · 83 entities
 
 - **[Migrate dplyr calls to collapse/data.table \(Phase 1: 3 files\)](.cg-docs/brainstorms/2026-04-27-dplyr-to-collapse-dt.md)** · `brainstorm` · _decided_ · `2026-04-27`
   > dplyr, tidyr, and tibble are used via `::` in 4 pipdata files but are **not declared in DESCRIPTION Imports** — phant…
@@ -18,6 +18,10 @@ _Keywords: `reporting_level` · `aaa.r` · `pd_deflation.r`_ · 75 entities
   > `update_pip_inventory()` is ~350 lines handling 8+ responsibilities: version extraction from in-memory `proc_dta`, sk…
 - **[Inventory assembler: delta/update strategy](.cg-docs/brainstorms/2026-05-27-inventory-delta-strategy.md)** · `brainstorm` · _decided_ · `2026-05-27`
   > The `build_pip_inventory()` function (implemented 2026-05-22) used a "rebuild-from-scratch" strategy: query entire ca…
+- **[valid_dlw_load review and inv_to_process rewrite](.cg-docs/brainstorms/2026-06-05-valid-dlw-load-review.md)** · `brainstorm` · _decided_ · `2026-06-05`
+  > Review of `valid_dlw_load()` and its helpers (`filter_aux_inv`, `inv_to_process`) to ensure the function accomplishes…
+- **[Gate aux-change detection in valid_dlw_load on aux version change](.cg-docs/brainstorms/2026-08-05-aux-version-gate-valid-dlw-load.md)** · `brainstorm` · _draft_ · `2026-08-05`
+  > `valid_dlw_load()` always calls `valid_aux_load()`, which invokes `pipaux::compare_aux_releases()` and `pipaux::compa…
 - **[Enrich log report with success metrics and aux changes](.cg-docs/plans/2026-04-06-enrich-log-report.md)** · `plan` · _completed_ · `2026-04-06`
   > Extend the pipeline logging and `log_report()` so the markdown report includes: (a) the number of surveys successfull…
 - **[Remove redundant get_wrk_release\(\) guards from DLW call chains](.cg-docs/plans/2026-04-23-remove-redundant-get-wrk-release.md)** · `plan` · _completed_ · `2026-04-23`
@@ -54,12 +58,18 @@ _Keywords: `reporting_level` · `aaa.r` · `pd_deflation.r`_ · 75 entities
   > Remove spurious `reporting_level.x` and `reporting_level.y` columns from the PIP master inventory. These are ALL-NA a…
 - **[Refactor inventory architecture: catalog-based assembler](.cg-docs/plans/2026-05-20-inventory-architecture-refactor.md)** · `plan` · _completed_ · `2026-05-20`
   > Replace the 350-line `update_pip_inventory()` + `format_vrs()` with a catalog-based assembler that reads version fact…
+- **[Refactor valid_dlw_load: logging, abort, and content-hash comparison](.cg-docs/plans/2026-06-17-valid-dlw-load-refactor.md)** · `plan` · _completed_ · `2026-06-17`
+  > Implement the 5 decisions from the 2026-06-05 brainstorm:
 - **[save_pip_data: memory-resilient save order and GC guard](.cg-docs/plans/2026-06-18-save-pip-memory-resilience.md)** · `plan` · _done_ · `2026-06-18`
   > Prevent `cannot allocate buffer` failures in `save_pip_data()` when serializing large survey artifacts (e.g. `ARG_201…
 - **[pd_process_data: OOM crash fix for 4000+ survey runs](.cg-docs/plans/2026-06-24-pd-process-data-oom-crash.md)** · `plan` · _completed_ · `2026-06-24`
   > Step 1 (explicit `rm()` + `gc()` in `process_data()`'s success branch) and Step 3 (roxygen `@details`) were implement…
 - **[pd_process_data: RAM regression fix \(spec-once + per-survey gc\)](.cg-docs/plans/2026-06-28-pd-process-data-ram-regression.md)** · `plan` · _completed_ · `2026-06-28`
   > Running `pd_process_data()` on the current `var_tabmaker2` branch makes RAM explode, whereas commit `3e00987a` did no…
+- **[2026-08-05-aux-version-gate-valid-dlw-load](.cg-docs/plans/2026-08-05-aux-version-gate-valid-dlw-load.md)** · `plan` · _—_ · `—`
+  > --- date: 2026-08-05 title: "Gate aux-change detection in valid_dlw_load on per-survey aux content-hash comparison" s…
+- **[Aux version gate for valid_dlw_load — revised run-level hash design](.cg-docs/plans/2026-08-06-aux-version-gate-valid-dlw-load-revised.md)** · `plan` · _completed_ · `2026-08-06`
+  > Prevent unnecessary survey re-cleaning by comparing the current content hash of each requested auxiliary artifact wit…
 - **[2026-04-06-enrich-log-report-review](.cg-docs/reviews/2026-04-06-enrich-log-report-review.md)** · `review` · _—_ · `—`
   > **Review depth**: standard **Plan**: `.cg-docs/plans/2026-04-06-enrich-log-report.md` **Files reviewed**: 5 **Finding…
 - **[2026-04-16-review](.cg-docs/reviews/2026-04-16-review.md)** · `review` · _—_ · `—`
@@ -100,6 +110,10 @@ _Keywords: `reporting_level` · `aaa.r` · `pd_deflation.r`_ · 75 entities
   > **Review depth**: standard **Files reviewed**: 3 (`R/update_pip_inventory.R`, `tests/testthat/test-update_pip_invento…
 - **[2026-05-20-inventory-architecture-refactor-review](.cg-docs/reviews/2026-05-20-inventory-architecture-refactor-review.md)** · `review` · _—_ · `2026-05-27`
   > **Review depth**: standard **Files reviewed**: 6 (`R/build_pip_inventory.R`, `R/pd_process_data.R`, `R/pd_deflation.R…
+- **[2026-08-06-aux-version-gate-valid-dlw-load-revised-review](.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-revised-review.md)** · `review` · _—_ · `2026-08-07`
+  > **Review mode**: standard **Files reviewed**: 33 changed paths from `origin/refactor_dlw_load..HEAD` **Findings**: 17…
+- **[2026-08-06-aux-version-gate-valid-dlw-load-revised-verify-review](.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-revised-verify-review.md)** · `review` · _—_ · `2026-08-07`
+  > **Review mode**: mode:verify / light **Parent review**: `.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-…
 - **[Internal logmeta type markers polluting build_type_summary\(\) table](.cg-docs/solutions/bugs/2026-04-07-internal-logmeta-types-polluting-type-summary.md)** · `solution` · _—_ · `2026-04-07`
   > `log_report()` renders a "Summary by Type" table via `build_type_summary()`. The table was supposed to show only genu…
 - **[Conditional column initialization produces inconsistent data.table schema](.cg-docs/solutions/bugs/2026-04-29-conditional-column-init-inconsistent-schema.md)** · `solution` · _—_ · `2026-04-29`
@@ -144,6 +158,8 @@ _Keywords: `reporting_level` · `aaa.r` · `pd_deflation.r`_ · 75 entities
   > After replacing `update_pip_inventory()` with `build_pip_inventory()`, `reporting_level` (and other stale columns) co…
 - **[joyn diagnostic columns in production data paths — reportvar and suffix discipline](.cg-docs/solutions/data-quality/2026-06-05-joyn-diagnostic-column-discipline.md)** · `solution` · _—_ · `2026-06-05`
   > Two distinct classes of bug in this codebase were caused by the same underlying pattern: `joyn::` join diagnostic col…
+- **[Gate auxiliary re-cleaning with run-level content hashes and row-level changes](.cg-docs/solutions/data-quality/2026-08-07-aux-content-hash-gated-recleaning.md)** · `solution` · _—_ · `2026-08-07`
+  > `valid_dlw_load()` always invoked `valid_aux_load()`, which compares current auxiliary releases/vintages and returns …
 - **[Unified package environment: merging .pipdata / .pipdataenv / .logenv into one](.cg-docs/solutions/environment-issues/2026-04-30-unified-package-environment-accessor-pattern.md)** · `solution` · _—_ · `2026-04-30`
   > `pipdata` had three separate package-level environments: | Environment | Purpose | Defined in | |-------------|------…
 - **[Guard-at-entry-point pattern for release validation in pipdata](.cg-docs/solutions/git-workflows/2026-04-23-guard-at-entry-point-pattern.md)** · `solution` · _—_ · `2026-04-23`

.cg-docs/BRAIN-log.md

@@ -1,6 +1,6 @@
 # 🧠 Project Brain — Chronological Log
 
-_Generated 2026-08-04 · 103 artifacts (newest first) + 0 roadmap features_
+_Generated 2026-08-07 · 109 artifacts (newest first) + 0 roadmap features_
 
 ## undated
 
@@ -36,6 +36,27 @@ _Generated 2026-08-04 · 103 artifacts (newest first) + 0 roadmap features_
   > **Review depth**: standard **Files reviewed**: 3 (`R/update_pip_inventory.R`, `tests/testthat/test-update_pip_invento…
 - **[2026-06-18-yaml-recode-dictionary](.cg-docs/plans/2026-06-18-yaml-recode-dictionary.md)** · `plan` · _done_ · `—`
   > The current variable recoding logic in `pd_dlw_clean.R` is hardcoded across four functions (`recode_edu()`, `recode_g…
+- **[2026-08-05-aux-version-gate-valid-dlw-load](.cg-docs/plans/2026-08-05-aux-version-gate-valid-dlw-load.md)** · `plan` · _—_ · `—`
+  > --- date: 2026-08-05 title: "Gate aux-change detection in valid_dlw_load on per-survey aux content-hash comparison" s…
+
+## 2026-08-07
+
+- **[2026-08-06-aux-version-gate-valid-dlw-load-revised-review](.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-revised-review.md)** · `review` · _—_ · `2026-08-07`
+  > **Review mode**: standard **Files reviewed**: 33 changed paths from `origin/refactor_dlw_load..HEAD` **Findings**: 17…
+- **[2026-08-06-aux-version-gate-valid-dlw-load-revised-verify-review](.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-revised-verify-review.md)** · `review` · _—_ · `2026-08-07`
+  > **Review mode**: mode:verify / light **Parent review**: `.cg-docs/reviews/2026-08-06-aux-version-gate-valid-dlw-load-…
+- **[Gate auxiliary re-cleaning with run-level content hashes and row-level changes](.cg-docs/solutions/data-quality/2026-08-07-aux-content-hash-gated-recleaning.md)** · `solution` · _—_ · `2026-08-07`
+  > `valid_dlw_load()` always invoked `valid_aux_load()`, which compares current auxiliary releases/vintages and returns …
+
+## 2026-08-06
+
+- **[Aux version gate for valid_dlw_load — revised run-level hash design](.cg-docs/plans/2026-08-06-aux-version-gate-valid-dlw-load-revised.md)** · `plan` · _completed_ · `2026-08-06`
+  > Prevent unnecessary survey re-cleaning by comparing the current content hash of each requested auxiliary artifact wit…
+
+## 2026-08-05
+
+- **[Gate aux-change detection in valid_dlw_load on aux version change](.cg-docs/brainstorms/2026-08-05-aux-version-gate-valid-dlw-load.md)** · `brainstorm` · _draft_ · `2026-08-05`
+  > `valid_dlw_load()` always calls `valid_aux_load()`, which invokes `pipaux::compare_aux_releases()` and `pipaux::compa…
 
 ## 2026-08-04
 
@@ -73,7 +94,7 @@ _Generated 2026-08-04 · 103 artifacts (newest first) + 0 roadmap features_
 
 ## 2026-06-17
 
-- **[Refactor valid_dlw_load: logging, abort, and content-hash comparison](.cg-docs/plans/2026-06-17-valid-dlw-load-refactor.md)** · `plan` · _active_ · `2026-06-17`
+- **[Refactor valid_dlw_load: logging, abort, and content-hash comparison](.cg-docs/plans/2026-06-17-valid-dlw-load-refactor.md)** · `plan` · _completed_ · `2026-06-17`
   > Implement the 5 decisions from the 2026-06-05 brainstorm:
 
 ## 2026-06-05

.cg-docs/BRAIN.md

@@ -1,6 +1,6 @@
 # 🧠 Project Brain
 
-_Generated 2026-08-04 · 103 entities · 2 topics · 27 edges_
+_Generated 2026-08-07 · 109 entities · 2 topics · 32 edges_
 
 ## How to Use
 
@@ -13,26 +13,26 @@ _Generated 2026-08-04 · 103 entities · 2 topics · 27 edges_
 
 | # | Topic | Entities | File |
 |---|-------|----------|------|
-| 1 | [Reporting_Level / Aaa.R / Pd_Deflation.R](BRAIN-01.md#reporting-level-aaa-r-pd-deflation-r) | 75 | BRAIN-01.md |
+| 1 | [Reporting_Level / Aaa.R / Valid_Dlw_Load\(\)](BRAIN-01.md#reporting-level-aaa-r-valid-dlw-load) | 83 | BRAIN-01.md |
 | 2 | [Old_Files/ / Archive / Devtools::Document\(\)](BRAIN-01.md#old-files-archive-devtools-document) | 4 | BRAIN-01.md |
 
 ## Entity Summary
 
 | Type | Count |
 |------|-------|
-| brainstorm | 8 |
-| plan | 33 |
-| review | 20 |
-| solution | 41 |
+| brainstorm | 9 |
+| plan | 35 |
+| review | 22 |
+| solution | 42 |
 | strategy | 1 |
 
-_24 entities not assigned to any topic_
+_22 entities not assigned to any topic_
 
 ## Relationship Summary
 
 | Edge Type | Count |
 |-----------|-------|
-| decided_from | 8 |
-| references | 3 |
-| reviews | 14 |
-| verifies | 2 |
+| decided_from | 9 |
+| references | 4 |
+| reviews | 16 |
+| verifies | 3 |

.cg-docs/active-state/current.json

@@ -1,30 +1,40 @@
 {
-  "plan": ".cg-docs/plans/2026-08-03-pipeline-wrapper-vignettes.md",
-  "executionReport": ".cg-docs/work-reports/2026-08-03-pipeline-wrapper-vignettes.md",
-  "currentPhase": 2,
+  "plan": ".cg-docs/plans/2026-08-06-aux-version-gate-valid-dlw-load-revised.md",
+  "executionReport": ".cg-docs/work-reports/2026-08-06-aux-version-gate-valid-dlw-load-revised.md",
+  "currentPhase": null,
   "completedPhases": [1, 2],
-  "lastUpdated": "2026-08-04",
+  "lastUpdated": "2026-08-07",
   "refs": {
-    "roadmapFeature": "new-orchestration-script",
+    "roadmapFeature": "aux-version-gate-valid-dlw-load",
     "roadmapMilestone": "Pipeline Alignment Audit"
   },
   "decisions": [
-    "Deflation is confirmed NOT integrated into pd_process_data(); vignettes must describe it as a standalone post-hoc step.",
-    "docs/pipeline_overview.qmd has two known stalenesses (purrr::map vs lapply; 'deflated' output claim) - do not copy verbatim.",
-    "No old .png diagrams (pipeline_flow.png, pd_functions.png) are reused in any rewritten/new vignette.",
-    "2026-08-04 mid-execution deviation (user-confirmed, plan amended first per deviation-policy 'ask'): added R6 - new Validating-Data.Rmd vignette covering pipdata_dlw_process() internals (pipdata_get_gmd, pipdata_validate_gmd); PIP-data-pipeline.Rmd cross-reference updated to link both companion articles; Processing-Data.Rmd scope unchanged.",
-    "2026-08-04 Step 3 complete: Processing-Data.Rmd fully rewritten around pd_process_data/deflation/log_report, knitr::knit() verified by user, all referenced functions confirmed exported. _pkgdown.yml updated (Step 5) with articles: and reference: indices; pkgdown not in DESCRIPTION Suggests so DESCRIPTION left untouched (C2) - V3 syntax/build verification still pending user run.",
-    "2026-08-04 Step 6 (final validation pass) complete: devtools::document() run by user, regenerated pd_process_data.Rd and log_report.Rd confirmed to exactly match prior manual patches. devtools::check() run by user - only 3 pre-existing NOTEs (future file timestamps; wbpip:::md_clean_data unexported import; object.size undefined global in save_pip_data), none attributable to this change (no touched file involved). Plan complete."
+    "Roadmap feature aux-version-gate-valid-dlw-load updated to status active and plan path repointed to the revised 2026-08-06 plan.",
+    "Phase 1 complete (Steps 1-2): get_aux_hashes() resolver added in R/utils.R; wired into pd_process_data() before aux loading; build_pip_inventory() gains aux_hashes param and writes aux_<measure>_hash columns on current-run rows, initialised to NA on assembled master for legacy/retained rows.",
+    "Phase 2 complete (Step 3): valid_dlw_load() rewritten with two-stage aux gating. Master loaded once (shared dt_master) and passed to inv_to_process() (now accepts dt_master) and aux_hash_candidates(). Stage 1 builds candidates from per-survey aux hash comparison over full filtered/latest inventory; missing historical hash is a candidate; conflicting same-survey/content hashes abort. Stage 2 runs valid_aux_load() only for changed measures and intersects affected surveys with candidates. force=TRUE skips master/aux comparisons and processes all rows.",
+    "Tests: test-get-aux-hashes.R (7), test-build_pip_inventory.R (3 aux-hash), test-valid_dlw_load.R (9 new gating + updated existing), test-valid_aux_load.R (updated verbose propagation). Full suite PASS (2 pre-existing skips). roxygenise regenerated .Rd files.",
+    "Plan marked status: completed, completed-date: 2026-08-07. Roadmap feature aux-version-gate-valid-dlw-load set to done.",
+    "Verification review 2026-08-07: full R suite PASS (0 failures, 2 pre-existing skips). Both light verification agents returned no usable output due to model-service 'Response contained no choices' errors; not retried per protocol."
   ],
   "evidenceStatus": {
-    "V1": "passed (all three vignettes: PIP-data-pipeline.Rmd, Validating-Data.Rmd, Processing-Data.Rmd; knitr::knit() pandoc-free verification)",
-    "V2": "passed (all three vignettes; no stale/archived references, no deflated mischaracterization)",
-    "V3": "passed (_pkgdown.yml articles:/reference: index added; devtools::document()/check() ran cleanly against it, no pkgdown-related errors)",
-    "V4": "passed (devtools::check() - only 3 pre-existing NOTEs, none attributable to this change)",
-    "V5": "passed (no pipeline_flow.png reference in PIP-data-pipeline.Rmd)",
-    "V6": "passed (no pd_functions.png reference in Processing-Data.Rmd - grep-confirmed 2026-08-04)",
-    "V7": "passed (Validating-Data.Rmd renders; all referenced functions exist and exported)",
-    "V8": "passed (PIP-data-pipeline.Rmd cross-reference links both Validating-Data.html and Processing-Data.html)"
+    "V1": "passed (resolver tests: all six measures incl. PFW)",
+    "V2": "passed (missing/ambiguous/empty/failure/missing-hash abort tests)",
+    "V3": "passed (pd_process_data resolves hashes once before aux loading)",
+    "V4": "passed (build_pip_inventory persists hashes on current-run rows)",
+    "V5": "passed (legacy/retained rows keep NA; schema initialised)",
+    "V6": "passed (master inventory loaded exactly once)",
+    "V7": "passed (changed measures only invoke valid_aux_load)",
+    "V8": "passed (COL/ARG vs USA/GER intersection test)",
+    "V9": "passed (conflicting same-survey hashes abort)",
+    "V10": "passed (new/DLW-changed surveys remain selected)",
+    "V11": "passed (force mode skips master/aux comparisons)",
+    "V12": "passed (no .joyn column, no duplicate survey IDs)",
+    "V13": "passed (targeted valid_dlw_load/valid_aux_load tests)",
+    "V14": "passed (full suite, 2 pre-existing skips)",
+    "V15": "passed (roxygen regenerated; context.md unchanged - no canonical logging semantics changed)",
+    "verification": "full suite passed; agent verification incomplete due to model-service errors"
   },
-  "nextCommand": "none - plan complete"
+  "nextCommand": "/cg-review mode:verify (rerun light agents if independent reviewer output is required)"
 }
+
+