feat(deflation): add pd_deflate_pipeline() batch orchestrator - #32
Open
randrescastaneda wants to merge 5 commits into
Open
feat(deflation): add pd_deflate_pipeline() batch orchestrator#32randrescastaneda wants to merge 5 commits into
randrescastaneda wants to merge 5 commits into
Conversation
- Orchestrates pd_deflation() over the master inventory (simple Mode B), skips NA/erroring surveys, saves each to the dedicated pip_deflated stamp alias, and writes the updated master. - Adds a five-column deflation schema (deflated, content_hash_deflated, aux_cpi/ppp/pop_hash_at_deflation) initialised to NA by build_pip_inventory() for fresh and legacy rows. - Logs a deflate_summary_inf entry via log_add() with explicit args; classifies piperr conditions by their specific class with unknown_error fallback. - Registers the pip_deflated alias in pipdata_dlw_process() and Pipdata_script.R.
- Regenerates man pages for pd_deflate_pipeline() and build_deflation_summary(), and updates pd_deflation()/log_report() docs. - Vignette gains a pd_deflate_pipeline() section and updates the log_report() scope to include deflation entries.
- Adds 60 tests for deflate_one() and pd_deflate_pipeline() (happy path, NA guard, save failure, piperr/generic errors, force, early returns, catalog dedup). - Adds deflation-column schema tests to test-build_pip_inventory.R and build_deflation_summary() tests to test-log_report.R.
- Adds the 2026-08-17 plan (Steps 1-6, evidence V1-V7), brainstorm, and execution record for the deflate-pipeline-wrapper feature.
- Roadmap feature deflate-pipeline-wrapper set to done with plan link. - Active-state current.json repointed to the 2026-08-17 plan/execution report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Create
pd_deflate_pipeline(), a batch orchestrator that iterates over a master inventory of cleaned surveys, deflates each viapd_deflation()(simple Mode B), correctly detects and skips failures (includingNAreturns), saves results to a new"pip_deflated"stamp alias, updates the master inventory with deflation columns, and logs a structured summary. The function operates as an independent pipeline stage — not called bypd_process_data().Requirements addressed
pd_deflation()to each"pip_deflated"stamp aliaspipdata_logwith pinned keysn_total,n_success,n_failed,surveys_success,surveys_failedtryCatchwith skip-and-log; detectNAreturns frompd_deflation()as failuresinv = NULL) and caller-supplied inventory, normalizing a missingdeflatedcolumnpip_idonly, noversionhint) to delegate loading topd_deflation()deflated,content_hash_deflated,aux_*_hash_at_deflation) in the master inventoryget_aux_hashes(c("cpi", "ppp", "pop"))save_pip_data()return value so save failures are not counted as success