Skip to content

feat: add 2026_may folder and nppes data example#13

Open
asenetcky wants to merge 1 commit into
CTOpenData:mainfrom
asenetcky:main
Open

feat: add 2026_may folder and nppes data example#13
asenetcky wants to merge 1 commit into
CTOpenData:mainfrom
asenetcky:main

Conversation

@asenetcky

Copy link
Copy Markdown
Contributor

Title: Add May 2026 Example: Handling Large NPPES Data with DuckDB & Parquet

Summary
This PR adds a new module for the May 2026 R User Group session focusing on efficient ingestion and processing of large datasets. The example uses the National Plan and
Provider Enumeration System (NPPES) "Full Replacement File" (~11GB CSV) to demonstrate how to bypass memory limitations using DuckDB and Parquet.

Key Changes

  • New Folder: 2026_may/working-with-large-nppes-data/
  • Ingestion Script (ingest-nppes.R):
    • Automates scraping the CMS website for the latest monthly NPPES download link.
    • Downloads and extracts the data using httr2 and utils::unzip.
    • Leverages DuckDB to "stream" the massive CSV from disk, filtering for Connecticut (CT) providers without loading the full file into RAM.
    • Exports the filtered results to a compressed Parquet file for high-performance downstream analysis.
  • Documentation (README.md): Detailed walkthrough explaining the "Out of Memory" problem and why the DuckDB + Parquet architecture is a superior solution for
    large-scale CSV work.
  • Environment Management (renv.lock): Full lockfile to ensure reproducibility across different machines.

Why this is valuable
Health care data (like NPPES) is often too large for standard read_csv() calls on consumer-grade hardware. This contribution provides the group with a template for
"out-of-memory" data engineering in R, making these datasets accessible to more users.

How to Test

  1. Clone the branch and navigate to 2026_may/working-with-large-nppes-data/.
  2. Run renv::restore() to install dependencies.
  3. Source ingest-nppes.R.
  4. Verify that data/nppes/nppes_ct/ is created containing the filtered Parquet dataset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant