feat: add 2026_may folder and nppes data example

2026_may/working-with-large-nppes-data/README.md

@@ -0,0 +1,95 @@
+# NPPES Data Ingest: Handling Large CSVs in R
+
+## What is CMS and NPPES?
+
+The **Centers for Medicare & Medicaid Services (CMS)** is a federal agency within the United States Department of Health and Human Services. One of their key responsibilities is maintaining the **National Plan and Provider Enumeration System (NPPES)**.
+
+NPPES assigns unique identifiers, known as **National Provider Identifiers (NPIs)**, to health care providers and organizations. This database is the "gold standard" for identifying who is providing care in the U.S. health system. You can learn more about [CMS Data Dissemination here](https://www.cms.gov/medicare/regulations-guidance/administrative-simplification/data-dissemination).
+
+## The Problem: "Out of Memory" and API Limits
+
+For most beginner to intermediate R users, the `tidyverse` is the go-to for data manipulation. However, the `tidyverse` generally expects your data to fit into your computer's RAM. 
+
+> When dealing with an 11GB file on a machine with 8GB or 16GB of RAM, you will hit a wall.
+
+### Why not just use the API?
+CMS provides an [NPI Registry Search API](https://npiregistry.cms.hhs.gov/search), but it is **not feasible for bulk data work**. The API is designed for quick, individual lookups (e.g., "What is Dr. Smith's NPI?"). If you need to analyze every provider in a specific state or specialty, the API's rate limits and results-per-query caps make it impossible to use. 
+
+This leaves us with one option: downloading the massive "Full Replacement File".
+
+Even if you have enough RAM, the NPPES file is **sparse**—it has hundreds of columns, many of which are empty for most providers. Reading all that "nothing" is a waste of time and resources.
+
+## The Solution: DuckDB + Parquet
+
+This project sidesteps the memory limit by using two powerful tools:
+
+1.  **[DuckDB](https://duckdb.org/):** An analytical database that can "stream" data. Instead of loading the whole CSV into R, DuckDB reads it in chunks, filters for the data we actually want (e.g., just Connecticut providers), and writes it directly to a new file.
+2.  **[Parquet](https://parquet.apache.org/):** A columnar storage format. Unlike a CSV, Parquet is compressed and highly efficient. A multi-gigabyte CSV can often be reduced to a few hundred megabytes as a Parquet file, while keeping all the data types intact.
+
+## Getting Started
+
+### 1. Setup the Environment
+This project uses `renv` to manage R packages. This ensures you have the exact versions of `duckdb`, `httr2`, and other dependencies needed.
+
+```r
+# Open R and run:
+renv::restore()
+```
+
+```r
+# and select #1
+It looks like you've called renv::restore() in a project that hasn't been activated yet.
+How would you like to proceed? 
+
+1: Activate the project and use the project library.
+2: Do not activate the project and use the current library paths.
+3: Cancel and resolve the situation another way.
+
+```
+
+### 2. Run the Ingest Pipeline
+The main logic is contained in `ingest-nppes.R`. You can run the entire process (download, extract, and convert) by sourcing `ingest-nppes.R`.
+
+```r
+source("ingest-nppes.R", echo=FALSE)
+```
+
+## How It Works
+
+The script follows a "Research -> Strategy -> Execution" lifecycle:
+
+1.  **Scrape:** It visits the CMS website to find the latest monthly download link.
+2.  **Download:** It uses `httr2` to download the ~1GB zip file.
+3.  **Extract:** It unzips the file to locate the main `npidata_pfile_xxx.csv`.
+4.  **Stream (The Magic):** It uses a SQL query within DuckDB to read the CSV from disk and write it to Parquet. **R never actually "holds" the full dataset in memory.**
+
+```sql
+COPY (
+  SELECT * FROM read_csv('path/to/huge.csv')
+  WHERE State = 'CT'
+) TO 'output.parquet' (FORMAT parquet);
+```
+
+## Why this is better for you
+
+-   **Speed:** DuckDB is incredibly fast at reading CSVs.
+-   **Stability:** Your R session won't crash.
+-   **Efficiency:** The resulting `.parquet` file is much smaller and ready for high-performance analysis using the `arrow` or `duckdb` packages.
+
+## Next Steps: Analyzing the Data
+
+Once you have your `.parquet` file, you can analyze it without loading the whole thing:
+
+```r
+library(arrow)
+library(dplyr)
+
+nppes_data <- open_dataset("data/nppes/nppes_ct/")
+
+nppes_data |>
+  filter(`Provider Business Practice Location Address City Name` == "HARTFORD") |>
+  collect() # Only now does the filtered data enter your RAM
+```
+
+---
+*Data Source: [CMS NPPES Data Dissemination](https://download.cms.gov/nppes/NPI_Files.html)*

2026_may/working-with-large-nppes-data/ingest-nppes.R

@@ -0,0 +1,198 @@
+# Download the NPPES Data and stream CT to a parquet with duckdb
+
+# Constants ------------------------------------------------------------
+
+NPPES_LISTING_URL <- "https://download.cms.gov/nppes/NPI_Files.html"
+NOW <- lubridate::now()
+
+
+# resources ------------------------------------------------------------
+
+# rvest package doco: section 'Web scraping 101' https://rvest.tidyverse.org/articles/rvest.html#attributes-1
+
+# Download -------------------------------------------------------------
+
+#' Find the Monthly Download URL for NPPES
+#'
+#' @param listing_url Website for NPPES download.
+#'
+#' @returns A url string invisibly.
+fetch_monthly_url <- function(listing_url = NPPES_LISTING_URL) {
+  page <- rvest::read_html(listing_url)
+
+  url <-
+    page |>
+    rvest::html_elements("a") |> # anchors
+    rvest::html_attr("href") |> # urls
+    stringr::str_subset("NPPES_Data_Dissemination.*V2\\.zip") |> # look for the zipped download (should be a monthly and a weekly)
+    stringr::str_subset("Weekly", negate = TRUE) |> # ignore the weekly
+    head(1)
+
+  if (length(url) == 0) {
+    rlang::abort("Could not find monthly NPPES file URL.")
+  }
+
+  if (!stringr::str_starts(url, "http")) {
+    url <- glue::glue("https://download.cms.gov/nppes/{fs::path_file(url)}")
+  }
+
+  cli::cli_alert_success(glue::glue("Found: {url}"))
+  url
+}
+
+#' Download the the zip file at the NPPES url
+#'
+#' @param url A string of the url with the zip file to download.
+#' @param dest_dir A string of the destination folder for the download.
+#'
+#' @returns An invisible string of the path to the compressed NPPES download.
+download_nppes_zip <- function(url, dest_dir) {
+  zip_path <-
+    fs::path(dest_dir, fs::path_file(url))
+
+  cli::cli_alert_info("Downloading (~1 GB, this will take a while)...")
+
+  httr2::request(url) |>
+    httr2::req_progress() |>
+    httr2::req_perform(path = zip_path)
+
+  cli::cli_alert_success(glue::glue("Saved: {zip_path}"))
+  invisible(zip_path)
+}
+
+#' Extract the Compressed NPPES Data
+#'
+#' @param zip_path A string of the path to the compressed nppes data.
+#' @param dest_dir A string of the path for extracted data destination.
+#'
+#' @returns A stringr invisibly of the file path to the extracted NPPES data csv.
+extract_nppes_zip <- function(zip_path, dest_dir) {
+  cli::cli_alert_info("Extracting archive...")
+  utils::unzip(zip_path, exdir = dest_dir)
+
+  csv_path <-
+    fs::dir_ls(dest_dir, regexp = "npidata_pfile.*\\.csv$") |>
+    stringr::str_subset("FileHeader", negate = TRUE) |>
+    head(1)
+
+  if (length(csv_path) == 0) {
+    stop("Could not locate main NPI CSV after extraction.")
+  }
+
+  message(glue::glue("Located: {fs::path_file(csv_path)}"))
+  invisible(csv_path)
+}
+
+# Streaming to Parquet -------------------------------------------------
+
+#' Stream the NPPES State Data into a Parquet file
+#'
+#' @description
+#' At time of the writing the NPPES API is limited to quick lookups
+#' and the file download is the only wait to get all data for a location.
+#' However, the data is for the entire United States and it is a massive
+#' csv that cannot be loaded fully into memory for most standard DPH machines.
+#' `duckdb` is used to stream smaller portions of the csv into a parquet file
+#' with tight restrictions of how much memory it can use. This will take a
+#' little bit longer, but ensure most machines can run this process. The
+#' parquet artifact will be much smaller, and CT only and have a big boost
+#' to performance downstream.
+#'
+#'
+#' @param csv_path A string of the file path to the extracted NPPES data csv.
+#' @param dest_dir A string of the file path for the parquet data.
+#' @param state A string of the state abbreviation - defaults to 'CT' in `main()`.
+#'
+#' @returns An invisible string
+stream_csv_to_state_parquet <-
+  function(csv_path, dest_dir, state) {
+    state_dir <- fs::path(
+      dest_dir,
+      glue::glue("nppes_{stringr::str_to_lower(state)}")
+    )
+
+    fs::dir_create(state_dir)
+
+    out_path <-
+      fs::path(
+        state_dir,
+        glue::glue("nppes-{lubridate::today()}.parquet")
+      )
+
+    cli::cli_alert_info(glue::glue(
+      "Streaming via DuckDB → {fs::path_file(out_path)}..."
+    ))
+
+    con <- duckdb::dbConnect(duckdb::duckdb(
+      config = list(threads = '24', memory_limit = "4GB"),
+      dbdir = ":memory:"
+    ))
+
+    on.exit(duckdb::dbDisconnect(con, shutdown = TRUE))
+    sql <- glue::glue(
+      "
+    COPY (
+      SELECT *
+      FROM read_csv(
+        '{csv_path}',
+        header      = true,
+        all_varchar = true,
+        buffer_size = 1000000
+      )
+      WHERE \"Provider Business Practice Location Address State Name\" = '{state}'
+        OR \"Provider Business Mailing Address State Name\"           = '{state}'
+    )
+    TO '{out_path}' (FORMAT parquet)
+  "
+    )
+
+    DBI::dbExecute(con, sql)
+
+    n <-
+      DBI::dbGetQuery(
+        con,
+        glue::glue("SELECT COUNT(*) AS n FROM '{out_path}'")
+      ) |>
+      purrr::pluck('n')
+
+    cli::cli_alert_success(
+      glue::glue(
+        "{format(n, big.mark = ',')} providers written for {state}"
+      )
+    )
+    invisible(out_path)
+  }
+
+
+# Entry point ----------------------------------------------------------
+
+#' Entry Point into NPPES Data Ingestion
+main <-
+  function(
+    dest_dir = "data/nppes",
+    state = "CT",
+    clean_source = FALSE
+  ) {
+    cli::cli_alert_info("Starting pipeline {NOW}")
+    fs::dir_create(dest_dir, recurse = TRUE)
+
+    url <- fetch_monthly_url()
+    zip_path <- download_nppes_zip(url, dest_dir)
+    csv_path <- extract_nppes_zip(zip_path, dest_dir)
+
+    out_path <- stream_csv_to_state_parquet(csv_path, dest_dir, state)
+
+    if (clean_source) {
+      fs::file_delete(zip_path)
+      fs::file_delete(csv_path)
+      cli::cli_alert_success("Source files removed.")
+    }
+
+    FIN <- lubridate::now()
+    cli::cli_alert_success(
+      "Pipeline complete - Duration: {prettyunits::pretty_dt(FIN-NOW)}"
+    )
+    invisible(dest_dir)
+  }
+
+main()