Skip to content

Mykeb96/csv-automation

Repository files navigation

ClaimFlow

Workers' compensation claims automation demo — ingest messy employer CSV exports, standardize and enrich the data, export a clean dataset, and generate an executive summary report. Includes a React UI for live walkthroughs and a CLI for scripted runs.

Built as a portfolio project to demonstrate automation patterns relevant to employer claims processing (similar workflows are often implemented with VBA, Power Automate Desktop, or ETL tools in production).


Features

Capability Description
CSV ingestion Loads employer claim exports from a defined input path
Data cleaning Normalizes dates, currency, states, statuses, and text fields; removes invalid rows and duplicates
Calculated fields Adds days_to_report (report date minus injury date) for timeliness analysis
File export Writes output/clean_claims.csv with consistent formatting
Summary report Writes output/claims_summary_report.txt with financial, status, and operational metrics
Web demo UI Single-page app to run the pipeline, preview raw vs. clean data, and view the report
REST API FastAPI layer that executes the pipeline and returns JSON for the frontend

Architecture

flowchart LR
  subgraph input [Input]
    CSV["data/raw_claims_messy.csv"]
  end

  subgraph backend [Python backend]
    MAIN["main.py\npipeline"]
    API["api.py\nFastAPI"]
  end

  subgraph output [Output]
    CLEAN["output/clean_claims.csv"]
    RPT["output/claims_summary_report.txt"]
  end

  subgraph ui [Frontend]
    VITE["React + Vite\nClaimFlow UI"]
  end

  CSV --> MAIN
  MAIN --> CLEAN
  MAIN --> RPT
  VITE -->|"POST /api/pipeline/run"| API
  API --> MAIN
  API -->|"JSON previews + report"| VITE
Loading

Pipeline stages

  1. Load — Read raw CSV into pandas
  2. Clean — Strip whitespace, standardize enums, parse dates/currency, validate claim IDs, deduplicate
  3. Calculate — Derive days_to_report
  4. Export — Save normalized CSV
  5. Report — Generate text summary (totals, status breakdown, top employers/injuries, open-claim snapshot)

Tech stack

Layer Technologies
Data processing Python 3, pandas
API FastAPI, Uvicorn
Frontend React 19, TypeScript, Vite
Sample data Synthetic workers' comp-style claims (logistics, healthcare staffing, manufacturing, etc.)

Project structure

automation-project/
├── main.py                 # Core pipeline (load, clean, calculate, export, report)
├── api.py                  # HTTP API for the demo UI
├── requirements.txt        # Python dependencies
├── data/
│   └── raw_claims_messy.csv    # Sample messy input (~100 rows)
├── output/                 # Generated artifacts (gitignored recommended)
│   ├── clean_claims.csv
│   └── claims_summary_report.txt
└── frontend/
    ├── src/
    │   ├── App.tsx             # Landing page + pipeline overview
    │   ├── api/pipeline.ts     # API client
    │   └── components/         # Workspace, tables, report view
    ├── package.json
    └── vite.config.ts          # Proxies /api → localhost:8000

Prerequisites

  • Python 3.10+ with pip
  • Node.js 18+ and npm (for the UI only)

Getting started

1. Install Python dependencies

From the project root:

pip install -r requirements.txt

2. Run via CLI (no UI)

python main.py

Console output includes row counts and output paths.

Generated files:

File Description
output/clean_claims.csv Normalized claims (17 columns including days_to_report)
output/claims_summary_report.txt Executive summary report

Re-running the script overwrites previous outputs.

3. Run the full demo (UI + API)

Use two terminals.

Terminal 1 — API server (project root):

uvicorn api:app --reload --port 8000

Terminal 2 — Frontend (project root):

cd frontend
npm install
npm run dev

Open http://localhost:5173, go to Live workspace, and click Run pipeline.

The Vite dev server proxies /api requests to http://127.0.0.1:8000. Both processes must be running for the UI to work.

4. Production build (frontend only)

cd frontend
npm run build
npm run preview

The preview server still requires the API on port 8000 for pipeline execution unless you deploy both services together.


Deploy to production (Vercel + Render)

The app is split into two services:

Service Host What it runs
Frontend Vercel Vite/React static site (frontend/)
API Render FastAPI + pandas (api.py, main.py)

Local dev uses Vite’s proxy (/apilocalhost:8000). Production uses VITE_API_URL to call the hosted API.

Step 1 — Deploy the API on Render

  1. Push this repo to GitHub.

  2. On Render: New → Blueprint (uses render.yaml) or New → Web Service and connect the repo.

  3. Settings (if not using Blueprint):

    • Root directory: repository root (not frontend/)
    • Build command: pip install -r requirements.txt
    • Start command: uvicorn api:app --host 0.0.0.0 --port $PORT
  4. After deploy, copy the service URL (e.g. https://claimflow-api.onrender.com).

  5. Optional environment variables on Render:

    Variable Purpose
    ALLOWED_ORIGINS Extra origins for CORS (comma-separated), e.g. your Vercel URL
    ALLOWED_ORIGIN_REGEX Default https://.*\.vercel\.app — allows all Vercel deploy URLs
  6. Verify: open https://YOUR-API.onrender.com/api/health — should return {"status":"ok"}.

Free-tier Render services spin down when idle; the first request after sleep may take 30–60 seconds.

Step 2 — Deploy the frontend on Vercel

  1. Import the same GitHub repo in Vercel.

  2. Root Directory: frontend

  3. Framework preset Vite (or use frontend/vercel.json).

  4. Environment variables:

    Name Value
    VITE_API_URL Your Render API URL, no trailing slash (e.g. https://claimflow-api.onrender.com)
  5. Deploy. Open your Vercel URL → Live workspaceRun pipeline.

Step 3 — Confirm CORS

If the UI loads but Run pipeline fails with a network/CORS error:

  1. Add your exact Vercel URL to Render’s ALLOWED_ORIGINS (comma-separated).
  2. Redeploy the API (or wait for env var reload).

*.vercel.app is already allowed via ALLOWED_ORIGIN_REGEX in api.py / render.yaml.

Environment files (reference)

  • frontend/.env.exampleVITE_API_URL for local/production frontend
  • .env.exampleALLOWED_ORIGINS / ALLOWED_ORIGIN_REGEX for the API

API reference

Method Endpoint Description
GET /api/health Health check — returns {"status": "ok"}
POST /api/pipeline/run Runs the full pipeline on data/raw_claims_messy.csv

POST /api/pipeline/run response (abbreviated):

{
  "stats": {
    "rawRows": 103,
    "cleanRows": 101,
    "removedRows": 2,
    "columnCount": 17,
    "totalPaid": 958302.4,
    "totalPaidFormatted": "$958,302.40"
  },
  "rawPreview": { "columns": [...], "rows": [...] },
  "cleanPreview": { "columns": [...], "rows": [...] },
  "report": "WORKERS' COMPENSATION CLAIMS SUMMARY REPORT\n...",
  "outputs": {
    "csv": "output/clean_claims.csv",
    "report": "output/claims_summary_report.txt"
  }
}

Previews return up to 15 rows for display in the UI.


Data cleaning rules (summary)

The sample messy CSV intentionally includes common real-world issues. The cleaner addresses:

  • Whitespace and casing on text fields
  • Date formats — multiple formats parsed to consistent dates
  • Currency — strips $, commas; handles N/A, -, and invalid values
  • State codes — maps variants (ca, Texas, new york) to two-letter abbreviations
  • Claim status — normalizes values (open, pending review, etc.)
  • Validation — drops rows with invalid claim_id format (CLM-YYYY-NNN)
  • Deduplication — one row per claim_id (first occurrence kept)
  • Data quality — negative days_lost set to missing

Sample input schema

data/raw_claims_messy.csv columns:

claim_id, employee_id, employee_name, employer_name, injury_date, report_date, body_part, injury_type, claim_status, days_lost, medical_cost, indemnity_cost, total_paid, state, adjuster, notes


Design notes for production

This demo assumes a fixed CSV schema. In environments with many employer or carrier export formats, a normalization layer (configuration-based column mapping or AI-assisted schema detection) would sit before the stable cleaning and reporting steps. The Python pipeline is intended to remain the auditable, deterministic core after intake is standardized.


Troubleshooting

Issue Solution
UI shows "Pipeline error" / fetch failed (local) Start the API: uvicorn api:app --reload --port 8000
UI works on Vercel but Run pipeline fails Set VITE_API_URL to your Render API URL; check /api/health on Render
CORS error in browser console Add your Vercel URL to Render ALLOWED_ORIGINS; ensure regex covers *.vercel.app
Render API very slow first time Free tier cold start — wait and retry
FileNotFoundError for input CSV API must run from repo root; ensure data/raw_claims_messy.csv is in the deployment
Port 8000 in use (local) Use another port and update frontend/vite.config.ts proxy target

License

This project is provided for portfolio and interview demonstration purposes.

About

Workers’ comp claims automation demo — clean messy CSV exports and generate summary reports (Python, FastAPI, React).

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors