Summary
DataComPy is currently a Python library only — users must write a script to compare two datasets. A command-line interface would make it immediately accessible for ad-hoc data validation, CI pipelines, and use by non-Python practitioners without any boilerplate.
Motivation
CICD pipelines often run as shell tasks (Airflow BashOperator, GitHub Actions, GitLab CI) where Python entry points are awkward. A native CLI that exits non-zero on mismatch and emits machine-readable JSON is the highest-leverage single CICD improvement DataComPy could add.
Other common friction points:
- Ops / data engineers want to diff two CSV or Parquet files without opening a Python REPL
- Analysts want a one-liner to spot-check a file diff without writing a throw-away script every time
- Automation wants to shell out to
datacompy compare ... and capture a structured result
Proposed interface
# Basic comparison
datacompy compare \
--left s3://bucket/before.parquet \
--right s3://bucket/after.parquet \
--on id,date \
--backend pandas \
--max-unequal-rows 0 \
--json
# exit 0 on match, 1 on mismatch, 2 on error
# Local files, HTML report written to file
datacompy compare \
--left df1.csv \
--right df2.csv \
--on id \
--output report.html
# Tolerances
datacompy compare \
--left before.parquet \
--right after.parquet \
--on account_id \
--abs-tol 0.01 \
--rel-tol 0.001
# Ignore whitespace and case
datacompy compare \
--left df1.csv \
--right df2.csv \
--on id \
--ignore-spaces \
--ignore-case
# Control report verbosity
datacompy compare \
--left df1.csv \
--right df2.csv \
--on id \
--sample-count 20 \
--column-count 5
Summary
DataComPy is currently a Python library only — users must write a script to compare two datasets. A command-line interface would make it immediately accessible for ad-hoc data validation, CI pipelines, and use by non-Python practitioners without any boilerplate.
Motivation
CICD pipelines often run as shell tasks (Airflow
BashOperator, GitHub Actions, GitLab CI) where Python entry points are awkward. A native CLI that exits non-zero on mismatch and emits machine-readable JSON is the highest-leverage single CICD improvement DataComPy could add.Other common friction points:
datacompy compare ...and capture a structured resultProposed interface