NYC Taxi Data Engineering Platform

End-to-end data engineering platform built on official NYC TLC taxi trip data. The platform ingests official monthly parquet files, processes them through a PostgreSQL raw layer and a DuckDB analytical warehouse, and delivers three products: a monthly zone-level demand forecast, a real-time fare estimator, and a reviewer-facing analytics dashboard.

Cloud infrastructure is provisioned on GCP (GCS raw lake + partitioned and clustered BigQuery tables) using Terraform. Local deployment runs on Docker Compose. A Minikube + Terraform path deploys the same platform components to a local Kubernetes cluster.

Technology Stack

Core data platform

Python 3.11
uv for the repo-local Python environment
Docker Compose for the local platform baseline
PostgreSQL for canonical raw storage
MinIO for the local bronze raw lake
DuckDB for the analytical warehouse and local marts
dbt for warehouse transformations
Apache Airflow for orchestration and unattended monthly control
MLflow for experiment tracking and release artifacts

Delivered products

Streamlit for the reviewer dashboard
Flask for the fare API and fare web app serving layer
HTML/CSS/JS frontend for the fare product
Leaflet for the fare map experience

Validation and quality

GitHub Actions for CI on push and pull request
Great Expectations for the committed fare training dataset expectation suite
Python unittest for orchestration and reporting logic
dbt tests plus custom parity/coverage/contract checks

Infrastructure and deployment

Terraform for GCP cloud infrastructure and local Minikube deployment
Minikube and Helm for the validated local Kubernetes path
GCP: GCS raw lake + BigQuery analytical warehouse

Global Pipeline

flowchart TD
    TLC["Official NYC TLC\nmonthly parquet"]
    MINIO["MinIO\nbronze raw lake"]
    PG["PostgreSQL\nraw.yellow_tripdata / green_tripdata"]
    DUCK["DuckDB\nanalytical warehouse"]

    TLC --> MINIO --> PG --> DUCK

    DUCK --> DBT["dbt\nmarts & features"]
    DUCK --> FARE_BUILD["Fare release build\nlookup bundles + dataset"]

    DBT --> DASHBOARD["Reviewer Dashboard\nStreamlit · :8501"]
    DBT --> TRACKA["Track A\nmonthly forecasting"]
    TRACKA --> MLFLOW["MLflow\nexperiments & artifacts · :5000"]

    FARE_BUILD --> FARE_API["Fare API\n/predict-fare · :8090"]
    FARE_API --> WEBAPP["Fare estimator\n/fare-demo"]

    AIRFLOW["Airflow · :8080"] -.->|schedules| TRACKA

What The Platform Delivers

Track A — Monthly demand forecasting

A governed monthly pipeline that ingests the latest NYC TLC trip data, builds zone-level features in DuckDB, trains and evaluates candidate forecasting models, and publishes a monthly zone-level demand prediction. The pipeline runs automatically when new TLC data is available (controller mode) or on demand (manual mode). Model candidates, metrics, and release decisions are tracked in MLflow. See Track A Scheduler Modes for operating details.

Reviewer dashboard

A Streamlit analytics dashboard at http://localhost:8501 backed directly by the DuckDB warehouse. It shows city-level monthly trip and revenue trends by service type, and the top pickup zones by trips or revenue for any selected month. See Service URLs to open it.

Fare estimator

A browser-accessible fare estimator at http://localhost:8090/fare-demo. Users enter a pickup and dropoff using a text search, a zone picker, or by placing pins on an interactive NYC map. All inputs resolve to TLC zones before prediction. The result includes a fare estimate, a confidence range, and a data-support note. The model served is a champion selected by a benchmarked release cycle from historical trip data. See The Fare Product for the full input and workspace reference.

Service URLs

Surface	URL	Notes
Reviewer dashboard	`http://localhost:8501`	Warehouse-backed Streamlit dashboard
Fare web app	`http://localhost:8090/fare-demo`	User-facing fare estimator
MLflow	`http://localhost:5000`	No login in the default local stack
Airflow	`http://localhost:8080`	Default login: `admin` / `airflow`. Port configurable via `AIRFLOW_HOST_PORT` in `.env.airflow`
Fare API health	`http://localhost:8090/health`	Quick service check
Fare API ready	`http://localhost:8090/ready`	Confirms model + lookup bundle readiness

When connecting from another device on the same network, replace localhost with the LAN IP of the machine running the services (e.g. http://<HOST_IP>:8501).

Prerequisites

The following tools must be installed before running any make commands:

Tool	Purpose	Install
`uv`	Python environment and dependency management	https://docs.astral.sh/uv/getting-started/installation/
Docker + Docker Compose	All platform services run in containers	https://docs.docker.com/get-docker/
`make`	Command runner	Pre-installed on Linux/macOS; `choco install make` on Windows
Python 3.11	Required by `uv` for the virtual environment	https://www.python.org/downloads/

Optional (for cloud and Kubernetes paths):

Tool	Purpose
Terraform `>= 1.8.0`	GCP and Minikube infrastructure provisioning
`gcloud` CLI	GCP authentication (`gcloud auth application-default login`)
Minikube, kubectl, Helm	Local Kubernetes deployment path

Quick Start

Use this order on a fresh machine.

1. Install the Python environment

make install

2. Start infrastructure

make infra-up

Starts PostgreSQL, pgAdmin, and MinIO.

3. Start MLflow

make mlops-up

Open http://localhost:5000.

4. Start Airflow

make airflow-up

Open http://localhost:8080.

Wait until the airflow-api-server container is healthy before continuing — auth setup, scheduler start, and DAG discovery all happen during this window. You can poll automatically:

make airflow-ready

Or verify manually: docker ps should show airflow-api-server as healthy, and http://localhost:8080 should load the login page.

Login credentials: admin / airflow (seeded automatically by airflow-init).

5. Bootstrap Track A

The bootstrap downloads official TLC parquet files and runs the forecasting pipeline. The number of months controls the download volume.

Quick test — 3 months, yellow+green:

Both yellow and green are required — the dbt int_trips_unioned model unions both services.

make ml-tracka-bootstrap RUN_MONTH=2025-03 EVAL_START=2025-01 \
  BOOTSTRAP_START_MONTH=2025-01 BOOTSTRAP_SERVICES=yellow,green \
  OFFICIAL_INGEST_HISTORY_MONTHS=3 FORCE_EMPTY_FEATURES=1

Full historical — 23 months, both services:

make ml-tracka-bootstrap RUN_MONTH=2025-11 EVAL_START=2024-02 \
  BOOTSTRAP_START_MONTH=2024-01 BOOTSTRAP_SERVICES=yellow,green \
  OFFICIAL_INGEST_HISTORY_MONTHS=23 FORCE_EMPTY_FEATURES=1

Key parameters:

Parameter	What it controls
`RUN_MONTH`	The target evaluation month (must be the last month in the range)
`BOOTSTRAP_START_MONTH`	First month of the historical lineage window
`OFFICIAL_INGEST_HISTORY_MONTHS`	How many months of raw TLC data to download
`BOOTSTRAP_SERVICES`	`yellow,green` (both required for dbt models)
`EVAL_START`	Earliest month used when computing evaluation metrics

6. Build fare artifacts

Run after bootstrap completes. Use the same end month as your RUN_MONTH above.

Quick test:

make ml-fare-demo-release END_MONTH=2025-03

Full historical:

make ml-fare-demo-release END_MONTH=2025-11

Builds lookup bundles, fare dataset, benchmarks models, selects champion.

7. Start the fare web app

make ml-fare-api-up

Open http://localhost:8090/fare-demo.

8. Start the reviewer dashboard

make de-dashboard-up
make de-dashboard-smoke

Open http://localhost:8501.

9. Set the Track A scheduler mode

Development default (manual trigger only):

make ml-tracka-controller-disable

Unattended production mode (checks for new TLC months automatically):

make ml-tracka-controller-enable

Inspect current mode and DAG state:

make ml-tracka-scheduler-status

Infrastructure As Code

GCP (cloud)

Provisions the cloud target: GCS raw lake + BigQuery analytical warehouse with partitioned and clustered tables.

Resources provisioned:

GCS bucket (nyc-taxi-de-dev-raw) — raw data lake, versioned, lifecycle to Coldline after 90 days
BigQuery dataset nyc_taxi_raw — raw landed data
BigQuery dataset nyc_taxi_analytics — reviewer-facing marts
BigQuery table monthly_city_metrics — partitioned by MONTH, clustered by service_type
BigQuery table monthly_zone_metrics — partitioned by MONTH, clustered by service_type, zone_borough, zone_id
Service account with BigQuery editor + GCS object admin roles

Apply:

cd infrastructure/terraform
cp terraform.tfvars.example terraform.tfvars
# Set gcp_project_id to your GCP project
terraform init
terraform plan -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars

Or from the repo root:

make terraform-gcp-init
make terraform-gcp-plan
make terraform-gcp-apply

Live deployment (applied 2026-04-17):

GCP project      : dataeng-nyc-taxi
Region           : us-central1
GCS bucket       : nyc-taxi-de-dev-raw
BQ dataset (raw) : nyc_taxi_raw
BQ dataset (analytics) : nyc_taxi_analytics
BQ table         : nyc_taxi_analytics.monthly_city_metrics
                   — partitioned MONTH on year_month, clustered by service_type
BQ table         : nyc_taxi_analytics.monthly_zone_metrics
                   — partitioned MONTH on year_month, clustered by service_type, zone_borough, zone_id
Service account  : nyc-taxi-de-pipeline@dataeng-nyc-taxi.iam.gserviceaccount.com

Minikube (local Kubernetes)

Deploys the same platform components (Airflow, reviewer dashboard) to a local Kubernetes cluster using Helm and the Kubernetes + Helm Terraform providers.

What it deploys:

Kubernetes namespace de-zoomcamp-local
Airflow via the official Helm chart
Reviewer dashboard Deployment + Service

Apply:

cd infrastructure/terraform/minikube
cp terraform.tfvars.example terraform.tfvars
terraform init
terraform plan -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars

Or from the repo root:

make terraform-minikube-init
make terraform-minikube-plan
make terraform-minikube-apply

See infrastructure/terraform/minikube/README.md for prerequisites and the required repo mount command.

After apply, the services are reachable via host port-forwarding:

reviewer dashboard: http://<HOST_IP>:18501
Airflow UI: http://<HOST_IP>:18080

Track A Scheduler Modes

Track A supports two scheduler modes controlled by the tracka_scheduler_mode Airflow variable:

manual: the pipeline DAG is triggered explicitly by an operator
controller_owned: the controller DAG checks for newly published TLC months and triggers exactly one monthly run when needed

Switch to production mode:

make ml-tracka-controller-enable
make ml-tracka-scheduler-status

Return to manual mode:

make ml-tracka-controller-disable

Day-To-Day Commands

Start services

Task	Command
Infrastructure	`make infra-up`
MLflow	`make mlops-up`
Airflow	`make airflow-up`
Reviewer dashboard	`make de-dashboard-up`
Fare web app	`make ml-fare-api-up`

Stop services

Task	Command
Stop infrastructure	`make infra-down`
Stop MLflow	`make mlops-down`
Stop Airflow	`make airflow-down`
Stop reviewer dashboard	`make de-dashboard-down`
Stop fare web app	`make ml-fare-api-down`

Smoke checks

Surface	Command	Success signal
Reviewer dashboard	`make de-dashboard-smoke`	`HTTP/1.1 200 OK`
Fare API	`make ml-fare-deploy-smoke`	deploy smoke passes

Validation

Scope	Command
Fast CI-equivalent gate	`make ci-fast`
Great Expectations fare validation	`make ge-validate-fare-dataset`
Fare API smoke	`make ml-fare-api-smoke`
Fare API deploy smoke	`make ml-fare-deploy-smoke`

Pipeline Commands By Scenario

Initial setup

Quick test (3 months, yellow+green):

make infra-up
make mlops-up
make airflow-up
make airflow-ready
make ml-tracka-bootstrap RUN_MONTH=2025-03 EVAL_START=2025-01 \
  BOOTSTRAP_START_MONTH=2025-01 BOOTSTRAP_SERVICES=yellow,green \
  OFFICIAL_INGEST_HISTORY_MONTHS=3 FORCE_EMPTY_FEATURES=1
make ml-fare-demo-release END_MONTH=2025-03
make ml-fare-deploy-smoke

Full historical (23 months, both services):

make infra-up
make mlops-up
make airflow-up
make airflow-ready
make ml-tracka-bootstrap RUN_MONTH=2025-11 EVAL_START=2024-02 \
  BOOTSTRAP_START_MONTH=2024-01 BOOTSTRAP_SERVICES=yellow,green \
  OFFICIAL_INGEST_HISTORY_MONTHS=23 FORCE_EMPTY_FEATURES=1
make ml-fare-demo-release END_MONTH=2025-11
make ml-fare-deploy-smoke

Track A steady-state monthly operation

After bootstrap, enable the controller:

make ml-tracka-controller-enable
make ml-tracka-scheduler-status

The controller DAG runs on schedule. On each cycle it either records a no-op (no newer official month) or triggers exactly one bounded monthly run.

Check whether a new TLC month is available at any time:

make ml-tracka-unattended-check

Monthly Track A release

MLFLOW_TRACKING_URI=http://localhost:5000 \
make ml-tracka-monthly-release RUN_MONTH=2025-11 EVAL_START=2020-01 \
  REQUIRE_SIGNIFICANCE=1 P_VALUE_MAX=0.05 MISSING_SIGNIFICANCE_MODE=keep_baseline

Historical repair

make ml-tracka-historical-repair RUN_MONTH=2025-11 EVAL_START=2024-02 \
  REPAIR_START_MONTH=2024-01 REPAIR_END_MONTH=2024-06 REPAIR_SERVICES=yellow,green

Fare release only

make ml-fare-demo-release END_MONTH=2025-11

The Fare Product

The fare estimator has two workspaces:

Workspace	Purpose
`Estimate a ride`	Rider-facing fare estimation — address, zone, and map entry all resolve to TLC zones
`Analyst workbench`	Direct access to the released request contract — zone IDs, pickup datetime, trip distance, passenger count

Input methods in Estimate a ride:

Method	How it works
`Search places`	Search NYC addresses and pick one pickup and one dropoff match
`Pick zones`	Choose TLC pickup and dropoff zones directly
`Use map`	Pick locations from the bounded NYC TLC map

The fare prediction is zone-based underneath all rider methods. The result panel shows the estimated fare, a confidence level, trip metadata, and the data support behind the prediction.

This product is served from a monthly released ML bundle, not a fixed rule-based estimator. Each fare release rebuilds deterministic lookup bundles, rebuilds a canonical training dataset from historical NYC TLC trips, benchmarks candidate models, and publishes one approved champion together with the exact serving artifacts the API needs.

Data and features: the release uses historical TLC trip records, released TLC zone geometry, and monthly lookup bundles for route, route-hour, borough-pair, and context features so the web app, API, and training data all share the same zone and feature contract.
Model families: the benchmark compares a linear baseline with tree-based candidates, then refits the selected family on the full release dataset and publishes the champion model artifact used by the API.
Monthly championing: make ml-fare-demo-release rebuilds lookups, rebuilds the fare dataset, runs benchmark folds, applies a confidence-aware release policy, and only promotes a model that is both metric-competitive and safe to serve with the released lookup bundle.

Logs

Surface	Command
Infrastructure logs	`make infra-logs`
MLflow logs	`make mlops-logs`
Airflow logs	`make airflow-logs`
Reviewer dashboard logs	`make de-dashboard-logs`

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.github/workflows		.github/workflows
dags		dags
dashboard		dashboard
data		data
data_engineering		data_engineering
dbt		dbt
docs		docs
great_expectations		great_expectations
images		images
infrastructure		infrastructure
scripts		scripts
tests		tests
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation

NYC Taxi Data Engineering Platform

Technology Stack

Core data platform

Delivered products

Validation and quality

Infrastructure and deployment

Global Pipeline

What The Platform Delivers

Track A — Monthly demand forecasting

Reviewer dashboard

Fare estimator

Service URLs

Prerequisites

Quick Start

1. Install the Python environment

2. Start infrastructure

3. Start MLflow

4. Start Airflow

5. Bootstrap Track A

6. Build fare artifacts

7. Start the fare web app

8. Start the reviewer dashboard

9. Set the Track A scheduler mode

Infrastructure As Code

GCP (cloud)

Minikube (local Kubernetes)

Track A Scheduler Modes

Day-To-Day Commands

Start services

Stop services

Smoke checks

Validation

Pipeline Commands By Scenario

Initial setup

Track A steady-state monthly operation

Monthly Track A release

Historical repair

Fare release only

The Fare Product

Logs

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages