AnyLake

Data engineering made simple — real-time or batch, no-code for business users, agent-native for the AI era. An open lakehouse that scales and runs anywhere: the anti-Databricks.

A lake is just a URI. Data is open Parquet in Apache Iceberg v2 tables. Compute is stateless DuckDB, embedded in-process. Identity plugs into your existing AD/LDAP or any cloud IdP, lineage into your existing OpenLineage catalog, transformations into your existing dbt project. There is no control plane, no cluster to babysit, no DBUs, and nothing proprietary to escape from.

flowchart LR
    people["Business users · engineers<br/>BI tools · AI agents"] --> runtime["AnyLake runtime<br/><i>stateless, runs anywhere</i>"]
    runtime --> format["Apache Iceberg v2<br/><i>Parquet + snapshots</i>"]
    format --> storage["S3 · GCS · Azure · MinIO<br/>plain disk · air-gapped"]

    classDef n fill:#ffffff,stroke:#9aa8b6,color:#161e2b
    classDef f fill:#e8f0fe,stroke:#1a73e8,color:#0c447c
    class people,runtime,storage n
    class format f

Loading

Architecture → · Integrations → · Security →

Quickstart

pip install -e .            # or: pip install -e '.[s3,iceberg,ldap,oidc]'

anylake init /data/lake                      # or s3://bucket/lake
anylake ingest /data/lake sales q1.csv
anylake sql /data/lake "SELECT city, sum(revenue) FROM sales GROUP BY city"
anylake sql /data/lake "SELECT count(*) FROM sales" --as-of 0   # time travel
anylake studio /data/lake                    # console + dashboards + Insights
anylake mcp /data/lake                       # expose the lake to AI agents

python3 demo.py                              # live demo: streams -> metrics -> agent

from anylake import Lake
from anylake.engine import Engine

lake = Lake.init("s3://bucket/lake")         # any fsspec URI
lake.table("sales").append(arrow_table)      # ACID commit
Engine(lake).sql("SELECT * FROM sales").arrow()
Engine(lake, as_of=3).sql("...")             # query the past

What it does

Streaming and batch are the same thing. A stream is a sequence of micro-batches, and every micro-batch is an ordinary ACID commit — one engine, one format, no Lambda architecture. Source offsets commit atomically with the data, so ingestion is effectively exactly-once and a crashed worker resumes from the lake itself. One YAML file; run it once for a batch job, loop it for a streaming pipeline. How it works →

lake: s3://bucket/lake
sources:
  - name: orders_feed        # kafka, file drop, api feed...
    table: raw_orders
transforms:
  - name: revenue_by_city
    sql: SELECT city, sum(revenue) AS revenue, count(*) AS orders
         FROM raw_orders GROUP BY city
    incremental:             # process only new rows per tick
      keys: city
      measures: {revenue: sum, orders: sum}
    expect:                  # bad data never becomes a commit
      on_fail: fail
      rules:
        - {column: revenue, not_null: true, min: 0}

No code needed. anylake studio serves a console for business users: browse tables, build metrics from dropdowns rendered as charts, time-travel to any past version, watch streams land live, export to CSV. Saved dashboards are built in — drag-resize panels persisted in the lake, so every teammate sees the same layout with no BI service to license.

Insights on screen. /insights renders a lineage DAG, every expectation result, and spend per table against the Databricks-equivalent price — all derived from commit metadata, so there's no metadata service to keep in sync and it works air-gapped.

Runs your existing stack. dbt projects run unchanged, lineage flows to Marquez/DataHub as OpenLineage, Grafana connects natively, and Spark/Trino/Snowflake read the same Iceberg tables. Integrations →

Agent-native. MCP tools give agents schemas, queries, time travel and writes; every agent write is a commit tagged with the actor, so the table history is the audit trail and mistakes revert with one time-travel query.

Why not Databricks

Cost. No cluster idling while you think. Compute is an in-process library — you pay for object storage and whatever box runs the query, which can be your laptop. The built-in cost meter prints the comparison, assumptions included.

Runs anywhere. s3://, gs://, az://, MinIO on-prem, or a directory on disk. Same code, same format, air-gapped or multi-cloud.

Simplicity. pip install, three commands, done. No workspaces, no cluster policies, no Unity Catalog tier to buy.

Open standards. Iceberg tables and Parquet that other engines read directly, OpenLineage for your catalog, dbt projects unchanged. Nothing to export, nothing to migrate off.

Tests

python3 tests/test_e2e.py          # local FS + real S3 API, time travel, commit races, auth/RBAC
python3 tests/test_streaming.py    # checkpoint resume, exactly-once, batch==stream pipelines
python3 tests/test_interfaces.py   # Studio no-code API, MCP agent tools + audit trail
python3 tests/test_hardening.py    # atomic S3 commits, schema enforcement, compact/vacuum, auth
python3 tests/test_iceberg.py      # Iceberg v2 ops, cross-engine interop, migration
python3 tests/test_mutations.py    # delete/update/MERGE, incremental == full recompute
python3 tests/test_auth_ui.py      # signup/login/sessions, SSO wiring, CSRF, token path
python3 tests/test_dashboards.py   # dashboard CRUD, panel SQL guard, ownership, layout
python3 tests/test_ecosystem.py    # OpenLineage, expectations, dbt projects, cost meter
python3 tests/test_insights.py     # lineage graph, quality report, spend view, routes

Documentation

Architecture	Layers, commit protocol, on-disk layout, table format, maintenance, roadmap
Integrations	dbt, OpenLineage, Grafana, MCP, expectations, cost meter
Security	Sign-in, SSO, LDAP/OIDC, RBAC, data protection, known limits