D A T A D E X

Collaborate on Open Data using Open Source Tools

Datadex is a platform where people collaborate on Open Data using modern and open source tools and frameworks.

💡 Principles

• Open: Play well with the ecosystem. Use open standards and share data in open formats.
• Modular: Each component can be replaced, extended, or removed. Works well in many environments (your laptop, in a cluster, or from the browser), and with multiple tools (thanks to the Arrow ecosystem).
• Permissionless. Don't ask, fork it and improve the models, add a new source or update any script.
• Data as Code. Declarative stateless transformations tracked in git. Version your data as code! Publish and share your reusable models for others to build on top.
• Modern: Supports types, tests, materialized views, and more.

Datadex isn't a new tool, it is an opinionated bridge between existing ones.

🚀 What can you do with Datadex?

• Model existing datasets using dbt and SQL like you would do in your company. You can use and abuse any of the other awesome dbt features like tests and docs (automatically generated and published on GitHub Pages).
• Explore your data with Rill or Jupyter Notebooks. Share your findings with others by publishing your notebooks online as beautiful static websites (thanks to Quarto).
• Add new sources via Dagster, dbt Python models, or dbt Plugins. Bring data locally and work with it!

⚙️ Setup

The fastest way to start using Datadex is via VSCode Remote Containers. Once inside the develpment environment, you'll only need to run make deps.

PS: The development environment can also run in your browser thanks to GitHub Codespaces!

🎯 Motivation

This small project was created after thinking how an Open Data Protocol could look like!

👏 Acknowledgements

• This proof of concept was created thanks to open source projects like DuckDB and dbt.
• Datadex name was inspired by Juan Benet awesome data projects.