Live, queryable datasets that update automatically.
Think of it like a spreadsheet that fills itself in — you describe the dataset you want (YC companies currently hiring, insurance quotes in your area, restaurants serving a specific brand), and BigSet builds it, keeps it fresh, and lets you query it with SQL.
Built on TinyFish APIs.
At the end of the day, the only thing that matters is data. Every decision, every agent, every product — it all comes down to having the right data at the right time.
So what if you could just… ask for it? Describe the dataset you want — in plain English — and have it built, structured, and kept fresh automatically. No scrapers to maintain. No pipelines to babysit. No waking up to broken cron jobs because some site changed a div.
You describe it. BigSet collects it. Your agents query it with SQL. It stays up to date on your schedule — every 30 minutes, every hour, whatever you need. And if something breaks, a healer agent patches it before you even notice.
Any dataset. Any source. Always fresh. That's the idea.
Prerequisites: Docker and Make
git clone https://github.com/tinyfish-io/bigset.git
cd bigset
make devThat's it. Postgres, backend, and frontend all spin up. Open localhost:3500.
| Layer | Tech |
|---|---|
| Frontend | Next.js 16, React 19, Tailwind 4 |
| Backend | Fastify, TypeScript |
| Auth | Better Auth (email/password, self-hosted) |
| Database | PostgreSQL via Drizzle ORM |
| Data Collection | TinyFish APIs (Search, Fetch, Browser) |
bigset/
├── frontend/ Next.js 16 — the UI
├── backend/ Fastify — API server, auth, database, cron jobs
├── docker-compose.dev.yml
└── Makefile
BigSet is a work in progress. We're building in the open because the best ideas come from the people who actually want to use the thing.
We'd love your feedback, ideas, or help building — come say hi:
- 🐦 Twitter: @Tiny_Fish for project updates
- 🗣 Twitter: @not_simantak for the unfiltered version
- 🐛 GitHub Issues: Report bugs or request features
Contributions are very welcome — whether it's code, feedback, or just telling us what datasets you'd want to build.
- Fork the repo
- Create a branch (
git checkout -b my-feature) - Make your changes
- Open a PR
If you're not sure where to start, open an issue or come say hi.