How to deal with streaming data?

[davidgasquez]

Some datasets will need to be updated every few minutes or every hour.

1. Do we want to take care of those?
2. How can we?

[davidgasquez]

Real time datasets might be better suit for other projects. If we're speaking about public data (science, governments, climate, ...), updating datasets once per day seems reasonable.

In theory, we could try doing microbatching each 5 minutes or so for datasets that need that recency. E.g: a GitHub action that scrapes a website every 5 minutes.