Speed UP DLB - Part 3 - replace 10 min wait witch check

[tombrooks248]

DLB: remove the 10-min wait-before-reporting sleep (replace with a datasette readiness check)

Type: Improvement

Background

build-digital-land-builder (prod only) contains a wait-before-reporting task that does time.sleep(600) between build-digital-land-builder and run-reporting-task. It exists to give datasette time to become consistent (pick up the freshly-built digital-land.sqlite3) before the reporting task queries it.

This is a fixed-time workaround, not a real readiness check: it adds 10 minutes of guaranteed dead time to every production run, regardless of whether datasette is actually ready sooner (usually it is) — or, in the worst case, still isn't ready after 10 minutes.

Code: dags/digital_land_builder.py (lines ~182–219)

def delay_execution(**kwargs):
    time.sleep(600)  # (10 minutes)

wait_before_reporting = PythonOperator(
    task_id="wait-before-reporting",
    python_callable=delay_execution,
    dag=dag,
)
...
build_digital_land_builder >> wait_before_reporting >> run_reporting_task

Goal

Eliminate the fixed 10-minute delay by gating run-reporting-task on an actual datasette-readiness signal, so the reporting task starts as soon as datasette has picked up the new database (typically well under 10 minutes) and doesn't proceed if it genuinely hasn't.

Scope / tasks

1. Identify what "datasette is ready / consistent" actually means here — i.e. what signal proves datasette is serving the freshly-built digital-land.sqlite3 (e.g. datasette has reloaded the DB / a health or version endpoint reflects the new build / the expected table or row count is queryable).
2. Replace delay_execution with a polling sensor/check that waits for that signal, with:
   • a sensible poll interval (e.g. 30s),
   • a timeout (cap at, say, the current 10 min or a bit more) so a genuinely-stuck datasette fails loudly instead of the reporting task running against stale data,
   • clear logging of how long it actually waited.
3. Wire it in place of wait-before-reporting (keep the build → wait → reporting ordering).
4. Validate in staging/dev, then confirm in prod that the reporting task starts after a real readiness signal and typical wait time drops well below 10 min.

Acceptance criteria

• No fixed time.sleep on the reporting path.
• run-reporting-task starts only once datasette is confirmed ready, and starts as soon as it is.
• If readiness isn't reached within the timeout, the task fails (or alerts) rather than running against stale data.
• Observed prod wait time recorded before/after.

Notes / risks

• This is prod-only (the wait_before_reporting / run_reporting_task block is inside if config["env"] == "production").
• Keep a timeout so removing the sleep doesn't turn an occasional slow-datasette into a silent stale-data reporting run — that's the one real risk the original sleep was guarding against.
• Airflow has built-in sensor patterns (PythonSensor / @task.sensor with poke_interval + timeout) that fit this cleanly.

[tombrooks248]

I have created some code which I think should work to fulfil this ticket, just running it in staging (this will take a couple of hours) and assuming all goes will I'll implement in PROD.

[tombrooks248]

ok so in the process of testing my new feature I have either discovered that the original 10 mins and then even my 20 mins worth of polling were both woefully inadequate, datasette takes many hours to do it or that my plan for checking the change does not work at all. Either way I think this is becomign too much work for what is going to be a 10 min saving and only for the reports, wont impact how soon data is in datasette. So I think better to fail fast than sing more time into this.

[tombrooks248]

Had a chat with Ben - I now have a much simpler version of the PR which I think is a good idea to implement but is a bit awkward to test, I feel like I am throwing good time after bad into this ticket when there are bigger fish to fry right now. So I am going to put this back into the backlog with a link to the latest version of the ticket. - the PR which is sort of 70% done but needs to tested in staging is here - digital-land/airflow-dags#88