Problem
The nightly job that builds the listed-building dataset has been failing. It runs
on a temporary machine with a fixed 200GB of disk, and as the data has grown the
build now uses up all 200GB and crashes with a "disk is full" error. 200GB is the
maximum for this type of machine, so we can't just give it more.
Fix
During the build, the data gets written to disk twice — once as intermediate
working files, then again in the final database format. The fix deletes those
intermediate files as soon as they've been copied into the final database, freeing
up space for the rest of the build to finish.
Note
This lowers disk usage but doesn't raise the 200GB ceiling — as the data grows it
will eventually hit the limit again. The longer-term fix is the planned PySpark
work.
Problem
The nightly job that builds the listed-building dataset has been failing. It runs
on a temporary machine with a fixed 200GB of disk, and as the data has grown the
build now uses up all 200GB and crashes with a "disk is full" error. 200GB is the
maximum for this type of machine, so we can't just give it more.
Fix
During the build, the data gets written to disk twice — once as intermediate
working files, then again in the final database format. The fix deletes those
intermediate files as soon as they've been copied into the final database, freeing
up space for the rest of the build to finish.
Note
This lowers disk usage but doesn't raise the 200GB ceiling — as the data grows it
will eventually hit the limit again. The longer-term fix is the planned PySpark
work.