fix: provision-quality - set max plan strigh length

[tombrooks248]

What type of PR is this? (check all applicable)

• Refactor
• Feature
• Bug Fix
• Optimization
• Documentation Update

Description

Add --conf spark.sql.maxPlanStringLength=8192 to the provision-quality EMR Serverless job's spark-submit parameters.

Why

• The overnight PROD provision-quality job was failing with a driver out-of-memory, despite processing only a small amount of data (~17 min job).
• The driver logs showed the failure was a query-plan string* getting very very long, not a data-volume problem: ~99.6% of the 74 MB driver log was a single plan tree (all datasets unioned + broadcast-joined to their lookups), which Spark materialises into one giant string on the driver.
• Capping the plan-string length stops that oversized string from being built, fixing the out-of-memory at its cause — without needing to throw more memory at the driver.

*The query-plan string is just a human readable logging of the plan that pyspark is going to take as it completes its pyspark job, in this case the act of generating this string in an attempt to put it into the logs is, took up all the memory of the pyspark driver and resulted in the job failing.

Related Tickets & Documents

• Ticket Link
• Related Issue #
• Closes #

QA Instructions, Screenshots, Recordings

Done a full test in Staging

• Completed end-to-end: wrote all 3 CSVs, all 3 Delta tables (provision_quality, dataset_quality, organisation_quality — the failing PROD run died at the first Delta table), all 3 Postgres writes, then 'status': 'success', exit code 0.
• The log shrank: stderr went from 74 MB → 7.4 MB, and plan-tree lines from 305,993 → 882.

Added/updated tests?

We encourage you to keep the code coverage percentage at 80% and above. Please refer to the Digital Land Testing Guidance for more information.

• Yes
• No, and this is why: just configuring the logging, no tests needed.
• I need help with writing tests

[optional] Are there any post deployment tasks we need to perform?

[optional] Are there any dependencies on other PRs or Work?