BUG: add_data fails with Postgrest JSONB NaN is invalid - multiple organisation change

[pooleycodes]

An add_data request for the developer-agreement dataset fails when saving the response. The transform completes, but save_response_to_db → _save_response_details → session.commit() raises:

sqlalchemy.exc.DataError: (psycopg2.errors.InvalidTextRepresentation)
invalid input syntax for type json
DETAIL: Token "NaN" is invalid.
CONTEXT: JSON data, line 1: ...national-park-authority:Q72617160", "entity": NaN...
[SQL: INSERT INTO response_details (response_id, detail) ...]

A Python float('nan') reaches a JSONB column. json.dumps emits a bare NaN (it allows non-finite floats by default); Postgres JSONB follows strict JSON and rejects it, so the whole batch insert fails.

Reproduction

• Request: type: add_data, collection: developer-contributions, dataset: developer-agreement, New Forest NPA source (Developer-Agreement-2023.xlsx), resolved organisations = ['national-park-authority:Q72617158'].
• Fails with digital-land-python main (0.1.1.dev210+g9240853a7).
• Passes with digital-land-python branch 109-bug-refresh-async-files-if-spec-has-updated.

Confirmed during debugging

• It takes the single-organisation path: organisations has length 1, so EntityLookupPhase uses super().process(), not the new multi-org fan-out.
• Entity lookup works — rows resolve normally (e.g. reference 09/93881-da → entity 4601954).
• The issue log is clean — after IssueLog.add_severity_column, self.rows contains no NaN (checked with [r for r in self.rows if any(v!=v for v in r.values())] → []). So pipeline-issues is not the source, despite the error text mentioning entity.
• Therefore the NaN is elsewhere in the response detail (converted_row / transformed_row / a pipeline-summary breakdown). Exact location not yet pinned

Leading hypothesis (not yet proven)

A change on digital-land-python main that is absent from branch 109-... introduced this. Strongest suspects, both touching entity/organisation resolution.

Suspected interaction: a row that has an organisation value in a column plus a resource default organisation ends up producing a non-finite/blank entity that is serialised as a float NaN rather than "" / null.

Two separate problems

1. Robustness (the crash): an unresolved/blank entity must never serialise as NaN. Coerce non-finite floats → None before the JSONB write — either in digital-land-python where the value is produced, or defensively at the async-backend boundary in _save_response_details. This turns a hard FAILED into a gracefully-reported row.
2. Behavioural (the trigger): identify the exact main change producing the NaN for the org-column-plus-default case, and fix the resolution so the value is valid.

[pooleycodes]

Okay so PR #577 introduced a change that breaks response-detail saving:

• check.py now logs duplicate_reference_check issues with entity as an int. This tips the issue-log entity column from object dtype to float64.

In IssueLog.add_severity_column, the issue rows go through a pandas round-trip (pd.DataFrame(rows).to_dict()). Because the column is now float64, pandas converts those None entities to float('nan'). NaN is not valid JSON, so when the async-request-backend writes the response detail to its JSONB column, Postgres rejects it (Token "NaN" is invalid) and the request fails.

Neither change crashes alone — before #577 the entity column was pure object (strings + None), so None serialised cleanly and this pandas coercion never fired.

Fix

In digital_land/log.py, add_severity_column now coerces NaN → None as the records leave pandas, restoring the correct null value so JSON/JSONB serialisation succeeds.