Nightly multi-DC Jepsen: setup-time schema-agreement failure misreported as an invariant violation

[bkearns]

Root-cause tracking for the recurring nightly tier-multi-dc failure (auto-filed instances land as separate issues, e.g. #301). Forge task t_bec3e5d6.

1. It is a setup failure, not an invariant violation

The run dies before any workload executes:

CREATE KEYSPACE IF NOT EXISTS jepsen WITH replication = {'class':'SimpleStrategy','replication_factor':3}
  -> Failed to await schema agreement:
     Host with id 0 required for schema agreement is not present in connection pool

The orchestrator then prints 1 of 1 combination(s) violated an invariant — its generic failure wording. Bank conservation was never tested. This misreporting is the most damaging part: a flaky infra failure is announced as a correctness regression.

2. Long-standing and intermittent — not caused by any recent merge

Nightly	Signature
2026-07-25	Failed to await schema agreement: Host with id 0 …
2026-07-21	same
2026-07-20	same
2026-07-18	same
2026-07-17	(different — an actual invariant violation)

Green on 07-19, 07-22, 07-23, 07-24. So it flaps, and this signature pre-dates the 2026-07-24/25 merges by roughly a week.

3. "Host with id 0" is a driver message bug — a red herring

scylla-1.6.0/src/errors.rs:

#[error("Host with id {} required for schema agreement is not present in connection pool", 0)]
RequiredHostAbsent(Uuid),

The format argument is a bare 0 — the integer literal — where thiserror needs .0 for the tuple field. The message therefore always prints "id 0" regardless of the actual UUID. There is no nil host_id in ferrosa; the real host's identity is being swallowed, which is what makes this look like metadata corruption.

Suspected cause

A node required for schema agreement is not in the driver's connection pool. The T3 compose hands the driver all six nodes (29042-29044 dc1, 29142-29144 dc2); the intermittency fits a startup race where the first DDL is issued before every node is pooled.

Proposed fixes

1. Gate the first DDL on all expected hosts being pooled (or on schema agreement) instead of racing it.
2. Distinguish setup failure from invariant violation in the orchestrator summary and in the auto-filed issue title/body — highest value, cheapest fix, and it stops these being read as correctness regressions.
3. Report the format-arg bug upstream to scylla-rust-driver (0 → .0).
4. Log connection-pool contents on schema-agreement failure so the absent host is identifiable despite the driver bug.