CQL v5: DROP KEYSPACE times out — DataStax Java driver schema-agreement race after CREATE INDEX

[bkearns]

Problem

When running the DataStax Java driver v5 smoke test suite against Ferrosa, DROP KEYSPACE IF EXISTS times out. The driver sends the DROP, Ferrosa processes it and emits a SCHEMA_CHANGE event, but the driver never achieves schema agreement and eventually times out (even with a 30s timeout).

Result: 37/38 tests pass. Only DROP KEYSPACE IF EXISTS java_test fails.

Root Cause

After Ferrosa processes a CREATE INDEX DDL, it emits a SCHEMA_CHANGE event. The DataStax Java driver receives this event on its control connection and closes that connection as part of its schema-refresh lifecycle. It then opens short-lived probe connections (~0.5–1ms lifetime) to re-establish a control connection.

When Ferrosa processes the subsequent DROP KEYSPACE, it emits another SCHEMA_CHANGE event. However:

1. The old control connection is already closed — the event cannot be delivered to it.
2. New probe connections open, receive STARTUP → REGISTER → system.local + system.peers_v2 queries → close in ~0.5ms.
3. Ferrosa delivers the retained SCHEMA_CHANGE EVENT frame to these probe connections (confirmed via server logs — response_version=0x85 for v5), but the connections close before the driver processes the event.
4. The driver schema-agreement logic only processes SCHEMA_CHANGE events on its designated control connection, not on probe connections.
5. The driver never re-establishes a stable control connection that receives the event, so schema agreement never completes and the request times out.

What Was Tried

Approach	Result
Retained event watch channel (3s window)	37/38 — probe connections close before event is processed
Retained event watch channel (10s window)	37/38 — same issue
50ms delay before event emission	37/38
200ms delay before event emission	37/38
600ms delay before event emission	37/38
Inline event_rx.try_recv() prioritization before frame processing	37/38
Post-REGISTER inline watch delivery (EVENT sent after READY, before next frame)	37/38 — EVENT delivered to TCP socket but driver does not treat probe connections as control connection
v4/v5 map-format SCHEMA_CHANGE event body	32/38 — regressed other tests, reverted

Proposed Fix Directions

1. Schema-version polling fallback: Implement system.local.schema_version polling so the driver can detect schema agreement even when EVENT delivery is missed. Ferrosa already bumps schema_version on every DDL, but the driver may not be polling it correctly.

2. Prevent control connection closure after CREATE INDEX: Investigate why the DataStax driver closes its control connection after receiving the CREATE INDEX schema-change event. Ferrosa may be sending something that causes the driver to close the connection.

3. Direct EVENT write to TCP socket: Bypass the forwarder → event_tx → event_rx → framed.send() pipeline and write EVENT frames directly to the TCP socket from the forwarder task.

Environment

• Ferrosa branch: feat/cql-protocol-v3-v4-v5-conformance (PR feat(cql): accept and conform to CQL native protocol v5 #223)
• DataStax Java driver: 4.18.1
• Protocol: v5 (CQL native protocol v5)
• Ferrosa: single-node deployment (no peers)
• system.peers and system.peers_v2 return 0 rows (single node)

Test

The DROP KEYSPACE IF EXISTS java_test test is currently disabled in the Java smoke test suite (see PR #223) to unblock CI. It should be re-enabled once this is fixed.

References

• Ferrosa PR: feat(cql): accept and conform to CQL native protocol v5 #223
• Forge task: t_56f17a7e
• FMEA entry: CQL-11 in ferrosa-cql/specs/fmea.md
• Roadmap: ferrosa-cql/specs/roadmap.md (Next section)