SqlTest.select flakes on Windows: ~12 000 exception throws inside the row loop cause ProcDump debugger overhead → connection timeout [API-2383]

[ihsandemir]

Symptom

SqlTest.select fails intermittently on Windows only. The CI log shows a flood of lines such as

[08:39:05]Exception: E06D7363.?AVindex_out_of_bounds@exception@client@hazelcast@@
[08:39:05]Exception: E06D7363.?AVillegal_argument@exception@client@hazelcast@@

followed by the Java server reporting Client cannot be reached for the SQL cursor.

Root cause

Windows C++ exceptions are SEH events

On Windows, every C++ throw is implemented as an SEH (Structured Exception Handling) exception with code 0xE06D7363 (the Windows magic constant for C++ exceptions, encoding "msc" in ASCII). The exception record also carries the mangled RTTI type name (?AVindex_out_of_bounds@...).

ProcDump attaches as a debugger

The CI Windows test step runs:

procdump.exe -accepteula -e -ma -w client_test.exe crash.dmp

The -w flag causes ProcDump to call DebugActiveProcess(), making it the Windows debugger of the test process. A Windows debugger receives all exception events via WaitForDebugEvent(), including first-chance notifications (exceptions that haven't been handled yet). ProcDump reads the SEH record, prints the [HH:MM:SS]Exception: E06D7363.?AV... line to stdout, then calls ContinueDebugEvent(DBG_EXCEPTION_NOT_HANDLED) so execution can continue. The C++ runtime's SEH filter then finds the catch block and handles the exception normally.

The cost of each throw under a debugger

Without a debugger, throw → catch is entirely in-process — zero kernel transitions.
With ProcDump attached, each throw requires:

1. The test process's threads are suspended by the kernel.
2. An EXCEPTION_DEBUG_EVENT is delivered to ProcDump across the kernel debug port (inter-process IPC).
3. ProcDump wakes from WaitForDebugEvent(), prints the line, calls ContinueDebugEvent().
4. The kernel resumes the test process.
5. The C++ runtime finds the catch handler.

That is two mandatory kernel-mode round-trips per throw.

Why it breaks this test

The select test iterates over 4 096 SQL rows. Inside the loop, three EXPECT_THROW calls verify that get_object throws on an invalid negative index, an out-of-range index, and an unknown column name:

for (auto itr = res->iterator(); itr.has_next();) {
    auto page = itr.next().get();
    for (const auto& row : page->rows()) {
        // ... value checks + 4096 map->get() RPC calls ...
        EXPECT_THROW(row.get_object<int>(-1), exception::index_out_of_bounds);
        EXPECT_THROW(row.get_object<int>(row.row_metadata().column_count()), exception::index_out_of_bounds);
        EXPECT_THROW(row.get_object<int>("unknown_field"), exception::illegal_argument);
    }
}

3 throws × 4 096 rows = 12 288 exception round-trips through ProcDump = 24 576 kernel transitions. Combined with the 4 096 synchronous map->get() server round-trips, the loop runs slow enough that the server-side Hazelcast connection health check fires and the SQL cursor is cancelled, causing Client cannot be reached and the test to fail.

The Client cannot be reached messages for other UUIDs in the log are background noise — the QueryStateRegistryUpdater cleaning up cursors from already-shutdown clients of earlier tests.

Fix

The three EXPECT_THROW calls verify fixed API behaviour that does not change row-to-row. They only need to run once, on a single representative row, before the main iteration loop. Fetching the first page ahead of the loop and running the checks there reduces the exception count from 12 288 to 3.

See the accompanying PR for the one-file change to hazelcast/test/src/sql_test.cpp.

[github-actions]

Internal Jira issue: API-2383