sumstats: intermittent truncated stdout with RC=0 on Windows

[Shourov1]

Summary

On Windows, sumstats.exe intermittently exits with return code 0 and empty stderr, but its stdout is truncated to roughly 4 KB and is missing the trailing Summary of results from N records footer. Because the exit code and stderr give no indication of failure, downstream automation silently consumes incomplete summary statistics.

Environment

• WFDB version: 10.7.1 (Windows binary, as reported by the version string embedded in wfdb-10.7.dll). Note: there is no matching 10.7.1 tag in bemoody/wfdb; the latest tagged source release is 10.7.0. The binary was obtained from the PhysioNet Windows distribution.
• OS: Windows 11 (x64)
• Invocation: subprocess.Popen([sumstats, table.txt], stdout=<file>, stderr=DEVNULL) from Python 3, but also reproducible from cmd.exe with plain redirection: sumstats.exe table.txt > out.txt
• Frequency: roughly 1 in 5–10 invocations on the same input; non-deterministic
• Input tables: standard bxb/rxr/epicmp line-per-record output (e.g. beat-by-beat_comparison_table.txt, run-by-run_comparison_table.txt, VF_detection_table.txt, afib_detection_table.txt)

Observed behaviour

• Process exits with RC = 0
• stderr is empty
• stdout is cut off mid-table, typically near the 4 KB boundary
• The expected closing block is missing:

  Summary of results from N records
  ...
  

Expected behaviour

Either:

1. Complete output is written to stdout on every successful invocation, or
2. The process exits with non-zero status / writes a diagnostic to stderr when output is truncated.

Reproduction

Minimal loop (PowerShell):

for ($i=0; $i -lt 50; $i++) {
    & .\sumstats.exe .\beat-by-beat_comparison_table.txt > "out_$i.txt"
    if (-not (Select-String -Path "out_$i.txt" -Pattern 'Summary of results' -Quiet)) {
        Write-Host "Truncated on iteration $i (size: $((Get-Item out_$i.txt).Length) bytes)"
    }
}

A representative input table can be provided on request.

Workaround

We retry up to 5 times and validate completeness by checking for the Summary of results footer in the captured stdout before accepting the result. A warm-up invocation (output discarded) is run first, which appears to reduce — but not eliminate — the failure rate.

max_attempts = 5
for attempt in range(1, max_attempts + 1):
    # Warm-up run (output discarded)
    subprocess.Popen(command, stdout=subprocess.DEVNULL,
                     stderr=subprocess.DEVNULL).wait()
    time.sleep(1)
    with open(restable, 'wb') as f:
        rc = subprocess.Popen(command, stdout=f,
                              stderr=subprocess.DEVNULL).wait()
    with open(restable, 'rb') as f:
        complete = b'Summary of results' in f.read()
    if rc == 0 and complete:
        break
    time.sleep(2)
else:
    raise RuntimeError(f"sumstats truncated output after {max_attempts} attempts")

Hypothesis

The ~4 KB cutoff strongly suggests a stdio buffer is not being flushed on process exit when stdout is redirected to a file/pipe on Windows (default _IOFBF for non-tty stdout, full buffer = 4096 bytes). A missing fflush(stdout) / setvbuf before exit, or an early _exit path that bypasses CRT cleanup, would match the symptoms exactly.

Impact

Silent data loss in any automated pipeline that consumes sumstats output on Windows — there is no signal (return code, stderr, or stream close error) that the data is incomplete.