CI: reduce flakiness in the libssh2, trackmemory, and socat jobs#10969
Merged
Conversation
The libssh2 integration tests start a fresh OpenSSH-server container per test with "docker run -d -p 22" and then immediately open the SSH connection. Docker maps the port as soon as the container starts, so the TCP connect succeeds before sshd inside the container is ready to send its banner, and the handshake fails with "Failed getting banner" (-43). This shows up as an intermittent failure of a single test, most often test_read, on unrelated pull requests. Fall back to "make recheck" when "make check" fails. The automake parallel test harness re-runs only the tests that did not pass, so a transient container-readiness race no longer fails the job while a genuine failure still does.
parallel-make-check.py schedules configs longest-first by their "minutes"
weight, but five of the seven trackmemory configs were weighted well below
their real runtime, which the run summary flags every time ("ran X min but
minutes says Y (>50% off)"). The underweighted configs include the two
wolfEntropy jitter configs, whose runtime already varies severalfold with
runner contention. Scheduling them last, on stale short weights, is what
lets a contended run drift into the 40-minute timeout.
Update the five flagged weights to the durations reported by the run
summary and re-sort the list descending so the slowest, most variable
configs start in the first scheduling wave and get the full time budget.
Config coverage is unchanged.
A few socat core tests intermittently fail the interop job with diff or timeout mismatches. They exercise socat's own system()/socket/exec-pty plumbing, not the wolfSSL backend, and race under the deliberate CPU oversubscription of the parallel shard scheduling. With parallel-make-check running fail-fast, one such flake in any shard aborts the whole job. Add the observed offenders to the per-version socat test.sh --expect-fail list: 23 (DUALSYSTEMFDSFLUSH) on 1.8.0.3, and 205 (TCP4ENDCLOSE) and 227 (EXECPTYKILL) on 1.8.0.0. test.sh keys its exit status only on unexpected failures, so a listed test that later passes is reported as "OK unexpected" and does not fail the job. Numbers come from each version's own test output, so they match that version's numbering.
Frauschi
changed the title
Frauschi
force-pushed
the
ci_flakyness_fixes
branch
3 times, most recently
from
July 23, 2026 06:15
20b1a09 to
adb0da7
Compare
Frauschi
changed the title
SparkiDev
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent CI reliability fixes.
1. libssh2: retry the Docker tests with
make recheckThe libssh2 integration tests start a fresh OpenSSH-server container per test with
docker run -d -p 22and then immediately open the SSH connection. Docker maps the host port as soon as the container starts, so the TCP connect succeeds beforesshdinside the container is ready to send its banner, and the handshake fails withFailed getting banner(-43). This surfaces as an intermittent failure of a single test, most oftentest_read, on unrelated pull requests.Fall back to
make recheckwhenmake checkfails. Automake's parallel test harness re-runs only the tests that did not pass, so a transient container-readiness race no longer fails the job while a genuine failure still does.2. trackmemory: correct stale make-check scheduling weights
parallel-make-check.pyschedules configs longest-first by theirminutesweight, but five of the seven trackmemory configs were weighted well below their real runtime, which the run summary flags every run (ran X min but minutes says Y (>50% off)). The underweighted set includes the two wolfEntropy jitter configs, whose runtime already varies severalfold with runner contention. Scheduling them last on stale short weights is what lets a contended run drift into the 40-minute timeout.Update the five flagged weights to the durations reported by the run summary and re-sort the list descending, so the slowest and most variable configs start in the first scheduling wave and get the full time budget. Config coverage is unchanged; this touches scheduling metadata only.
3. socat: tolerate timing-flaky non-TLS tests
A few socat tests intermittently fail the interop job with diff or timeout mismatches:
23 DUALSYSTEMFDSFLUSHon 1.8.0.3, and205 TCP4ENDCLOSEand227 EXECPTYKILLon 1.8.0.0. They exercise socat's ownsystem(), socket, and exec-pty plumbing rather than the wolfSSL backend, and race under the deliberate CPU oversubscription of the parallel shard scheduling. Withparallel-make-check.pyrunning fail-fast, one such flake in any shard aborts the whole job.Add the observed tests to socat
test.sh's per-version--expect-faillist.test.shkeys its exit status only on unexpected failures, so a listed test that later passes is reported asOK unexpectedand does not fail the job. The numbers come from each version's own test output, so they match that version's numbering.