What happened
The MySQL 8.0.45 (with replicas) /w docker-compose workflow (mysql8-docker.yml) failed on PR #961 before any Go test ran: docker compose up gave up waiting for the primary MySQL container's healthcheck.
Failing run: https://github.com/block/spirit/actions/runs/27386543696/job/80934676959
00:36:34 Container replication-tls-mysql-1 Created
00:36:36 [Entrypoint]: Entrypoint script for MySQL Server 8.0.45-1.el9 started.
00:36:37 [Entrypoint]: Initializing database files
00:36:53 [InnoDB] InnoDB initialization has ended.
00:37:09 [Server] root@localhost is created with an empty password
00:37:39 dependency failed to start: container replication-tls-mysql-1 is unhealthy
##[error]Process completed with exit code 1.
Why
compose/replication-tls/replication-ci.yml uses:
healthcheck:
test: mysqladmin ping -h 127.0.0.1 -u root --password=$$MYSQL_ROOT_PASSWORD
start_period: 1s
interval: 1s
timeout: 2s
retries: 60
That's a ~60s budget from container start. On a slow/loaded runner, MySQL's first-boot initialization (--initialize data-dir creation, then the real server start) can eat the whole budget — and during --initialize the server accepts no connections at all, so every ping attempt burns a retry. In the failing run the entrypoint was still in the initialize phase 35s in (root user created at 00:37:09), and compose declared the container unhealthy 65s after creation, before the real server ever started listening.
The top-level compose/compose.yml and compose/semisync.yml have identical timing and are exposed to the same flake.
Evidence it's a flake, not the PR
- The job failed during environment standup; zero tests executed.
- The same workflow passed on the same branch ~1h earlier (run at 2026-06-11T23:26:57Z) and passed on 6 other branches within minutes of this failure (00:19–00:33 window).
- The 7 sibling matrix builds on the same PR commit all passed.
Suggested fix
Raise start_period (e.g. to 120s) while keeping interval: 1s / retries: 60. Healthcheck failures during start_period don't count against retries, and a container that becomes healthy early still passes immediately — so this adds no latency to the common case and stops first-boot initialization from being charged against the retry budget. Apply the same change to compose/compose.yml, compose/semisync.yml, and the replica's replication_health_check.sh healthcheck in replication-ci.yml.
What happened
The
MySQL 8.0.45 (with replicas) /w docker-composeworkflow (mysql8-docker.yml) failed on PR #961 before any Go test ran:docker compose upgave up waiting for the primary MySQL container's healthcheck.Failing run: https://github.com/block/spirit/actions/runs/27386543696/job/80934676959
Why
compose/replication-tls/replication-ci.ymluses:That's a ~60s budget from container start. On a slow/loaded runner, MySQL's first-boot initialization (
--initializedata-dir creation, then the real server start) can eat the whole budget — and during--initializethe server accepts no connections at all, so every ping attempt burns a retry. In the failing run the entrypoint was still in the initialize phase 35s in (root user created at 00:37:09), and compose declared the container unhealthy 65s after creation, before the real server ever started listening.The top-level
compose/compose.ymlandcompose/semisync.ymlhave identical timing and are exposed to the same flake.Evidence it's a flake, not the PR
Suggested fix
Raise
start_period(e.g. to120s) while keepinginterval: 1s/retries: 60. Healthcheck failures duringstart_perioddon't count againstretries, and a container that becomes healthy early still passes immediately — so this adds no latency to the common case and stops first-boot initialization from being charged against the retry budget. Apply the same change tocompose/compose.yml,compose/semisync.yml, and the replica'sreplication_health_check.shhealthcheck inreplication-ci.yml.