Skip to content

flaky CI: replication-ci compose fails with 'container mysql-1 is unhealthy' (first-boot init exceeds ~60s healthcheck budget) #962

Description

@morgo

What happened

The MySQL 8.0.45 (with replicas) /w docker-compose workflow (mysql8-docker.yml) failed on PR #961 before any Go test ran: docker compose up gave up waiting for the primary MySQL container's healthcheck.

Failing run: https://github.com/block/spirit/actions/runs/27386543696/job/80934676959

00:36:34  Container replication-tls-mysql-1  Created
00:36:36  [Entrypoint]: Entrypoint script for MySQL Server 8.0.45-1.el9 started.
00:36:37  [Entrypoint]: Initializing database files
00:36:53  [InnoDB] InnoDB initialization has ended.
00:37:09  [Server] root@localhost is created with an empty password
00:37:39  dependency failed to start: container replication-tls-mysql-1 is unhealthy
##[error]Process completed with exit code 1.

Why

compose/replication-tls/replication-ci.yml uses:

healthcheck:
  test: mysqladmin ping -h 127.0.0.1 -u root --password=$$MYSQL_ROOT_PASSWORD
  start_period: 1s
  interval: 1s
  timeout: 2s
  retries: 60

That's a ~60s budget from container start. On a slow/loaded runner, MySQL's first-boot initialization (--initialize data-dir creation, then the real server start) can eat the whole budget — and during --initialize the server accepts no connections at all, so every ping attempt burns a retry. In the failing run the entrypoint was still in the initialize phase 35s in (root user created at 00:37:09), and compose declared the container unhealthy 65s after creation, before the real server ever started listening.

The top-level compose/compose.yml and compose/semisync.yml have identical timing and are exposed to the same flake.

Evidence it's a flake, not the PR

  • The job failed during environment standup; zero tests executed.
  • The same workflow passed on the same branch ~1h earlier (run at 2026-06-11T23:26:57Z) and passed on 6 other branches within minutes of this failure (00:19–00:33 window).
  • The 7 sibling matrix builds on the same PR commit all passed.

Suggested fix

Raise start_period (e.g. to 120s) while keeping interval: 1s / retries: 60. Healthcheck failures during start_period don't count against retries, and a container that becomes healthy early still passes immediately — so this adds no latency to the common case and stops first-boot initialization from being charged against the retry budget. Apply the same change to compose/compose.yml, compose/semisync.yml, and the replica's replication_health_check.sh healthcheck in replication-ci.yml.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions