Skip to content

ci: retry transient Docker pulls during integration setup #1535

Description

@fallintoplace

Two integration jobs failed during setup when Docker Hub timed out while pulling images:

Both failed before tests ran. Their Spark 4 counterparts passed.

Integration setup currently runs docker compose up -d --wait once, so a temporary registry timeout fails the whole job.

A few possible fixes:

  1. Retry docker compose pull in CI with backoff, then start with --pull never. This seems safest and does not change local behavior.
  2. Retry the full integration setup command. This is simpler, but may recreate a partially started environment.
  3. Limit concurrent image pulls and add backoff to reduce registry pressure.
  4. Use a registry mirror or preloaded image cache.

Would you prefer the CI-only pull retry, or should the retry behavior live in the Makefile so it can also be used locally?

Happy to send a PR with the preferred approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions