Skip to content

Commit ef9ba96

Browse files
Yolean k8s-qaclaude
andcommitted
e2e: retry provision on apiserver-not-yet-host-reachable race
A second y-cluster v0.3.3 docker-provider race surfaces in CI (filed as specs/y-cluster/ISSUE_DOCKER_K3S_READY_BEFORE_APISERVER.md): provision declares "k3s ready" once /etc/rancher/k3s/k3s.yaml exists inside the container, then immediately runs `kubectl apply` for the envoy-gateway install against the host-mapped 127.0.0.1:6443 -- on slower hosts (GHA runners) the host port forward isn't yet functional, the apply fails with "dial tcp 127.0.0.1:6443: connect: connection refused", and provision aborts. Extend the existing pre-pull workaround into a unified retry loop: on each provision attempt, if the failure log contains - "No such image" -> docker pull, retry - "dial tcp 127.0.0.1:6443: connect: connection refused" -> sleep 10s, retry - anything else -> propagate the failure as before Up to 4 attempts. Becomes dead code once y-cluster ships auto-pull + a stronger readiness check on the host port. Verified locally with cold image cache: pre-pull fires once, provision succeeds on second attempt, validate-ystack reports 37 passed, 0 failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent bd57a85 commit ef9ba96

1 file changed

Lines changed: 37 additions & 15 deletions

File tree

e2e/agents-clusterautomation-acceptance-linux-amd64.sh

Lines changed: 37 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -76,27 +76,49 @@ cleanup
7676

7777
# --- provision (no converge) ---
7878
#
79-
# y-cluster v0.3.3's docker provider does NOT auto-pull the k3s
80-
# image; it calls `docker create` directly and errors with "No
81-
# such image" when the image isn't already on the host. Until
82-
# y-cluster ships auto-pull, parse the image ref out of a
83-
# verbose-mode `provision --print-image`-style invocation isn't
84-
# available either, so we fall back to running provision once,
85-
# scraping the image from its progress log on failure, pulling
86-
# it, and retrying. Harmless when the image is already cached.
79+
# y-cluster v0.3.3's docker provider has two known races (filed
80+
# against y-cluster as ISSUE_DOCKER_PROVIDER_NO_AUTO_PULL.md and
81+
# ISSUE_DOCKER_K3S_READY_BEFORE_APISERVER.md):
82+
#
83+
# 1. ContainerCreate is called without a prior `docker pull`,
84+
# so a fresh host errors with "No such image". Workaround:
85+
# scrape the image ref from the failed log, pull, retry.
86+
# 2. The "k3s ready" signal fires when /etc/rancher/k3s/k3s.yaml
87+
# exists in the container, but the host's :6443 port forward
88+
# isn't always reachable yet -- the next step
89+
# (envoy-gateway install via kubectl apply) fails with
90+
# "dial tcp 127.0.0.1:6443: connect: connection refused".
91+
# Workaround: detect the connect-refused error, sleep, retry.
92+
#
93+
# Both branches reduce to a single `y-cluster provision -c "$CONFIG"`
94+
# once y-cluster ships fixes.
8795
if [ "$(grep -E '^provider:' "$CONFIG/y-cluster-provision.yaml" | awk '{print $2}')" = "docker" ]; then
88-
_PRE_OUT=$(mktemp -t ystack-acceptance-image-probe.XXXXXX)
89-
if ! y-cluster provision -c "$CONFIG" 2>&1 | tee "$_PRE_OUT"; then
90-
_IMG=$(grep -oE 'ghcr\.io/yolean/k3s:[a-zA-Z0-9._-]+' "$_PRE_OUT" | head -1)
91-
if [ -n "$_IMG" ]; then
92-
echo "# Pre-pulling $_IMG (y-cluster v0.3.3 docker provider does not auto-pull)"
93-
docker pull "$_IMG"
94-
y-cluster provision -c "$CONFIG"
96+
_PRE_OUT=$(mktemp -t ystack-acceptance-provision.XXXXXX)
97+
_attempt=1
98+
while [ "$_attempt" -le 4 ]; do
99+
if y-cluster provision -c "$CONFIG" 2>&1 | tee "$_PRE_OUT"; then
100+
break
101+
fi
102+
if grep -q 'No such image' "$_PRE_OUT"; then
103+
_IMG=$(grep -oE 'ghcr\.io/yolean/k3s:[a-zA-Z0-9._-]+' "$_PRE_OUT" | head -1)
104+
if [ -n "$_IMG" ]; then
105+
echo "# Pre-pulling $_IMG (y-cluster v0.3.3 docker provider does not auto-pull)"
106+
docker pull "$_IMG"
107+
fi
108+
elif grep -q 'dial tcp 127.0.0.1:6443: connect: connection refused' "$_PRE_OUT"; then
109+
echo "# k3s apiserver host port not reachable yet (y-cluster v0.3.3 readiness race); sleeping 10s before retry"
110+
sleep 10
95111
else
96112
cat "$_PRE_OUT" >&2
97113
rm -f "$_PRE_OUT"
98114
exit 1
99115
fi
116+
_attempt=$((_attempt + 1))
117+
done
118+
if [ "$_attempt" -gt 4 ]; then
119+
echo "# Provision failed after 4 attempts" >&2
120+
rm -f "$_PRE_OUT"
121+
exit 1
100122
fi
101123
rm -f "$_PRE_OUT"
102124
else

0 commit comments

Comments
 (0)