Rootfs automount reload race: umount2(<rootfs>/proc) in host ns vs new prototype's mount() (flaky test_go_isolation_rootfs_automount_tmpfs)

[TuzelKO]

Summary

Under a rapid same-rootfs reload loop, the previous worker generation's mount teardown races the new prototype's proc mount on the same host path <rootfs>/proc, with no serialization. The loser hits mount(...proc...) ENOENT, the prototype exits 1, a respawn storm follows, the generation never converges, and dangling app↔router fds trip the fd-leak teardown check.

This surfaces as a flaky CI failure in test_go_isolation_rootfs_automount_tmpfs / ..._regression — see PR #78, run 27793786107 (test (go-1.25) red, test (go-1.26) green on the identical daemon and ns_inspect app). Original flaky context: #60.

Credit: root cause first pinned by @andypost in #78 (review).

Symptom (failing run, go-1.25)

[alert] mount("none", "/tmp/unit-test-…/proc", "proc", 2097162, "") (2: No such file or directory)
[notice] process … exited with code 1
FAILED test_go_isolation_rootfs_automount_tmpfs - mounts did not converge after reload (status=None)
ERROR  ...                                      - AssertionError: descriptors leak router (assert 53 <= 0)

The race

Each reload swaps the worker generation; two operations touch <rootfs>/proc concurrently:

• new prototype (child): mkdir + mount("none", <rootfs>/proc, "proc", …) — nxt_isolation_prepare_rootfs() (src/nxt_application.c:562 → src/nxt_isolation.c:778)
• old-gen teardown (main process, host mount ns): umount2(<rootfs>/proc, MNT_DETACH) — nxt_isolation_unmount_all() (src/nxt_isolation.c:744, registered as isolation.cleanup at :602)

The lazy detach lands on the path the new generation is mid-mounting → ENOENT → prototype exit 1 → respawn storm → status=None + leaked port fds. The Go toolchain version only shifts when the new mount() fires relative to the old umount2(); 1.25 hits the window, 1.26 misses it. This is timing, not a Go-version code path.

Why the existing mitigation is insufficient

The private-mount-namespace mitigation is already in place (cceba1e9 private mount namespace, 2063c666 depin reload race; tests load with namespaces: {mount: True} and skip_alert the ENOENT; proc-before-tmpfs ordering in nxt_isolation_mount_compare, src/nxt_isolation.c:716). It is necessary but not sufficient, because of the ordering in the child setup:

1. nxt_isolation_prepare_rootfs mounts <rootfs>/proc — src/nxt_application.c:562
2. only afterwards nxt_isolation_change_root → nxt_isolation_pivot_root severs propagation with mount("", "/", MS_SLAVE|MS_REC, "") — src/nxt_isolation.c:878

unshare(CLONE_NEWNS) alone does not detach mount propagation. On a systemd host, / (and /tmp, where temp_dir lives) is MS_SHARED by default, so in the window between unshare and pivot_root the new generation's proc mount is still propagation-linked to the host — exactly while the host-side teardown umount2s the same path. MS_SLAVE arrives too late to close the window.

Proposed fixes (pick one / combine)

1. Sever propagation before mounting: make the rootfs// MS_REC|MS_PRIVATE (or MS_SLAVE) before nxt_isolation_prepare_rootfs, not inside pivot_root after it.
2. Don't host-umount per-worker automounts: when the worker has a private mount ns, let its automounts die with the namespace instead of umount2-ing them from the host ns in nxt_isolation_unmount_all.
3. Serialize reload teardown: ensure the old generation's mount teardown completes before the new prototype starts mounting the same rootfs.

Test-side follow-up (PR #78)

• Until the C fix lands, mark test_go_isolation_rootfs_automount_tmpfs[_regression] xfail/skip on go-1.25, referencing this issue, to unblock the CI-workflow refactor (do not drop go-1.25 from the matrix — it is a supported runtime per pkg/eol.json).
• The _regression variant (×100 reload loop) is a CI time-bomb regardless of toolchain; consider gating it behind an opt-in marker.

Environment

• ubuntu-latest GitHub runner, root (is_su=True), rootfs isolation + namespaces: {mount: True} + tmpfs automount toggled per reload.
• Reproduces deterministically on the go-1.25 toolchain; not observed on go-1.26.

[andypost]

@TuzelKO ready for review

[TuzelKO]

Heads-up: a green go-1.25 run does NOT mean this is fixed

PR #78 (refactor/ci-workflows, head 65bd6aff) had test (go-1.25) pass — without the #84 fix merged. Confirmed it is not a fix landing early:

• fix(isolation): close rootfs automount reload race (#83) #84 is still open; the fix is not on pre-1.35.6 (no MS_REC | MS_PRIVATE / #83 in src/nxt_isolation.c on the base).
• The PR branch does not touch nxt_isolation.c, so the CI merge tree runs the unfixed daemon — same as the originally failing runs.
• Yet test_go_isolation_rootfs_automount_tmpfs and _regression (×100) both PASSED.

This is the race being flaky, not deterministic. The "always fails on go-1.25" framing was an artifact of a small sample (3 consecutive reds); the failure is timing-dependent by nature.

Timing evidence (same daemon, same test, opposite outcome):

• Failing run 27793786107: go test phase 347s — the worker wedged, _reload_and_poll_mounts exhausted its retries and failed with status=None.
• Passing run 27966929566: whole go suite 40.86s, the ×100 regression loop converged in ~16s — the worker never wedged, the race window was simply missed (the same luck-of-timing that keeps go-1.26 green).

Takeaway: an occasional green on go-1.25 is luck, not a fix. Until #84 lands on pre-1.35.6 (so PR #78 picks it up via the merge base, or after a rebase), the red can return on any unlucky run. The fix in #84 is still required for a durable green.

[andypost]

Merged so could be closed, please rebase #78