Skip to content

fix: honor context cancellation in node health/exit wait loops - #3257

Open
mvanhorn wants to merge 5 commits into
srl-labs:mainfrom
mvanhorn:fix/3162-ctx-cancel-wait-loops
Open

fix: honor context cancellation in node health/exit wait loops#3257
mvanhorn wants to merge 5 commits into
srl-labs:mainfrom
mvanhorn:fix/3162-ctx-cancel-wait-loops

Conversation

@mvanhorn

Copy link
Copy Markdown
Contributor

Summary

Make the per-node deploy wait loops honor context cancellation so a stuck containerlab deploy unwinds on Ctrl-C instead of hanging until SIGQUIT.

Why this matters

When a node container crashes or disappears mid-deploy, a dependent worker in scheduleNodeWorkerF (core/clab.go) spins on time.Sleep(time.Second) waiting for a state that never arrives. Ctrl-C only cancels the deploy context, so these bare sleeps and the blocking stage waits never observe the cancellation and the process hangs with open sockets until SIGQUIT (issue #3162; the current workaround is --skip-post-deploy).

The fix closes the hang at every layer it could occur:

  • Replace the bare time.Sleep calls in the WaitForHealthy and WaitForExit poll loops, and the StartupDelay sleep, with select { case <-ctx.Done(): return; case <-time.After(...): } (core/clab.go).
  • Make DependencyNode.EnterStage wait on its stage waitgroup through a select on ctx.Done() so a node blocked on an unmet dependency returns on cancellation instead of blocking forever (core/dependency_manager/dependency_node.go). This is the single blocking wait path for all stage dependencies, so making it cancellation-aware unwinds dependent workers too.
  • After each EnterStage call in the worker, return early when ctx.Err() != nil. In scheduleNodes, move wfcwg.Done() into a defer, bail on cancellation after the create-stage wait, and make the workerChan send a select on ctx.Done() so the channel always closes and the send cannot block once workers have unwound.
  • In Deploy, check ctx.Err() right after NodesWg.Wait() and return the cancellation error instead of running host endpoint deployment, vxlan stitching, exports, and inventories on a partially deployed lab (core/deploy.go).

Happy-path behavior is unchanged: the loops still poll every second and break when a node turns healthy or exits, and post-deploy ordering is untouched.

Testing

Added focused unit tests: core/clab_test.go drives scheduleNodeWorkerF into the health wait loop with a context cancelled during the health check and asserts the worker returns rather than blocking past a short deadline; core/dependency_manager/dependency_node_test.go blocks EnterStage on an unmet dependency, cancels the context, and asserts it returns. gofmt is clean and core/ plus core/dependency_manager/ (and their test binaries) build and vet under GOOS=linux. Full go test execution requires a Linux host because containerlab pulls Linux-only dependencies.

Fixes #3162

mvanhorn and others added 5 commits June 29, 2026 02:11
The per-node wait loops in scheduleNodeWorkerF polled node health and
container exit status with a bare time.Sleep(time.Second), so a deploy
that was interrupted (Ctrl-C only cancels the context) kept spinning
when a dependency never reached the awaited state (e.g. a crashed or
removed container). The worker goroutine could not unwind, leaving
containerlab deploy hanging until SIGQUIT.

Replace the bare sleeps in the WaitForHealthy and WaitForExit loops, and
the StartupDelay sleep, with a select on ctx.Done() vs time.After so the
worker returns promptly once the deploy context is cancelled. Happy-path
polling cadence is unchanged.

Fixes srl-labs#3162

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQ7G74DcZPFPEgDJMTEcCR
Addresses review feedback: cancelling the deploy context unwinds the
polling worker, but a node already blocked in EnterStage waiting on a
dependency stage (a bare sync.WaitGroup.Wait) stayed parked, so
nodesWg.Wait could still hang after Ctrl-C. Wait on the stage waitgroup
in a goroutine and select against ctx.Done so EnterStage returns when the
deploy is cancelled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQ7G74DcZPFPEgDJMTEcCR
Addresses review feedback:
- Initialize CLab.Config in the worker test so scheduleNodeWorkerF does
  not panic dereferencing Config.Name before reaching the cancellation
  path.
- After each EnterStage call the worker now checks ctx.Err() and returns
  if the deploy was cancelled, so a cancelled stage wait no longer falls
  through into PostDeploy / the health and exit polls.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQ7G74DcZPFPEgDJMTEcCR
Addresses review feedback: the scheduler goroutine called
EnterStage(ctx, WaitForCreate) and then sent the node on the worker
channel unconditionally. After Ctrl-C the workers have already returned,
so the send blocked forever and close(concurrentChan) never ran, leaving
the deploy hanging. Bail out when ctx is cancelled after the create wait,
and select the worker-channel send against ctx.Done so it can never
block once the workers have unwound.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQ7G74DcZPFPEgDJMTEcCR
Addresses review feedback: after the node workers unwind on Ctrl-C,
Deploy treated their completion as success and continued into host
endpoint deployment, vxlan stitching, export and inventory generation
for a partially deployed lab. Check ctx.Err() right after NodesWg.Wait()
and return the cancellation error so a cancelled deploy stops promptly
instead of doing post-deploy work on an incomplete topology.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQ7G74DcZPFPEgDJMTEcCR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deploy hangs indefinitely with nokia_sros (vrnetlab) nodes after bootstrap completes

1 participant