Fix lgrep pool wedge from abandoned index#3
Merged
Conversation
Change: fixLgrepPoolWedgeAbandoned Task: tk-45441be6f257 Mode: complete Verification: TDD red: 4 tests in tests/test_runtime_cancellation.py, all fail against current code. Failures: ImportError on OperationCancelled (AC1, doesn't exist); AssertionError on cancel_event not set by run_blocking (AC2, current code doesn't propagate); AttributeError on state.db (AC3, mock setup needs more attrs).
Change: fixLgrepPoolWedgeAbandoned Task: tk-cd9d5fa9cb07 Mode: complete Verification: Added OperationCancelled exception and threading.Event kwarg to Indexer.index_all. Loop checks cancel_event at top of each iteration. AC1 tests pass (2/2).
Change: fixLgrepPoolWedgeAbandoned Task: tk-dafdc284c4ca Mode: complete Verification: RuntimeSupervisor.run_blocking now accepts cancel_event kwarg and sets it on asyncio.CancelledError before propagating. AC2 test passes.
Change: fixLgrepPoolWedgeAbandoned Task: tk-f9aa510faf46 Mode: complete Verification: _auto_index_project_single_flight now constructs cancel_event per attempt, passes it to runtime.run_blocking, and catches OperationCancelled to return state cleanly. AC1 + AC2 tests pass.
Change: fixLgrepPoolWedgeAbandoned Task: tk-3e97da692b6b Mode: complete Verification: _check_staleness now bounded by LGREP_STALENESS_DEADLINE_S (default 4.0s). On deadline, returns (False, 0) and emits staleness_check_deadline_exceeded log. All 4 tests pass.
Change: fixLgrepPoolWedgeAbandoned Task: tk-a66cd1a02904 Mode: complete Verification: Updated skills/lgrep/SKILL.md with staleness deadline section. Updated instructions/lgrep-tools.md with one-line fallback note. No code changes.
Change: fixLgrepPoolWedgeAbandoned Task: tk-98253123e32d Mode: complete Verification: Full pytest: 601 passed, 2 skipped, 0 failed. Ruff: clean.
Change: fixLgrepPoolWedgeAbandoned Task: tk-7e160f0d3360 Mode: complete Verification: TDD red: added 5 v2 tests covering AC8 (embed_documents between-batch cancel + _embed_batch_with_retry immediate wait-abort), AC9 (index_file pre-embed + pre-storage cancel), AC10 (index_all wall-clock backstop). All 5 FAIL against v1 code (ImportError on lgrep.exceptions for AC8; no cancel_event on index_file/embed_documents for AC9; no wall-clock backstop for AC10). Confirmed 5 failed.
Change: fixLgrepPoolWedgeAbandoned Task: tk-2f2785109a4a Mode: complete Verification: Created src/lgrep/exceptions.py with OperationCancelled; indexing.py now imports + re-exports it (from lgrep.exceptions import OperationCancelled, plus __all__). Verified `is` identity between lgrep.indexing.OperationCancelled and lgrep.exceptions.OperationCancelled; v1 tests (AC1 index_all cancel + mid-loop) still pass 2/2. Circular import broken: embeddings can now import from lgrep.exceptions.
…checks) - Add cancel_event kwarg to index_file signature and docstring - Raise OperationCancelled before embed step when event is set - Pass cancel_event into embed_documents call - Raise OperationCancelled before storage step when event is set - Wire cancel_event through index_all per-file calls - Update test monkey-patch to accept **kwargs for new signature Verification: 2 AC9 tests + 2 existing index tests pass; ruff clean.
- embed_documents: accept cancel_event, check between batches - _embed_batch_with_retry: accept cancel_event, check before each attempt, pass through recursive token-split calls, replace time.sleep(delay) with interruptible cancel_event.wait(timeout=delay) for immediate backoff abort - Imports: threading (TYPE_CHECKING), OperationCancelled from lgrep.exceptions Verification: - pytest tests/test_runtime_cancellation.py -k embed : 3 passed - pytest tests/ -k embed -q : 23 passed - ruff check src/lgrep/embeddings.py : clean
Change: fixLgrepPoolWedgeAbandoned Task: tk-d738b1eb0c9a Mode: complete Verification: AC10 wall-clock backstop added to index_all (indexing.py:92 reads LGREP_INDEX_MAX_WALL_S default 60.0s; line 126 emits index_all_wall_clock_exceeded warning + raises OperationCancelled when budget exceeded, placed right after the cancel_event check at the loop top). Verified via adv_run_test: all 9 tests in test_runtime_cancellation.py pass (AC1,AC2,AC3,AC8,AC9,AC10 + existing). Engineer's checkpoint did not land so committing here.
Change: fixLgrepPoolWedgeAbandoned Task: tk-14cc20156f84 Mode: complete Verification: AC7 verified via adv_run_test: full suite `uv run pytest` = 606 passed, 2 skipped, 0 failed (up from 601; +5 v2 cancellation tests). `uv run ruff check src tests` = All checks passed. AC6 satisfied: tests/test_runtime_cancellation.py now covers AC1,AC2,AC3,AC8,AC9,AC10 + rq-daemon-cancel01.3 (9 tests, all green). Also updated skills/lgrep/SKILL.md with the cooperative-cancellation + LGREP_INDEX_MAX_WALL_S backstop documentation.
Acceptance review caught that _auto_index_project_single_flight passed cancel_event to run_blocking (which only .set()s it on cancellation) but did NOT forward it into index_all() itself — so the per-file + mid-embed cooperative cancel checks never fired in the live search path; only the wall-clock backstop could terminate an abandoned index. Wrap the call in a lambda that passes cancel_event=cancel_event into index_all so AC1/AC8/AC9 cooperative cancellation is effective in production, not just unit tests. Also update two test_server.py doubles to accept ignored kwargs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
lgrep's shared MCP daemon permanently wedged its 2-worker bounded executor when a search-triggered auto-index was abandoned at the 8s tool timeout. The thread kept running with no cooperative cancellation, so abandoned
index_alljobs exhausted the pool and every subsequent pokeedge search timed out.Fix
Cooperative cancellation threaded to every blocking seam, plus a wall-clock backstop:
OperationCancelledmoved tolgrep/exceptions.py(breaks indexing↔embeddings import cycle; re-exported fromindexing.pyfor back-compat)cancel_eventthreaded throughindex_all→index_file(pre-embed/pre-storage) →embed_documents(per-batch) →_embed_batch_with_retry(per-attempt +cancel_event.wait(timeout=delay)replacing an up-to-31s un-cancellable retry sleep)RuntimeSupervisor.run_blockingsets the event onasyncio.CancelledErrorbefore re-raisingLGREP_INDEX_MAX_WALL_S(default 60s) wall-clock backstop onindex_all_check_stalenessbounded byLGREP_STALENESS_DEADLINE_S(default 4.0s)_auto_index_project_single_flightnow passescancel_eventINTOindex_all(caught in acceptance review)Verification
tests/test_runtime_cancellation.py)index_allreached terminalfailed_after_abandonand freed its worker — permanent wedge resolvedlgrepDaemonOperationalSafety(rq-daemon-cancel01.1/.2/.3, rq-daemon-executor01.1/.2) honored + strengthenedFollow-ups (out of scope)
client.embed()HTTP timeout for sub-30s abandoned-job termination (changeaddVoyageEmbedHttpTimeoutcreated)cancel_eventthroughindex_semantictool + watcher paths🤖 Generated with ADV