fix(adapters): propagate budget halts out of the LangChain callback - #55
Merged
Merged
Conversation
The callback handler raised BudgetExceeded / ApprovalDenied from its hooks, but LangChain swallows exceptions thrown inside a callback (it logs them and keeps running) unless the handler sets raise_error=True. So on a real LangChain agent the halt was silently dropped and the chain kept spending past budget — enforcement was a no-op. Set raise_error=True so the deterministic halt actually stops the run. Tests: a unit test drives the hooks against the stub daemon and asserts the loop budget halts at the cap (and that raise_error stays set, so this can't silently regress); a tool-gating test asserts a denied tool raises ApprovalDenied; and a skip-unless-langchain integration test runs a real FakeListLLM in a runaway loop and confirms it stops after exactly the budgeted number of calls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The LangChain
RiskKernelCallbackHandlerenforces budgets by raisingBudgetExceeded(andApprovalDenied) from its callback hooks. But LangChain swallows exceptions raised inside a callback — it logs them and keeps running — unless the handler opts in withraise_error = True.The handler never set it. So on a real LangChain agent, a budget halt raised in
on_llm_startwas silently dropped and the chain kept spending past its budget. Enforcement was a no-op on the exact path the adapter exists for.It is the kind of bug a hooks-only unit test cannot see (calling
on_llm_start()directly does raise) — only driving a real LangChaininvoke()surfaces it.The fix
Set
raise_error = Trueon the handler so the deterministic halt propagates out ofllm.invoke()and actually stops the run.Tests (
test_sdk.py)test_langchain_handler_enforces_loop_budget— drives the hooks against the stub daemon; the loop budget halts at the cap, and assertsraise_errorstays set so this can't silently regress (runs in CI, no langchain needed).test_langchain_handler_gates_denied_tool— a denied tool raisesApprovalDenied.test_langchain_integration_stops_runaway_loop—@skipUnless(langchain installed): a realFakeListLLMin a runawaywhile Trueloop, confirmed to stop after exactly the budgeted number of calls. This is the test that actually catches the bug.Verification
langchain-coreinstalled: all 12 SDK tests pass (the integration test runs and stops the loop after 2 calls).