test(v2): align v2 functional tests with current backend behavior#996
Merged
Conversation
- Inspector step detection: the backend now reports the inspector's own
name as the step agent id (e.g. 'abort_output_inspector',
is_system_agent=true) instead of 'inspector'/'inspector|...'; match on
substring. Verified via a live run (also returns a new governance block
with status BLOCKED_BY_INSPECTOR).
- Tavily: fetch the 'Tavily Web Search' connector tool by id
(6931bdf462eb386b7158def3). The marketplace path
tavily/tavily-search-api/Tavily resolves to the Legacy asset whose only
action is a generic 'run', breaking the 'search'-action tests.
- functions filter: v2 Function enum values are uppercase
(TEXT_GENERATION); compare enum name, and require non-empty results now
that the backend honors the filter.
- snake_case round-trips: Agent.get hydrates tools into objects, so read
the raw payload via client.request('get', 'sdk/agents/{id}') and assert
on the wire shape (assetId, allowMulti, supportsVariables). Verified
against the test platform.
- session reactions: skip both tests — verified backend bug: the sessions
service validates role but stores every message as role='user', so
assistant messages can never be reacted to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes 12 of the 17 remaining
setup-and-test (v2)failures ontest(run 29813378427). Each fix was verified against the live test platform where my credentials allowed.Fixed
agent.idis now the inspector's own name (abort_output_inspector,is_system_agent: true) instead of the oldinspector|...prefix, and there's a newgovernanceblock (BLOCKED_BY_INSPECTOR). Matcher updated to substring match ininspector_functional_test.pyandtest_arabic_agent.py.Tavily Web Searchby id6931bdf462eb386b7158def3(same asset the passing agent test uses). The pathtavily/tavily-search-api/Tavilyresolves to the Legacy asset whose only action is a genericrun— that's whyactions['search']failed. (Backend data issue worth reporting: two assets claim the same path.)Functionenum has uppercase values (TEXT_GENERATION), so the lowercase.valuecomparison was wrong. Compare enum name; also require non-empty results.Agent.gethydrates tools into objects, so the tests now read the raw wire payload viaclient.client.request('get', 'sdk/agents/{id}')and assert onassetId/allowMulti/supportsVariables. Passing against the live test platform.role(must be one of: user, assistant) but then stores every message asrole='user'(POST response + GET both confirm), so assistant messages can never be liked/disliked. Needs a sessions-service fix; unskip after.Not fixed (backend / needs investigation)
err.supplier_error: Invalid request data provided— actionSLACK_SEND_MESSAGEnow resolves, but the supplier rejects the payload; I can't inspect the action's input schema with local credentials.SUCCESSoutputs fromclaude-opus-4.6/gpt-4.1-nano(credits billed) — model-serving issue on test env.Verified: 1388 unit tests pass; 3 of the fixed tests run green against the live test platform locally.
🤖 Generated with Claude Code
https://claude.ai/code/session_018dCb6BE5bqjCrmsiPDGLRr