Skip to content

test: prune flaky/failing functional tests and trim CI matrix#1003

Merged
aix-ahmet merged 1 commit into
developmentfrom
skip-remove-flaky-functional-tests
Jul 23, 2026
Merged

test: prune flaky/failing functional tests and trim CI matrix#1003
aix-ahmet merged 1 commit into
developmentfrom
skip-remove-flaky-functional-tests

Conversation

@hadi-aix

Copy link
Copy Markdown
Contributor

Summary

Cleans up functional tests that are currently failing against the test backend and trims the CI matrix accordingly. Changes are grouped as removed, skipped, and CI.

Removed tests

  • tests/functional/model/run_utility_model_test.py — deleted the six utility-model run tests that fail with ResponseStatus.FAILED instead of SUCCESS: test_run_utility_model, test_utility_model_with_decorator, test_utility_model_string_concatenation, test_utility_model_code_as_string, test_utility_model_simple_function, test_utility_model_update. Kept test_utility_model_status and test_model_tool_creation; dropped the now-unused UtilityModelInput/utility_tool import.
  • Metrics tests — removed all metrics-related tests:
    • tests/functional/general_assets/asset_functional_test.py: test_list_metrics and test_run_metric (plus the unused MetricFactory import).
    • tests/functional/pipelines/designer_test.py: test_metric_pipeline (metric node) and its now-unused DatasetFactory import.

Skipped tests (@pytest.mark.skip with reason)

  • tests/functional/model/run_model_test.pytest_index_model_with_filter (all four embedding-model params: OpenAI Ada 002, Jina Clip v2 Multimodal, Multilingual E5 Large, BGE M3) and test_index_model_air_with_image (sample image asset 404s during document parsing).
  • tests/functional/model/run_connect_model_test.pytest_run_script_connection_tool and test_run_script_connection_tool_with_complex_inputs (script tool run returns Failure).
  • tests/functional/v2/test_agent.pytest_slack_tool_integration_with_agent (backend rejects the Slack send-message payload).
  • tests/functional/agent/ and tests/functional/team_agent/ — new directory-scoped conftest.py files skip every test collected under each directory (path-scoped so running the whole suite doesn't skip unrelated tests). Tests still show as skipped rather than silently uncollected.

CI

  • .github/workflows/main.yaml — removed the benchmark, pipeline_3.0_v1, pipeline_designer, and pipeline_create steps from the test matrix (both the test-suite list and the include block).

Notes

  • "Remove all tests related to metrics" was interpreted to include test_metric_pipeline (a metric-node pipeline test). The benchmark suite uses metrics but is about benchmarking, so it was left as-is (and is dropped from CI anyway).
  • All edited Python files pass py_compile.

🤖 Generated with Claude Code

Remove utility-model run tests that fail against the test backend, skip
flaky index-model, script-connection, and Slack tool tests, drop the
MetricFactory-based tests, skip the agent/team_agent suites via directory
conftests, and remove the benchmark/pipeline_3.0_v1/pipeline_designer/
pipeline_create steps from the CI matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hadi-aix
hadi-aix requested a review from aix-ahmet July 23, 2026 14:56
@aix-ahmet
aix-ahmet merged commit 04dea96 into development Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants