Skip to content

fix: nightly hardening - http empty-content error surfacing#23

Open
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260508-http-empty-content-error
Open

fix: nightly hardening - http empty-content error surfacing#23
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260508-http-empty-content-error

Conversation

@mouse-value-add

Copy link
Copy Markdown
Contributor

Problem

The HTTP provider could return a successful with empty content when upstream endpoints returned unsupported payload shapes or structured error payloads. This made failures look like normal empty generations and hid root-cause diagnostics.

Approach

  • Added to normalize common error fields (, , , including nested ).
  • In , if extracted content is empty:
    • raise a descriptive error when an upstream error payload is present
    • otherwise raise an explicit unsupported-shape error with top-level keys
  • Added focused unit tests for error extraction behavior.

Verification

  • Ran:
    • ============================= test session starts ==============================
      platform darwin -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0
      rootdir: /Users/mouse/.openclaw/workspace/PromptLens
      configfile: pyproject.toml
      plugins: cov-7.1.0, anyio-4.13.0
      collected 7 items

tests/test_http_provider_empty_content_guard.py ... [ 42%]
tests/test_http_provider_response_parsing.py .... [100%]

=============================== warnings summary ===============================
promptlens/models/tools.py:14
/Users/mouse/.openclaw/workspace/PromptLens/promptlens/models/tools.py:14: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.13/migration/
class ToolParameter(BaseModel):

promptlens/models/test_case.py:10
/Users/mouse/.openclaw/workspace/PromptLens/promptlens/models/test_case.py:10: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.13/migration/
class TestCase(BaseModel):

promptlens/models/test_case.py:66
/Users/mouse/.openclaw/workspace/PromptLens/promptlens/models/test_case.py:66: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.13/migration/
class GoldenSet(BaseModel):

promptlens/models/config.py:100
/Users/mouse/.openclaw/workspace/PromptLens/promptlens/models/config.py:100: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.13/migration/
class RunConfig(BaseModel):

promptlens/providers/google.py:8
/Users/mouse/.openclaw/workspace/PromptLens/promptlens/providers/google.py:8: FutureWarning:

All support for the google.generativeai package has ended. It will no longer be receiving
updates or bug fixes. Please switch to the google.genai package as soon as possible.
See README for more details:

https://github.com/google-gemini/deprecated-generative-ai-python/blob/main/README.md

import google.generativeai as genai

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
______________ coverage: platform darwin, python 3.12.13-final-0 _______________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 169 0%
promptlens/exporters/init.py 6 6 0%
promptlens/exporters/base.py 15 15 0%
promptlens/exporters/csv_exporter.py 26 26 0%
promptlens/exporters/html_exporter.py 38 38 0%
promptlens/exporters/json_exporter.py 16 16 0%
promptlens/exporters/markdown_exporter.py 77 77 0%
promptlens/judges/init.py 3 3 0%
promptlens/judges/base.py 15 15 0%
promptlens/judges/llm_judge.py 71 71 0%
promptlens/judges/parser.py 107 107 0%
promptlens/judges/prompts.py 30 30 0%
promptlens/loaders/init.py 4 4 0%
promptlens/loaders/base.py 14 14 0%
promptlens/loaders/json_loader.py 23 23 0%
promptlens/loaders/yaml_loader.py 34 34 0%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 41 0 100%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 25 0 100%
promptlens/models/tools.py 74 34 54%
promptlens/providers/init.py 3 0 100%
promptlens/providers/anthropic.py 52 32 38%
promptlens/providers/base.py 23 6 74%
promptlens/providers/factory.py 21 10 52%
promptlens/providers/google.py 47 28 40%
promptlens/providers/http.py 84 32 62%
promptlens/providers/openai.py 57 36 37%
promptlens/providers/you.py 57 39 32%
promptlens/runners/init.py 2 2 0%
promptlens/runners/runner.py 96 96 0%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 11 27%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 15 29%
promptlens/utils/timing.py 24 13 46%

TOTAL 1386 1043 25%
Coverage HTML written to dir htmlcov
======================== 7 passed, 5 warnings in 1.51s =========================

  • Result: all tests passed (7 passed).

Risks

  • Some custom endpoints may intentionally return empty content for valid calls. Those now return explicit errors instead of silent empties.

Rollback Plan

  • Revert this PR commit to restore previous permissive behavior for empty content responses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant