Skip to content

feat: nightly hardening - robust HTTP response parsing#35

Open
mouse-value-add wants to merge 2 commits into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260527-http-json-guard
Open

feat: nightly hardening - robust HTTP response parsing#35
mouse-value-add wants to merge 2 commits into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260527-http-json-guard

Conversation

@mouse-value-add

Copy link
Copy Markdown
Contributor

Problem

The generic HTTP provider assumes every successful HTTP response is JSON and calls directly. When local model servers or proxies return non-JSON payloads (for example, HTML/plain-text errors with 200/502 wrappers), PromptLens surfaces a low-context decode failure and provides limited diagnostics.

Approach

  • Switched HTTP response parsing to read raw text first, then parse with .
  • Added explicit non-JSON handling with a clear, user-facing error message.
  • Added structured logging that includes endpoint and a truncated response snippet (first 500 chars) for safer debugging.
  • Added async test coverage to verify returns a clear error when non-JSON content is returned.

Verification

  • Ran: ============================= test session starts ==============================
    platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0
    rootdir: /private/tmp/oss-nightly/PromptLens
    configfile: pyproject.toml
    plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
    asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
    collected 5 items

tests/test_http_provider_response_parsing.py ..... [100%]

=============================== warnings summary ===============================
../../../../Users/mouse/Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242: FutureWarning: You are using a non-supported Python version (3.9.6). Google will not post any further updates to google.api_core supporting this Python version. Please upgrade to the latest Python version, or at least Python 3.10, and then update google.api_core.
warnings.warn(message, FutureWarning)

../../../../Users/mouse/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35
/Users/mouse/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: urllib3/urllib3#3020
warnings.warn(

../../../../Users/mouse/Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

../../../../Users/mouse/Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

promptlens/providers/google.py:8
/private/tmp/oss-nightly/PromptLens/promptlens/providers/google.py:8: FutureWarning:

All support for the google.generativeai package has ended. It will no longer be receiving
updates or bug fixes. Please switch to the google.genai package as soon as possible.
See README for more details:

https://github.com/google-gemini/deprecated-generative-ai-python/blob/main/README.md

import google.generativeai as genai

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 169 0%
promptlens/exporters/init.py 6 6 0%
promptlens/exporters/base.py 15 15 0%
promptlens/exporters/csv_exporter.py 26 26 0%
promptlens/exporters/html_exporter.py 38 38 0%
promptlens/exporters/json_exporter.py 16 16 0%
promptlens/exporters/markdown_exporter.py 77 77 0%
promptlens/judges/init.py 3 3 0%
promptlens/judges/base.py 15 15 0%
promptlens/judges/llm_judge.py 71 71 0%
promptlens/judges/parser.py 107 107 0%
promptlens/judges/prompts.py 30 30 0%
promptlens/loaders/init.py 4 4 0%
promptlens/loaders/base.py 14 14 0%
promptlens/loaders/json_loader.py 23 23 0%
promptlens/loaders/yaml_loader.py 34 34 0%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 40 0 100%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 23 0 100%
promptlens/models/tools.py 73 34 53%
promptlens/providers/init.py 3 0 100%
promptlens/providers/anthropic.py 52 32 38%
promptlens/providers/base.py 23 6 74%
promptlens/providers/factory.py 21 10 52%
promptlens/providers/google.py 47 28 40%
promptlens/providers/http.py 73 7 90%
promptlens/providers/openai.py 57 36 37%
promptlens/providers/you.py 57 39 32%
promptlens/runners/init.py 2 2 0%
promptlens/runners/runner.py 96 96 0%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 11 27%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 2 90%
promptlens/utils/timing.py 24 1 96%

TOTAL 1371 993 28%
Coverage HTML written to dir htmlcov
======================== 5 passed, 5 warnings in 4.06s =========================

  • Result: 5 passed.

Risks

  • Slight behavior change in error strings for malformed responses.
  • Response body snippet logging could include server text, though truncated.

Rollback Plan

  • Revert this PR commit to restore previous behavior.
  • Remove the new non-JSON parsing test if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant