Skip to content

fix: nightly hardening - HTTP endpoint URL validation#29

Open
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260516-http-endpoint-url-validation
Open

fix: nightly hardening - HTTP endpoint URL validation#29
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260516-http-endpoint-url-validation

Conversation

@mouse-value-add

Copy link
Copy Markdown
Contributor

Problem

The HTTP provider accepted any string without validating URL shape. Misconfigured endpoints (for example or malformed ) fail later during runtime with less actionable errors.

Approach

  • Added validation with a Pydantic field validator.
  • Enforced / schemes only.
  • Required a host () so malformed URLs are rejected at config load time.
  • Added focused unit tests for accepted and rejected endpoint inputs.

Verification

  • Ran: ============================= test session starts ==============================
    platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0
    rootdir: /private/tmp/oss-nightly/PromptLens
    configfile: pyproject.toml
    plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
    asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
    collected 3 items

tests/test_provider_config_endpoint_validation.py ... [100%]

=============================== warnings summary ===============================
promptlens/models/tools.py:14
/private/tmp/oss-nightly/PromptLens/promptlens/models/tools.py:14: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class ToolParameter(BaseModel):

promptlens/models/test_case.py:10
/private/tmp/oss-nightly/PromptLens/promptlens/models/test_case.py:10: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class TestCase(BaseModel):

promptlens/models/test_case.py:66
/private/tmp/oss-nightly/PromptLens/promptlens/models/test_case.py:66: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class GoldenSet(BaseModel):

promptlens/models/config.py:115
/private/tmp/oss-nightly/PromptLens/promptlens/models/config.py:115: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class RunConfig(BaseModel):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 169 0%
promptlens/exporters/init.py 6 6 0%
promptlens/exporters/base.py 15 15 0%
promptlens/exporters/csv_exporter.py 26 26 0%
promptlens/exporters/html_exporter.py 38 38 0%
promptlens/exporters/json_exporter.py 16 16 0%
promptlens/exporters/markdown_exporter.py 77 77 0%
promptlens/judges/init.py 3 3 0%
promptlens/judges/base.py 15 15 0%
promptlens/judges/llm_judge.py 71 71 0%
promptlens/judges/parser.py 107 107 0%
promptlens/judges/prompts.py 30 30 0%
promptlens/loaders/init.py 4 4 0%
promptlens/loaders/base.py 14 14 0%
promptlens/loaders/json_loader.py 23 23 0%
promptlens/loaders/yaml_loader.py 34 34 0%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 53 1 98%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 25 0 100%
promptlens/models/tools.py 74 34 54%
promptlens/providers/init.py 3 3 0%
promptlens/providers/anthropic.py 52 52 0%
promptlens/providers/base.py 23 23 0%
promptlens/providers/factory.py 21 21 0%
promptlens/providers/google.py 47 47 0%
promptlens/providers/http.py 67 67 0%
promptlens/providers/openai.py 57 57 0%
promptlens/providers/you.py 57 57 0%
promptlens/runners/init.py 2 2 0%
promptlens/runners/runner.py 96 96 0%
promptlens/utils/init.py 1 1 0%
promptlens/utils/cost.py 15 15 0%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 21 0%
promptlens/utils/timing.py 24 24 0%

TOTAL 1381 1210 12%
Coverage HTML written to dir htmlcov
======================== 3 passed, 4 warnings in 0.30s =========================

  • Result: 3 tests passed.

Risks

  • Users with previously accepted but malformed endpoints will now fail fast at config parsing.
  • This is intentional hardening; risk is limited to invalid configs.

Rollback Plan

  • Revert this PR commit to restore previous permissive endpoint behavior.
  • If needed, relax validation rules in in a follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant