Skip to content

[NOT-491] Skip live SDK tests without NOTTE_API_KEY #860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HUAN2022A wants to merge 1 commit into from
+10 −3
Closed

[NOT-491] Skip live SDK tests without NOTTE_API_KEY #860

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
from pathlib import Path

import notte_core
import pytest

CONFIG_PATH = Path(__file__).parent / "test_notte_config.toml"
notte_core.set_error_mode("developer")
Expand Down Expand Up @@ -49,3 +50,9 @@ def pytest_generate_tests(metafunc):
# Apply parameterization only if any matching arguments exist
if params:
metafunc.parametrize(",".join(params.keys()), [next(iter(params.values()))])


@pytest.fixture
def require_notte_api_key():
if os.getenv("NOTTE_API_KEY") is None:

@coderabbitai coderabbitai Bot Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Consider checking for empty-string API key values.

os.getenv("NOTTE_API_KEY") returns "" (not None) when the variable is set but empty. The test won't skip in that case but will likely fail at the API call. Consider using a truthiness check or strip.

🛡️ Proposed fix 
-    if os.getenv("NOTTE_API_KEY") is None:
+    if not os.getenv("NOTTE_API_KEY"):
         pytest.skip("NOTTE_API_KEY is required for live Notte API tests")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if os.getenv("NOTTE_API_KEY") is None:
if not os.getenv("NOTTE_API_KEY"):
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/conftest.py` at line 57, Update the NOTTE_API_KEY availability check in
the test setup to treat both missing and empty or whitespace-only values as
unavailable, using a truthiness or stripped-value check so tests skip before
making API calls.

pytest.skip("NOTTE_API_KEY is required for live Notte API tests")
Comment on lines +55 to +58

@coderabbitai coderabbitai Bot Jul 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check whether load_dotenv is already called in conftest.py before the fixture,
# and whether os and dotenv are imported.

# Search for load_dotenv calls in conftest.py
rg -n 'load_dotenv' tests/conftest.py

# Check imports for os and dotenv
rg -n '^import os|^from dotenv|^from notte' tests/conftest.py

# Check if other test files also call load_dotenv inside the test body
rg -n 'load_dotenv' tests/sdk/test_replay_frames.py tests/sdk/test_validator.py

Repository: nottelabs/notte

Length of output: 165

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== tests/conftest.py =="
wc -l tests/conftest.py
cat -n tests/conftest.py | sed -n '1,140p'

echo
echo "== load_dotenv references in tests/ =="
rg -n 'load_dotenv|NOTTE_API_KEY|require_notte_api_key' tests

echo
echo "== files that mention dotenv loading outside tests/ =="
rg -n 'load_dotenv|dotenv' .

Repository: nottelabs/notte

Length of output: 21594

Load .env before checking NOTTE_API_KEY
require_notte_api_key runs before any per-test load_dotenv() call, so tests like tests/sdk/test_list.py can be skipped even when the key exists in .env. Call load_dotenv() in the fixture or move dotenv loading to session startup.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/conftest.py` around lines 55 - 58, Update the require_notte_api_key
fixture to load environment variables from .env before checking NOTTE_API_KEY,
using the existing dotenv utility/import; preserve the current skip behavior
when the key remains unavailable.

2 changes: 1 addition & 1 deletion tests/sdk/test_list.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from notte_sdk import NotteClient


def test_simple_listing():
def test_simple_listing(require_notte_api_key):
_ = load_dotenv()
notte = NotteClient()

Expand Down
2 changes: 1 addition & 1 deletion tests/sdk/test_replay_frames.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import notte


def test_sdk_screenshots():
def test_sdk_screenshots(require_notte_api_key):
"""Make sure everything is in bytes (and not base64 encoded), all in JPEG"""
client = NotteClient()

Expand Down
2 changes: 1 addition & 1 deletion tests/sdk/test_validator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ class Product(BaseModel):


@pytest.mark.flaky(reruns=3, reruns_delay=2)
def test_validator_message_received():
def test_validator_message_received(require_notte_api_key):
"""Test that validation failures are logged and agent can recover with a valid response."""
log_buffer = io.StringIO()
_ = logger.add(log_buffer, format="{message}")
Expand Down