Currently, testing our demo user journeys relies heavily on full evaluation runs or manual testing. To catch regressions early and ensure we don't break existing demos, we need a lightweight, deterministic verification mechanism that runs end-to-end user journeys without the cost, latency, or non-determinism of an LLM.
Proposed Solution
Introduce an automated smoke-test runner that reuses our existing eval files (similar to the patterns seen in PR #299).
Instead of passing prompts to an LLM to decide the next actions, the test runner will programmatically and deterministically trigger the tool calls defined in the eval files against a target website. This allows us to verify that the core journeys, tool integrations, and UI states function as expected.
Scope
Execute predefined tool calls sequence-by-sequence from eval files without LLM involvement.
Provide a fast feedback loop in CI/local development to confirm demos and core journeys are operational.
Leverage existing eval file formats so we don't need to maintain a separate set of test definitions.
Challenges & Considerations
-
The runner requires an active, running web application to test against. I am considering supporting two execution modes:
-
The smoke test runner automatically spins up the local app/server before executing tests and tears it down afterward.
-
The runner accepts a target URL (e.g., staging or a live deployed demo) to run the checks against.
-
Because these tests interact with running websites and real browser/network states, they may be prone to environmental flakiness. We should implement sensible retry logic, explicit wait states, or timeout handling to mitigate false positives.
Acceptance Criteria
-
A test runner script/utility is implemented that can parse existing eval files and execute tool sequences deterministically.
-
Support for configuring the target environment (e.g., via environment variables or CLI flags for local vs. deployed URLs).
-
Basic error reporting that clearly highlights which tool call or step failed during the journey.
-
Documentation updated with instructions on how to run the smoke tests locally and how to format eval files for compatibility.
-
(Optional) Initial integration into our CI/CD pipeline to automatically run against pull requests.
Currently, testing our demo user journeys relies heavily on full evaluation runs or manual testing. To catch regressions early and ensure we don't break existing demos, we need a lightweight, deterministic verification mechanism that runs end-to-end user journeys without the cost, latency, or non-determinism of an LLM.
Proposed Solution
Introduce an automated smoke-test runner that reuses our existing eval files (similar to the patterns seen in PR #299).
Instead of passing prompts to an LLM to decide the next actions, the test runner will programmatically and deterministically trigger the tool calls defined in the eval files against a target website. This allows us to verify that the core journeys, tool integrations, and UI states function as expected.
Scope
Execute predefined tool calls sequence-by-sequence from eval files without LLM involvement.
Provide a fast feedback loop in CI/local development to confirm demos and core journeys are operational.
Leverage existing eval file formats so we don't need to maintain a separate set of test definitions.
Challenges & Considerations
The runner requires an active, running web application to test against. I am considering supporting two execution modes:
The smoke test runner automatically spins up the local app/server before executing tests and tears it down afterward.
The runner accepts a target URL (e.g., staging or a live deployed demo) to run the checks against.
Because these tests interact with running websites and real browser/network states, they may be prone to environmental flakiness. We should implement sensible retry logic, explicit wait states, or timeout handling to mitigate false positives.
Acceptance Criteria
A test runner script/utility is implemented that can parse existing eval files and execute tool sequences deterministically.
Support for configuring the target environment (e.g., via environment variables or CLI flags for local vs. deployed URLs).
Basic error reporting that clearly highlights which tool call or step failed during the journey.
Documentation updated with instructions on how to run the smoke tests locally and how to format eval files for compatibility.
(Optional) Initial integration into our CI/CD pipeline to automatically run against pull requests.