Qa3

[Iamsdt]

This pull request introduces a comprehensive plan and initial implementation for improving Agentflow's testing and evaluation CLI commands and the evaluation report dashboard. The main changes include adding new CLI commands (agentflow test and agentflow eval), enhancing configuration and auto-discovery for evaluations, and redesigning the HTML report for better usability and visual clarity. Additional improvements include template updates, documentation enhancements, and minor fixes.

CLI Command Additions and Improvements

• Added a detailed plan for the new agentflow test and agentflow eval commands, specifying their behaviors, configuration, and integration with agentflow.json. The plan includes auto-discovery of evaluation files, improved report generation, and CLI options for customization. (EVAL_CLI_PLAN.md EVAL_CLI_PLAN.mdR1-R340)
• Implemented the evaluation key in the dev template's agentflow.json to support new evaluation features. (agentflow_cli/cli/templates/dev/agentflow.json agentflow_cli/cli/templates/dev/agentflow.jsonL5-R11)

Evaluation Protocol and Discovery

• Updated the evaluation file protocol to prioritize get_eval_set() for CLI discovery, with optional get_eval_config() for configuration, and run() as an escape hatch for custom evaluation flows. Improved warning messages for missing entry points. (agentflow_cli/cli/commands/eval.py [1] [2] [3]
• Updated the prod template's weather_agents_eval.py to export a get_eval_config() function, making it compatible with the new CLI auto-discovery. (agentflow_cli/cli/templates/prod/evals/weather_agents_eval.py agentflow_cli/cli/templates/prod/evals/weather_agents_eval.pyL1-R6)

HTML Report Redesign

• Outlined a redesign for the HTML evaluation report, including a new visual dashboard, dark mode, inline SVG charts, and improved layout for criteria and case results. The plan details splitting the template into focused files for maintainability and offline sharing. (EVAL_CLI_PLAN.md EVAL_CLI_PLAN.mdR1-R340)

Documentation and Template Enhancements

• Added references to new and updated documentation for unit testing and evaluation in the skill template's metadata and documentation lists. (agentflow_cli/cli/templates/skills/agent-skills/SKILL.md [1] [2]

Miscellaneous Fixes

• Excluded .ruff_cache from template copying to avoid unnecessary files in initialized projects. (agentflow_cli/cli/commands/init.py agentflow_cli/cli/commands/init.pyL19-R19)

These changes lay the groundwork for a more robust, user-friendly, and visually informative testing and evaluation workflow in Agentflow.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!