Skip to content

feat(evaluation): unify validators with azureml-assets#47526

Draft
m7md7sien wants to merge 4 commits into
mainfrom
mohessie/update_eval_validators
Draft

feat(evaluation): unify validators with azureml-assets#47526
m7md7sien wants to merge 4 commits into
mainfrom
mohessie/update_eval_validators

Conversation

@m7md7sien

@m7md7sien m7md7sien commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Description

  • add DEVELOPER role, EvaluationLevel, MessagesOrQueryResponseInputValidator + level utils
  • support actions/expected_actions aliases in TaskNavigationEfficiencyValidator
  • align check_for_unsupported_tools flags in tool_call/input/output evaluators

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

- add DEVELOPER role, EvaluationLevel, MessagesOrQueryResponseInputValidator + level utils
- support actions/expected_actions aliases in TaskNavigationEfficiencyValidator
- align check_for_unsupported_tools flags in tool_call/input/output evaluators
@github-actions github-actions Bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Jun 16, 2026
@m7md7sien m7md7sien requested a review from Copilot June 16, 2026 17:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates azure-ai-evaluation’s internal evaluator input validation layer to better align with azureml-assets naming and behavior, while expanding supported conversation roles and adding utilities for evaluation-level handling.

Changes:

  • Added DEVELOPER message role support and introduced EvaluationLevel plus evaluation-level utility helpers.
  • Added MessagesOrQueryResponseInputValidator to support both multi-turn (messages) and single-turn (query/response) input shapes.
  • Added actions/expected_actions aliases for task navigation efficiency inputs, and aligned check_for_unsupported_tools behavior across tool-related evaluators/validators.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_output_utilization/_tool_output_utilization.py Enables unsupported-tool checking for tool output utilization inputs.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_input_accuracy/_tool_input_accuracy.py Adjusts unsupported-tool checking behavior for tool input accuracy validation.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy.py Adjusts unsupported-tool checking behavior for tool call accuracy validation.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_validation_constants.py Adds DEVELOPER role and introduces the EvaluationLevel enum.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_task_navigation_efficiency_validator.py Adds normalization to accept actions/expected_actions aliases.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_messages_or_query_response_validator.py New validator supporting either messages or query/response input formats.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_evaluation_level_utils.py New helper utilities for resolving evaluation levels and reshaping message inputs.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_conversation_validator.py Adds developer-role validation handling and minor error-message cleanup.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/init.py Exposes new enums/validators/utilities from the validators package.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI and others added 2 commits June 16, 2026 17:28
Co-authored-by: m7md7sien <16615690+m7md7sien@users.noreply.github.com>
…luated

Co-authored-by: m7md7sien <16615690+m7md7sien@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants