Skip to content

Document LLM judgment provider support, retries, and failure reporting - #12859

Open
RexyZhang2001 wants to merge 4 commits into
opensearch-project:mainfrom
RexyZhang2001:doc/srw-llm-judgment-failures
Open

Document LLM judgment provider support, retries, and failure reporting#12859
RexyZhang2001 wants to merge 4 commits into
opensearch-project:mainfrom
RexyZhang2001:doc/srw-llm-judgment-failures

Conversation

@RexyZhang2001

Copy link
Copy Markdown
Contributor

Description

Updates the Search Relevance Workbench (SRW) LLM-as-a-Judge documentation to reflect three recent search-relevance and ml-commons changes:

  • Provider-neutral LLM judgment generation: LLM-as-a-Judge now works with any provider through an ML Commons connector blueprint, not just OpenAI. Adds a "Using different LLM providers" section and a Bedrock/Claude connector example to judgments.md, and a new LLM judgment blueprints table to supported-connectors.md.
  • Judgment failure reporting: A judgment can now finish as COMPLETED with some documents unrated. Adds a "Handling judgment failures" section documenting the failures array, the metadata summary fields (totalQueries, successfulQueries, failedQueries, lastFailureReason), and the Dashboards Judgment view's new Status column.
  • Connector-level retries: Adds client_config retry settings to the OpenAI connector example in the LLM-as-a-Judge tutorial and to the new Bedrock example in judgments.md.

Files changed:

  • _search-plugins/search-relevance/judgments.md
  • _tutorials/llm-as-a-judge-tutorial.md
  • _ml-commons-plugin/remote-models/supported-connectors.md

Issues Resolved

opensearch-project/search-relevance#509
opensearch-project/search-relevance#510

Version

3.8

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.

Signed-off-by: Rex Zhang <yichenzg@amazon.com>
@github-actions

Copy link
Copy Markdown

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference).

}
```

The judgment's `metadata` field also includes a summary of the run:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be easier for user to understand by showing the actual response of judgement in json format and explain each field instead of using table format.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in new commit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. However, I think below is enough here.

### Handling judgment failures

Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters).

For the remaining, you can merge in existing Managing judgment lists section.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed this part in new commit

Comment thread _tutorials/llm-as-a-judge-tutorial.md Outdated

For more information about this connector and other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). The following fields are specific to LLM-as-a-Judge:

| Field | Description |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These field explanations would be better placed in ML's connector documentation page. Here in the tutorial, since we're not covering every connector field anyway, it's more appropriate to highlight a few important ones in narrative form rather than as a table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added in new commit

Signed-off-by: Rex Zhang <yichenzg@amazon.com>
Signed-off-by: Rex Zhang <yichenzg@amazon.com>
@kolchfa-aws kolchfa-aws added release-notes PR: Include this PR in the automated release notes v3.8.0 labels Jul 24, 2026
- `failedQueries`: The number of queries with at least one unrated document.
- `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`.

A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update metadata field here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the example to LLM_JUDGMENT type instead of IMPORT_JUDGMENT. For IMPORT_JUDGMENT, we currently don't support showing metadata information and failedQueries as LLM_JUDGMENT does. Also removed all the redundant description, please check again

}
```

The judgment's `metadata` field summarizes the run:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove everything from this line onward — the parameter name is self-explanatory.

Signed-off-by: Rex Zhang <yichenzg@amazon.com>

@heemin32 heemin32 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@kolchfa-aws kolchfa-aws added Doc review PR: Doc review in progress and removed Tech review PR: Tech review in progress labels Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Doc review PR: Doc review in progress release-notes PR: Include this PR in the automated release notes v3.8.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants