Document LLM judgment provider support, retries, and failure reporting - #12859
Document LLM judgment provider support, retries, and failure reporting#12859RexyZhang2001 wants to merge 4 commits into
Conversation
Signed-off-by: Rex Zhang <yichenzg@amazon.com>
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). |
| } | ||
| ``` | ||
|
|
||
| The judgment's `metadata` field also includes a summary of the run: |
There was a problem hiding this comment.
I think it might be easier for user to understand by showing the actual response of judgement in json format and explain each field instead of using table format.
There was a problem hiding this comment.
added in new commit
There was a problem hiding this comment.
Thanks for the update. However, I think below is enough here.
### Handling judgment failures
Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters).
For the remaining, you can merge in existing Managing judgment lists section.
There was a problem hiding this comment.
addressed this part in new commit
|
|
||
| For more information about this connector and other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). The following fields are specific to LLM-as-a-Judge: | ||
|
|
||
| | Field | Description | |
There was a problem hiding this comment.
These field explanations would be better placed in ML's connector documentation page. Here in the tutorial, since we're not covering every connector field anyway, it's more appropriate to highlight a few important ones in narrative form rather than as a table.
There was a problem hiding this comment.
added in new commit
Signed-off-by: Rex Zhang <yichenzg@amazon.com>
Signed-off-by: Rex Zhang <yichenzg@amazon.com>
| - `failedQueries`: The number of queries with at least one unrated document. | ||
| - `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. | ||
|
|
||
| A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document. |
There was a problem hiding this comment.
| A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document. |
There was a problem hiding this comment.
Please update metadata field here.
There was a problem hiding this comment.
Updated the example to LLM_JUDGMENT type instead of IMPORT_JUDGMENT. For IMPORT_JUDGMENT, we currently don't support showing metadata information and failedQueries as LLM_JUDGMENT does. Also removed all the redundant description, please check again
| } | ||
| ``` | ||
|
|
||
| The judgment's `metadata` field summarizes the run: |
There was a problem hiding this comment.
We can remove everything from this line onward — the parameter name is self-explanatory.
Signed-off-by: Rex Zhang <yichenzg@amazon.com>
Description
Updates the Search Relevance Workbench (SRW) LLM-as-a-Judge documentation to reflect three recent search-relevance and ml-commons changes:
judgments.md, and a new LLM judgment blueprints table tosupported-connectors.md.COMPLETEDwith some documents unrated. Adds a "Handling judgment failures" section documenting thefailuresarray, themetadatasummary fields (totalQueries,successfulQueries,failedQueries,lastFailureReason), and the Dashboards Judgment view's new Status column.client_configretry settings to the OpenAI connector example in the LLM-as-a-Judge tutorial and to the new Bedrock example injudgments.md.Files changed:
_search-plugins/search-relevance/judgments.md_tutorials/llm-as-a-judge-tutorial.md_ml-commons-plugin/remote-models/supported-connectors.mdIssues Resolved
opensearch-project/search-relevance#509
opensearch-project/search-relevance#510
Version
3.8
Checklist