From 77137ceb268395520338b0de3bc50a71aa968bc7 Mon Sep 17 00:00:00 2001 From: Rex Zhang Date: Wed, 22 Jul 2026 17:42:07 -0700 Subject: [PATCH 1/4] Document LLM judgment provider support, retries, and failure reporting Signed-off-by: Rex Zhang --- .../remote-models/supported-connectors.md | 14 +++ _search-plugins/search-relevance/judgments.md | 85 ++++++++++++++----- _tutorials/llm-as-a-judge-tutorial.md | 27 ++++-- 3 files changed, 96 insertions(+), 30 deletions(-) diff --git a/_ml-commons-plugin/remote-models/supported-connectors.md b/_ml-commons-plugin/remote-models/supported-connectors.md index 41c114a32bb..432570cd772 100644 --- a/_ml-commons-plugin/remote-models/supported-connectors.md +++ b/_ml-commons-plugin/remote-models/supported-connectors.md @@ -63,4 +63,18 @@ The following table provides a comprehensive list of connector blueprints availa | [OpenAI](https://openai.com/) | [Text embedding models for batch inference](https://platform.openai.com/docs/guides/batch/overview#model-availability) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) | | [Yandex Cloud](https://yandex.cloud/en) | [Text embedding models](https://yandex.cloud/en/docs/ai-studio/concepts/embeddings) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/yandexcloud_connector_embedding_legacy_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/standard_blueprints/yandexcloud_connector_embedding_standard_blueprint.md) | +## LLM judgment blueprints for Search Relevance Workbench +[Search Relevance Workbench (SRW)]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/using-search-relevance-workbench/) uses a chat model to generate relevance ratings with LLM-as-a-Judge. For an overview of these blueprints, see the [LLM judgment blueprints README](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/README.md). The following blueprints let you connect SRW to a chat model from any of these providers: + +| Provider | Blueprint | +|:---|:---| +| OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/openai_chat_blueprint.md) | +| Azure OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/azure_openai_chat_blueprint.md) | +| DeepSeek | [DeepSeek-V3, DeepSeek-R1](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/deepseek_chat_blueprint.md) | +| Ollama (and other local OpenAI-compatible servers) | [Qwen 3, Llama 3.3, Mistral, Phi-4, Gemma 3](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/ollama_chat_blueprint.md) | +| Google Gemini | [Gemini 2.5 Pro, Gemini 2.5 Flash](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/google_gemini_chat_blueprint.md) | +| Amazon Bedrock | [Anthropic Claude (native `invoke` API)](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/anthropic_claude_bedrock_blueprint.md) | +| Amazon Bedrock Converse | [Anthropic Claude, Amazon Nova, Meta Llama, Mistral, AI21, Cohere](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/bedrock_converse_blueprint.md) | + +For more information about using these blueprints with SRW, see [Using different LLM providers]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#using-different-llm-providers). diff --git a/_search-plugins/search-relevance/judgments.md b/_search-plugins/search-relevance/judgments.md index 4dae8030a40..0290f288631 100644 --- a/_search-plugins/search-relevance/judgments.md +++ b/_search-plugins/search-relevance/judgments.md @@ -82,6 +82,35 @@ The following table lists the parameters for creating LLM-based judgments. | `promptTemplate` | String | Optional. A custom prompt template for the LLM. Supports {% raw %}`{{queryText}}`{% endraw %} and {% raw %}`{{hits}}`{% endraw %} placeholders. If not provided, the default template is used. | | `overwriteCache` | Boolean | Whether to overwrite existing cached judgments for the same query-document pairs. Default is `false` (reuse cached judgments). | +### Handling judgment failures + +Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). + +If a document still doesn't receive a rating after retries, SRW keeps a record of it instead of dropping it silently. Each entry in `judgmentRatings` lists its successfully rated documents in `ratings` and any documents that were sent to the LLM but never rated in a `failures` array: + +```json +{ + "query": "laptop", + "ratings": [ + { "docId": "1", "rating": "1.0" } + ], + "failures": [ + { "docId": "6" } + ] +} +``` + +The judgment's `metadata` field also includes a summary of the run: + +| Field | Data type | Description | +| :--- | :--- | :--- | +| `totalQueries` | Integer | The total number of queries in the judgment run. | +| `successfulQueries` | Integer | The number of queries for which every document received a rating. | +| `failedQueries` | Integer | The number of queries with at least one document that didn't receive a rating. | +| `lastFailureReason` | String | The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. | + +A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated, so check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, making missing ratings visible without inspecting the raw judgment document. + ### Custom prompt templates You can customize the prompt template to focus on specific aspects of relevance: @@ -120,36 +149,46 @@ PUT /_plugins/_search_relevance/judgments ### Using different LLM providers -You can adapt the connector configuration for other providers. +LLM-as-a-Judge works with any LLM provider that you can connect to through an [ML Commons connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/). For ready-to-use blueprints covering OpenAI, Azure OpenAI, DeepSeek, Ollama, Google Gemini, and Amazon Bedrock, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). #### Amazon Bedrock example -The following example creates a connector for Amazon Bedrock: +The following example creates a connector for an Anthropic Claude model on Amazon Bedrock: ```json POST /_plugins/_ml/connectors/_create { - "name": "Amazon Bedrock Connector", - "description": "Connector to Amazon Bedrock", - "version": "1", - "protocol": "aws_sigv4", - "parameters": { - "region": "us-east-1", - "service_name": "bedrock", - "model": "anthropic.claude-v2" - }, - "credential": { - "access_key": "YOUR_ACCESS_KEY", - "secret_key": "YOUR_SECRET_KEY" - }, - "actions": [ - { - "action_type": "predict", - "method": "POST", - "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke", - "request_body": "{ \"prompt\": \"${parameters.messages}\", \"max_tokens_to_sample\": 300 }" - } - ] + "name": "Amazon Bedrock Anthropic Claude", + "description": "Anthropic Claude via Bedrock for SRW LLM judgments", + "version": 1, + "protocol": "aws_sigv4", + "credential": { + "access_key": "", + "secret_key": "" + }, + "parameters": { + "region": "", // example: us-east-1 + "service_name": "bedrock", + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 8000, + "model": "" // example: us.anthropic.claude-haiku-4-5-20251001-v1:0 + }, + "client_config": { + "max_retry_times": 3, + "retry_backoff_policy": "exponential_full_jitter" + }, + "actions": [ + { + "action_type": "predict", + "method": "POST", + "headers": { + "content-type": "application/json" + }, + "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke", + "request_body": "{\"anthropic_version\":\"${parameters.anthropic_version}\",\"max_tokens\":${parameters.max_tokens},\"system\":\"${parameters.system_prompt}\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"${parameters.user_prompt}\"}]}]}", + "post_process_function": "def text = params.content[0].text; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'" + } + ] } ``` {% include copy-curl.html %} diff --git a/_tutorials/llm-as-a-judge-tutorial.md b/_tutorials/llm-as-a-judge-tutorial.md index 84640c942d0..a577d1bca77 100644 --- a/_tutorials/llm-as-a-judge-tutorial.md +++ b/_tutorials/llm-as-a-judge-tutorial.md @@ -35,7 +35,7 @@ PUT /_cluster/settings ### Step 1: Configure a model -First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock. Replace `YOUR_API_KEY` with your OpenAI API key: +First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock; for the full list of ready-to-use blueprints, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). Replace `` with your OpenAI API key: ```json POST /_plugins/_ml/connectors/_create @@ -46,33 +46,46 @@ POST /_plugins/_ml/connectors/_create "protocol": "http", "parameters": { "endpoint": "api.openai.com", - "model": "gpt-3.5-turbo" + "model": "gpt-4o-mini" }, "credential": { - "openAI_key": "YOUR_API_KEY" + "openAI_key": "" + }, + "client_config": { + "max_retry_times": 3, + "retry_backoff_policy": "exponential_full_jitter" }, "actions": [ { "action_type": "predict", "method": "POST", - "url": "https://api.openai.com/v1/chat/completions", + "url": "https://${parameters.endpoint}/v1/chat/completions", "headers": { "Authorization": "Bearer ${credential.openAI_key}", "Content-Type": "application/json" }, - "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages}, \"temperature\": 0 }" + "request_body": "{\"model\":\"${parameters.model}\",\"messages\":[{\"role\":\"system\",\"content\":\"${parameters.system_prompt}\"},{\"role\":\"user\",\"content\":\"${parameters.user_prompt}\"}]}", + "post_process_function": "def text = params.choices[0].message.content; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'" } ] } ``` {% include copy-curl.html %} +For more information about this connector and other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). The following fields are specific to LLM-as-a-Judge: + +| Field | Description | +| :--- | :--- | +| `client_config` | Enables automatic retries with jittered backoff, so that a temporary rate limit or server error doesn't cause the request to fail outright. For more information, see [Handling judgment failures]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#handling-judgment-failures). | +| `request_body` | Maps the neutral `system_prompt` and `user_prompt` parameters that SRW sends on every call into the shape the provider's API expects. Every provider blueprint maps these same two parameters into its own format, so switching providers doesn't require changing how you call the judgment API. | +| `post_process_function` | Copies the model's answer from the provider's response shape into a neutral `response` field, which SRW reads to get the rating results. | + Then register and deploy the model. Replace `{connector_id}` with the ID returned in the previous response: ```json POST /_plugins/_ml/models/_register?deploy=true { - "name": "openai_gpt-3.5-turbo", + "name": "openai_gpt-4o-mini", "function_name": "remote", "description": "External LLM model via OpenAI", "connector_id": "{connector_id}" @@ -160,7 +173,7 @@ Create an LLM judgment that uses your deployed model to evaluate search results. PUT /_plugins/_search_relevance/judgments { "name": "LLM Judgment via OpenAI", - "description": "Uses GPT-3.5-turbo to evaluate product search results", + "description": "Uses GPT-4o mini to evaluate product search results", "type": "LLM_JUDGMENT", "modelId": "{model_id}", "querySetId": "{query_set_id}", From c48bac11bb551adc6656b2750e255199992536f2 Mon Sep 17 00:00:00 2001 From: Rex Zhang Date: Thu, 23 Jul 2026 11:47:52 -0700 Subject: [PATCH 2/4] Address review feedback on judgment metadata and connector field docs Signed-off-by: Rex Zhang --- _search-plugins/search-relevance/judgments.md | 21 +++++++++++++------ _tutorials/llm-as-a-judge-tutorial.md | 13 ++++++------ 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/_search-plugins/search-relevance/judgments.md b/_search-plugins/search-relevance/judgments.md index 0290f288631..db8ac287f9d 100644 --- a/_search-plugins/search-relevance/judgments.md +++ b/_search-plugins/search-relevance/judgments.md @@ -102,12 +102,21 @@ If a document still doesn't receive a rating after retries, SRW keeps a record o The judgment's `metadata` field also includes a summary of the run: -| Field | Data type | Description | -| :--- | :--- | :--- | -| `totalQueries` | Integer | The total number of queries in the judgment run. | -| `successfulQueries` | Integer | The number of queries for which every document received a rating. | -| `failedQueries` | Integer | The number of queries with at least one document that didn't receive a rating. | -| `lastFailureReason` | String | The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. | +```json +{ + "metadata": { + "totalQueries": 2, + "successfulQueries": 1, + "failedQueries": 1, + "lastFailureReason": "Rate limit exceeded" + } +} +``` + +- `totalQueries`: The total number of queries in the judgment run. +- `successfulQueries`: The number of queries for which every document received a rating. +- `failedQueries`: The number of queries with at least one document that didn't receive a rating. +- `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated, so check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, making missing ratings visible without inspecting the raw judgment document. diff --git a/_tutorials/llm-as-a-judge-tutorial.md b/_tutorials/llm-as-a-judge-tutorial.md index a577d1bca77..bc3cb68abdf 100644 --- a/_tutorials/llm-as-a-judge-tutorial.md +++ b/_tutorials/llm-as-a-judge-tutorial.md @@ -72,13 +72,12 @@ POST /_plugins/_ml/connectors/_create ``` {% include copy-curl.html %} -For more information about this connector and other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). The following fields are specific to LLM-as-a-Judge: - -| Field | Description | -| :--- | :--- | -| `client_config` | Enables automatic retries with jittered backoff, so that a temporary rate limit or server error doesn't cause the request to fail outright. For more information, see [Handling judgment failures]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#handling-judgment-failures). | -| `request_body` | Maps the neutral `system_prompt` and `user_prompt` parameters that SRW sends on every call into the shape the provider's API expects. Every provider blueprint maps these same two parameters into its own format, so switching providers doesn't require changing how you call the judgment API. | -| `post_process_function` | Copies the model's answer from the provider's response shape into a neutral `response` field, which SRW reads to get the rating results. | +> The `client_config` block enables automatic retries with jittered backoff for temporary rate limits or server errors. For information on how SRW reports judgment failures, see [Handling judgment failures]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#handling-judgment-failures). +> +> The `request_body` and `post_process_function` map the same neutral `system_prompt`, `user_prompt`, and `response` parameters into every provider's request and response format, so the judgment API call stays the same across providers. For information on how these fields work, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). +> +> For blueprints covering other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). +{: .note} Then register and deploy the model. Replace `{connector_id}` with the ID returned in the previous response: From 89456471a372f1fcf19ceadcfe222869a4780507 Mon Sep 17 00:00:00 2001 From: Rex Zhang Date: Thu, 23 Jul 2026 14:50:15 -0700 Subject: [PATCH 3/4] Merge failure-reporting docs into Managing judgment lists Signed-off-by: Rex Zhang --- _search-plugins/search-relevance/judgments.md | 70 ++++++++++--------- _tutorials/llm-as-a-judge-tutorial.md | 2 +- 2 files changed, 37 insertions(+), 35 deletions(-) diff --git a/_search-plugins/search-relevance/judgments.md b/_search-plugins/search-relevance/judgments.md index db8ac287f9d..a5652d40906 100644 --- a/_search-plugins/search-relevance/judgments.md +++ b/_search-plugins/search-relevance/judgments.md @@ -86,40 +86,6 @@ The following table lists the parameters for creating LLM-based judgments. Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). -If a document still doesn't receive a rating after retries, SRW keeps a record of it instead of dropping it silently. Each entry in `judgmentRatings` lists its successfully rated documents in `ratings` and any documents that were sent to the LLM but never rated in a `failures` array: - -```json -{ - "query": "laptop", - "ratings": [ - { "docId": "1", "rating": "1.0" } - ], - "failures": [ - { "docId": "6" } - ] -} -``` - -The judgment's `metadata` field also includes a summary of the run: - -```json -{ - "metadata": { - "totalQueries": 2, - "successfulQueries": 1, - "failedQueries": 1, - "lastFailureReason": "Rate limit exceeded" - } -} -``` - -- `totalQueries`: The total number of queries in the judgment run. -- `successfulQueries`: The number of queries for which every document received a rating. -- `failedQueries`: The number of queries with at least one document that didn't receive a rating. -- `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. - -A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated, so check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, making missing ratings visible without inspecting the raw judgment document. - ### Custom prompt templates You can customize the prompt template to focus on specific aspects of relevance: @@ -452,6 +418,42 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade +### Failed and unrated documents + +When you view or search for an LLM judgment, a document can appear unrated if it still fails after any connector-level retries. Instead of being dropped, the document appears in a `failures` array alongside the `ratings` array for its query: + +```json +{ + "query": "wireless headphones", + "ratings": [ + { "docId": "B003VJPPI4", "rating": "0.900" } + ], + "failures": [ + { "docId": "B0091B7RUI" } + ] +} +``` + +The judgment's `metadata` field summarizes the run: + +```json +{ + "metadata": { + "totalQueries": 2, + "successfulQueries": 1, + "failedQueries": 1, + "lastFailureReason": "Rate limit exceeded" + } +} +``` + +- `totalQueries`: The total number of queries in the judgment run. +- `successfulQueries`: The number of queries for which every document received a rating. +- `failedQueries`: The number of queries with at least one unrated document. +- `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. + +A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document. + ### Deleting a judgment list Delete a judgment list by its ID. diff --git a/_tutorials/llm-as-a-judge-tutorial.md b/_tutorials/llm-as-a-judge-tutorial.md index bc3cb68abdf..cb4f8f87c52 100644 --- a/_tutorials/llm-as-a-judge-tutorial.md +++ b/_tutorials/llm-as-a-judge-tutorial.md @@ -72,7 +72,7 @@ POST /_plugins/_ml/connectors/_create ``` {% include copy-curl.html %} -> The `client_config` block enables automatic retries with jittered backoff for temporary rate limits or server errors. For information on how SRW reports judgment failures, see [Handling judgment failures]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#handling-judgment-failures). +> The `client_config` block enables automatic retries with jittered backoff for temporary rate limits or server errors. If a document still fails after retries, it's reported instead of dropped; for more information, see [Failed and unrated documents]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#failed-and-unrated-documents). > > The `request_body` and `post_process_function` map the same neutral `system_prompt`, `user_prompt`, and `response` parameters into every provider's request and response format, so the judgment API call stays the same across providers. For information on how these fields work, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). > From 2949508c4e10cb9671a37015d067241533f39630 Mon Sep 17 00:00:00 2001 From: Rex Zhang Date: Fri, 24 Jul 2026 14:11:21 -0700 Subject: [PATCH 4/4] Show LLM judgment failures in the main example response Signed-off-by: Rex Zhang --- _search-plugins/search-relevance/judgments.md | 56 +++++-------------- 1 file changed, 14 insertions(+), 42 deletions(-) diff --git a/_search-plugins/search-relevance/judgments.md b/_search-plugins/search-relevance/judgments.md index a5652d40906..ca405f12840 100644 --- a/_search-plugins/search-relevance/judgments.md +++ b/_search-plugins/search-relevance/judgments.md @@ -353,10 +353,15 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "_source": { "id": "b54f791a-3b02-49cb-a06c-46ab650b2ade", "timestamp": "2025-06-11T06:07:23.766Z", - "name": "Imported Judgments", + "name": "LLM Judgments", "status": "COMPLETED", - "type": "IMPORT_JUDGMENT", - "metadata": {}, + "type": "LLM_JUDGMENT", + "metadata": { + "totalQueries": 2, + "successfulQueries": 1, + "failedQueries": 1, + "lastFailureReason": "Rate limit exceeded" + }, "judgmentRatings": [ { "query": "red dress", @@ -369,10 +374,6 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "rating": "2.000", "docId": "B071S6LTJJ" }, - { - "rating": "2.000", - "docId": "B01IDSPDJI" - }, { "rating": "0.000", "docId": "B07QRCGL3G" @@ -381,6 +382,11 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "rating": "1.000", "docId": "B074V6Q1DR" } + ], + "failures": [ + { + "docId": "B01IDSPDJI" + } ] }, { @@ -418,41 +424,7 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade -### Failed and unrated documents - -When you view or search for an LLM judgment, a document can appear unrated if it still fails after any connector-level retries. Instead of being dropped, the document appears in a `failures` array alongside the `ratings` array for its query: - -```json -{ - "query": "wireless headphones", - "ratings": [ - { "docId": "B003VJPPI4", "rating": "0.900" } - ], - "failures": [ - { "docId": "B0091B7RUI" } - ] -} -``` - -The judgment's `metadata` field summarizes the run: - -```json -{ - "metadata": { - "totalQueries": 2, - "successfulQueries": 1, - "failedQueries": 1, - "lastFailureReason": "Rate limit exceeded" - } -} -``` - -- `totalQueries`: The total number of queries in the judgment run. -- `successfulQueries`: The number of queries for which every document received a rating. -- `failedQueries`: The number of queries with at least one unrated document. -- `lastFailureReason`: The error message from the most recent failure. Included only when `failedQueries` is greater than `0`. - -A judgment can finish with a `status` of `COMPLETED` even if some documents weren't rated. Check `metadata.failedQueries` or each query's `failures` array to confirm that every document received a rating. In OpenSearch Dashboards, the Judgment view's ratings table includes a **Status** column, labeled **Rated** or **Failed** for each document, so missing ratings are visible without inspecting the raw judgment document. +Unrated documents appear in each query's `failures` array. The run's overall counts appear in the judgment's `metadata` field. ### Deleting a judgment list