Skip to content

Document LLM judgment provider support, retries, and failure reporting #12859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RexyZhang2001 wants to merge 4 commits into
base: main
Choose a base branch
from
+85 −37
Open

Document LLM judgment provider support, retries, and failure reporting #12859

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
77137ce
Document LLM judgment provider support, retries, and failure reporting
RexyZhang2001 Jul 23, 2026
c48bac1
Address review feedback on judgment metadata and connector field docs
RexyZhang2001 Jul 23, 2026
8945647
Merge failure-reporting docs into Managing judgment lists
RexyZhang2001 Jul 23, 2026
2949508
Show LLM judgment failures in the main example response
RexyZhang2001 Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions _ml-commons-plugin/remote-models/supported-connectors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,18 @@
| [OpenAI](https://openai.com/) | [Text embedding models for batch inference](https://platform.openai.com/docs/guides/batch/overview#model-availability) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) |
| [Yandex Cloud](https://yandex.cloud/en) | [Text embedding models](https://yandex.cloud/en/docs/ai-studio/concepts/embeddings) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/yandexcloud_connector_embedding_legacy_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/standard_blueprints/yandexcloud_connector_embedding_standard_blueprint.md) |

## LLM judgment blueprints for Search Relevance Workbench

[Search Relevance Workbench (SRW)]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/using-search-relevance-workbench/) uses a chat model to generate relevance ratings with LLM-as-a-Judge. For an overview of these blueprints, see the [LLM judgment blueprints README](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/README.md). The following blueprints let you connect SRW to a chat model from any of these providers:

Check warning on line 68 in _ml-commons-plugin/remote-models/supported-connectors.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.

Raw Output:
{"message": "[OpenSearch.AcronymParentheses] 'README': Spell out acronyms the first time that you use them on a page and follow them with the acronym in parentheses. Subsequently, use the acronym alone.", "location": {"path": "_ml-commons-plugin/remote-models/supported-connectors.md", "range": {"start": {"line": 68, "column": 384}}}, "severity": "WARNING"}

| Provider | Blueprint |
|:---|:---|
| OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/openai_chat_blueprint.md) |
| Azure OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/azure_openai_chat_blueprint.md) |
| DeepSeek | [DeepSeek-V3, DeepSeek-R1](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/deepseek_chat_blueprint.md) |
| Ollama (and other local OpenAI-compatible servers) | [Qwen 3, Llama 3.3, Mistral, Phi-4, Gemma 3](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/ollama_chat_blueprint.md) |

Check failure on line 75 in _ml-commons-plugin/remote-models/supported-connectors.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Qwen. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Qwen. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_ml-commons-plugin/remote-models/supported-connectors.md", "range": {"start": {"line": 75, "column": 57}}}, "severity": "ERROR"}
| Google Gemini | [Gemini 2.5 Pro, Gemini 2.5 Flash](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/google_gemini_chat_blueprint.md) |
| Amazon Bedrock | [Anthropic Claude (native `invoke` API)](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/anthropic_claude_bedrock_blueprint.md) |
| Amazon Bedrock Converse | [Anthropic Claude, Amazon Nova, Meta Llama, Mistral, AI21, Cohere](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/bedrock_converse_blueprint.md) |

For more information about using these blueprints with SRW, see [Using different LLM providers]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#using-different-llm-providers).
82 changes: 52 additions & 30 deletions _search-plugins/search-relevance/judgments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ The following table lists the parameters for creating LLM-based judgments.
| `promptTemplate` | String | Optional. A custom prompt template for the LLM. Supports {% raw %}`{{queryText}}`{% endraw %} and {% raw %}`{{hits}}`{% endraw %} placeholders. If not provided, the default template is used. |
| `overwriteCache` | Boolean | Whether to overwrite existing cached judgments for the same query-document pairs. Default is `false` (reuse cached judgments). |

### Handling judgment failures

Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters).

### Custom prompt templates

You can customize the prompt template to focus on specific aspects of relevance:
Expand Down Expand Up @@ -120,36 +124,46 @@ PUT /_plugins/_search_relevance/judgments

### Using different LLM providers

You can adapt the connector configuration for other providers.
LLM-as-a-Judge works with any LLM provider that you can connect to through an [ML Commons connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/). For ready-to-use blueprints covering OpenAI, Azure OpenAI, DeepSeek, Ollama, Google Gemini, and Amazon Bedrock, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench).

#### Amazon Bedrock example

The following example creates a connector for Amazon Bedrock:
The following example creates a connector for an Anthropic Claude model on Amazon Bedrock:

```json
POST /_plugins/_ml/connectors/_create
{
"name": "Amazon Bedrock Connector",
"description": "Connector to Amazon Bedrock",
"version": "1",
"protocol": "aws_sigv4",
"parameters": {
"region": "us-east-1",
"service_name": "bedrock",
"model": "anthropic.claude-v2"
},
"credential": {
"access_key": "YOUR_ACCESS_KEY",
"secret_key": "YOUR_SECRET_KEY"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
"request_body": "{ \"prompt\": \"${parameters.messages}\", \"max_tokens_to_sample\": 300 }"
}
]
"name": "Amazon Bedrock Anthropic Claude",
"description": "Anthropic Claude via Bedrock for SRW LLM judgments",
"version": 1,
"protocol": "aws_sigv4",
"credential": {
"access_key": "<YOUR AWS ACCESS KEY>",
"secret_key": "<YOUR AWS SECRET KEY>"
},
"parameters": {
"region": "<YOUR AWS REGION>", // example: us-east-1
"service_name": "bedrock",
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 8000,
"model": "<INFERENCE_PROFILE_ID>" // example: us.anthropic.claude-haiku-4-5-20251001-v1:0
},
"client_config": {
"max_retry_times": 3,
"retry_backoff_policy": "exponential_full_jitter"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"headers": {
"content-type": "application/json"
},
"url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
"request_body": "{\"anthropic_version\":\"${parameters.anthropic_version}\",\"max_tokens\":${parameters.max_tokens},\"system\":\"${parameters.system_prompt}\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"${parameters.user_prompt}\"}]}]}",
"post_process_function": "def text = params.content[0].text; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'"
}
]
}
```
{% include copy-curl.html %}
Expand Down Expand Up @@ -339,10 +353,15 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade
"_source": {
"id": "b54f791a-3b02-49cb-a06c-46ab650b2ade",
"timestamp": "2025-06-11T06:07:23.766Z",
"name": "Imported Judgments",
"name": "LLM Judgments",
"status": "COMPLETED",
"type": "IMPORT_JUDGMENT",
"metadata": {},
"type": "LLM_JUDGMENT",
"metadata": {
"totalQueries": 2,
"successfulQueries": 1,
"failedQueries": 1,
"lastFailureReason": "Rate limit exceeded"
},
"judgmentRatings": [
{
"query": "red dress",
Expand All @@ -355,10 +374,6 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade
"rating": "2.000",
"docId": "B071S6LTJJ"
},
{
"rating": "2.000",
"docId": "B01IDSPDJI"
},
{
"rating": "0.000",
"docId": "B07QRCGL3G"
Expand All @@ -367,6 +382,11 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade
"rating": "1.000",
"docId": "B074V6Q1DR"
}
],
"failures": [
{
"docId": "B01IDSPDJI"
}
]
},
{
Expand Down Expand Up @@ -404,6 +424,8 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade

</details>

Unrated documents appear in each query's `failures` array. The run's overall counts appear in the judgment's `metadata` field.

### Deleting a judgment list

Delete a judgment list by its ID.
Expand Down
26 changes: 19 additions & 7 deletions _tutorials/llm-as-a-judge-tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@

### Step 1: Configure a model

First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock. Replace `YOUR_API_KEY` with your OpenAI API key:
First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock; for the full list of ready-to-use blueprints, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). Replace `<YOUR_API_KEY>` with your OpenAI API key:

```json
POST /_plugins/_ml/connectors/_create
Expand All @@ -46,33 +46,45 @@
"protocol": "http",
"parameters": {
"endpoint": "api.openai.com",
"model": "gpt-3.5-turbo"
"model": "gpt-4o-mini"
},
"credential": {
"openAI_key": "YOUR_API_KEY"
"openAI_key": "<YOUR_API_KEY>"
},
"client_config": {
"max_retry_times": 3,
"retry_backoff_policy": "exponential_full_jitter"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://api.openai.com/v1/chat/completions",
"url": "https://${parameters.endpoint}/v1/chat/completions",
"headers": {
"Authorization": "Bearer ${credential.openAI_key}",
"Content-Type": "application/json"
},
"request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages}, \"temperature\": 0 }"
"request_body": "{\"model\":\"${parameters.model}\",\"messages\":[{\"role\":\"system\",\"content\":\"${parameters.system_prompt}\"},{\"role\":\"user\",\"content\":\"${parameters.user_prompt}\"}]}",
"post_process_function": "def text = params.choices[0].message.content; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'"
}
]
}
```
{% include copy-curl.html %}

> The `client_config` block enables automatic retries with jittered backoff for temporary rate limits or server errors. If a document still fails after retries, it's reported instead of dropped; for more information, see [Failed and unrated documents]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#failed-and-unrated-documents).

Check failure on line 75 in _tutorials/llm-as-a-judge-tutorial.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: jittered. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: jittered. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_tutorials/llm-as-a-judge-tutorial.md", "range": {"start": {"line": 75, "column": 62}}}, "severity": "ERROR"}
>
> The `request_body` and `post_process_function` map the same neutral `system_prompt`, `user_prompt`, and `response` parameters into every provider's request and response format, so the judgment API call stays the same across providers. For information on how these fields work, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters).

Check failure on line 77 in _tutorials/llm-as-a-judge-tutorial.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.SubstitutionsError] Use 'for information about' instead of 'For information on'.

Raw Output:
{"message": "[OpenSearch.SubstitutionsError] Use 'for information about' instead of 'For information on'.", "location": {"path": "_tutorials/llm-as-a-judge-tutorial.md", "range": {"start": {"line": 77, "column": 240}}}, "severity": "ERROR"}
>
> For blueprints covering other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench).
{: .note}

Then register and deploy the model. Replace `{connector_id}` with the ID returned in the previous response:

```json
POST /_plugins/_ml/models/_register?deploy=true
{
"name": "openai_gpt-3.5-turbo",
"name": "openai_gpt-4o-mini",
"function_name": "remote",
"description": "External LLM model via OpenAI",
"connector_id": "{connector_id}"
Expand Down Expand Up @@ -160,7 +172,7 @@
PUT /_plugins/_search_relevance/judgments
{
"name": "LLM Judgment via OpenAI",
"description": "Uses GPT-3.5-turbo to evaluate product search results",
"description": "Uses GPT-4o mini to evaluate product search results",
"type": "LLM_JUDGMENT",
"modelId": "{model_id}",
"querySetId": "{query_set_id}",
Expand Down
Loading