diff --git a/_ml-commons-plugin/remote-models/supported-connectors.md b/_ml-commons-plugin/remote-models/supported-connectors.md index 41c114a32b..432570cd77 100644 --- a/_ml-commons-plugin/remote-models/supported-connectors.md +++ b/_ml-commons-plugin/remote-models/supported-connectors.md @@ -63,4 +63,18 @@ The following table provides a comprehensive list of connector blueprints availa | [OpenAI](https://openai.com/) | [Text embedding models for batch inference](https://platform.openai.com/docs/guides/batch/overview#model-availability) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/batch_inference_openAI_connector_blueprint.md) | | [Yandex Cloud](https://yandex.cloud/en) | [Text embedding models](https://yandex.cloud/en/docs/ai-studio/concepts/embeddings) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/yandexcloud_connector_embedding_legacy_blueprint.md) | [Blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/standard_blueprints/yandexcloud_connector_embedding_standard_blueprint.md) | +## LLM judgment blueprints for Search Relevance Workbench +[Search Relevance Workbench (SRW)]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/using-search-relevance-workbench/) uses a chat model to generate relevance ratings with LLM-as-a-Judge. For an overview of these blueprints, see the [LLM judgment blueprints README](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/README.md). The following blueprints let you connect SRW to a chat model from any of these providers: + +| Provider | Blueprint | +|:---|:---| +| OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/openai_chat_blueprint.md) | +| Azure OpenAI | [GPT-5, GPT-4.1, GPT-4o](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/azure_openai_chat_blueprint.md) | +| DeepSeek | [DeepSeek-V3, DeepSeek-R1](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/deepseek_chat_blueprint.md) | +| Ollama (and other local OpenAI-compatible servers) | [Qwen 3, Llama 3.3, Mistral, Phi-4, Gemma 3](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/ollama_chat_blueprint.md) | +| Google Gemini | [Gemini 2.5 Pro, Gemini 2.5 Flash](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/google_gemini_chat_blueprint.md) | +| Amazon Bedrock | [Anthropic Claude (native `invoke` API)](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/anthropic_claude_bedrock_blueprint.md) | +| Amazon Bedrock Converse | [Anthropic Claude, Amazon Nova, Meta Llama, Mistral, AI21, Cohere](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/llm_judgment/bedrock_converse_blueprint.md) | + +For more information about using these blueprints with SRW, see [Using different LLM providers]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#using-different-llm-providers). diff --git a/_search-plugins/search-relevance/judgments.md b/_search-plugins/search-relevance/judgments.md index 4dae8030a4..ca405f1284 100644 --- a/_search-plugins/search-relevance/judgments.md +++ b/_search-plugins/search-relevance/judgments.md @@ -82,6 +82,10 @@ The following table lists the parameters for creating LLM-based judgments. | `promptTemplate` | String | Optional. A custom prompt template for the LLM. Supports {% raw %}`{{queryText}}`{% endraw %} and {% raw %}`{{hits}}`{% endraw %} placeholders. If not provided, the default template is used. | | `overwriteCache` | Boolean | Whether to overwrite existing cached judgments for the same query-document pairs. Default is `false` (reuse cached judgments). | +### Handling judgment failures + +Generating a judgment sends one LLM request for every query-document pair. Occasional failures are expected at scale: for example, the provider might throttle a request or a request might time out. To make these failures self-healing, configure retries on the connector you use for the judgment, using the connector's `client_config` settings (`max_retry_times`, `retry_backoff_policy`, and related options). For more information, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). + ### Custom prompt templates You can customize the prompt template to focus on specific aspects of relevance: @@ -120,36 +124,46 @@ PUT /_plugins/_search_relevance/judgments ### Using different LLM providers -You can adapt the connector configuration for other providers. +LLM-as-a-Judge works with any LLM provider that you can connect to through an [ML Commons connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/). For ready-to-use blueprints covering OpenAI, Azure OpenAI, DeepSeek, Ollama, Google Gemini, and Amazon Bedrock, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). #### Amazon Bedrock example -The following example creates a connector for Amazon Bedrock: +The following example creates a connector for an Anthropic Claude model on Amazon Bedrock: ```json POST /_plugins/_ml/connectors/_create { - "name": "Amazon Bedrock Connector", - "description": "Connector to Amazon Bedrock", - "version": "1", - "protocol": "aws_sigv4", - "parameters": { - "region": "us-east-1", - "service_name": "bedrock", - "model": "anthropic.claude-v2" - }, - "credential": { - "access_key": "YOUR_ACCESS_KEY", - "secret_key": "YOUR_SECRET_KEY" - }, - "actions": [ - { - "action_type": "predict", - "method": "POST", - "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke", - "request_body": "{ \"prompt\": \"${parameters.messages}\", \"max_tokens_to_sample\": 300 }" - } - ] + "name": "Amazon Bedrock Anthropic Claude", + "description": "Anthropic Claude via Bedrock for SRW LLM judgments", + "version": 1, + "protocol": "aws_sigv4", + "credential": { + "access_key": "", + "secret_key": "" + }, + "parameters": { + "region": "", // example: us-east-1 + "service_name": "bedrock", + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 8000, + "model": "" // example: us.anthropic.claude-haiku-4-5-20251001-v1:0 + }, + "client_config": { + "max_retry_times": 3, + "retry_backoff_policy": "exponential_full_jitter" + }, + "actions": [ + { + "action_type": "predict", + "method": "POST", + "headers": { + "content-type": "application/json" + }, + "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke", + "request_body": "{\"anthropic_version\":\"${parameters.anthropic_version}\",\"max_tokens\":${parameters.max_tokens},\"system\":\"${parameters.system_prompt}\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"${parameters.user_prompt}\"}]}]}", + "post_process_function": "def text = params.content[0].text; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'" + } + ] } ``` {% include copy-curl.html %} @@ -339,10 +353,15 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "_source": { "id": "b54f791a-3b02-49cb-a06c-46ab650b2ade", "timestamp": "2025-06-11T06:07:23.766Z", - "name": "Imported Judgments", + "name": "LLM Judgments", "status": "COMPLETED", - "type": "IMPORT_JUDGMENT", - "metadata": {}, + "type": "LLM_JUDGMENT", + "metadata": { + "totalQueries": 2, + "successfulQueries": 1, + "failedQueries": 1, + "lastFailureReason": "Rate limit exceeded" + }, "judgmentRatings": [ { "query": "red dress", @@ -355,10 +374,6 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "rating": "2.000", "docId": "B071S6LTJJ" }, - { - "rating": "2.000", - "docId": "B01IDSPDJI" - }, { "rating": "0.000", "docId": "B07QRCGL3G" @@ -367,6 +382,11 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade "rating": "1.000", "docId": "B074V6Q1DR" } + ], + "failures": [ + { + "docId": "B01IDSPDJI" + } ] }, { @@ -404,6 +424,8 @@ GET _plugins/_search_relevance/judgments/b54f791a-3b02-49cb-a06c-46ab650b2ade +Unrated documents appear in each query's `failures` array. The run's overall counts appear in the judgment's `metadata` field. + ### Deleting a judgment list Delete a judgment list by its ID. diff --git a/_tutorials/llm-as-a-judge-tutorial.md b/_tutorials/llm-as-a-judge-tutorial.md index 84640c942d..cb4f8f87c5 100644 --- a/_tutorials/llm-as-a-judge-tutorial.md +++ b/_tutorials/llm-as-a-judge-tutorial.md @@ -35,7 +35,7 @@ PUT /_cluster/settings ### Step 1: Configure a model -First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock. Replace `YOUR_API_KEY` with your OpenAI API key: +First, create a connector to an externally hosted LLM. This tutorial uses OpenAI, but you can adapt it for other providers such as Amazon Bedrock; for the full list of ready-to-use blueprints, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). Replace `` with your OpenAI API key: ```json POST /_plugins/_ml/connectors/_create @@ -46,33 +46,45 @@ POST /_plugins/_ml/connectors/_create "protocol": "http", "parameters": { "endpoint": "api.openai.com", - "model": "gpt-3.5-turbo" + "model": "gpt-4o-mini" }, "credential": { - "openAI_key": "YOUR_API_KEY" + "openAI_key": "" + }, + "client_config": { + "max_retry_times": 3, + "retry_backoff_policy": "exponential_full_jitter" }, "actions": [ { "action_type": "predict", "method": "POST", - "url": "https://api.openai.com/v1/chat/completions", + "url": "https://${parameters.endpoint}/v1/chat/completions", "headers": { "Authorization": "Bearer ${credential.openAI_key}", "Content-Type": "application/json" }, - "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages}, \"temperature\": 0 }" + "request_body": "{\"model\":\"${parameters.model}\",\"messages\":[{\"role\":\"system\",\"content\":\"${parameters.system_prompt}\"},{\"role\":\"user\",\"content\":\"${parameters.user_prompt}\"}]}", + "post_process_function": "def text = params.choices[0].message.content; return '{\"name\":\"response\",\"dataAsMap\":{\"response\":\"' + escape(text) + '\"}}'" } ] } ``` {% include copy-curl.html %} +> The `client_config` block enables automatic retries with jittered backoff for temporary rate limits or server errors. If a document still fails after retries, it's reported instead of dropped; for more information, see [Failed and unrated documents]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/judgments/#failed-and-unrated-documents). +> +> The `request_body` and `post_process_function` map the same neutral `system_prompt`, `user_prompt`, and `response` parameters into every provider's request and response format, so the judgment API call stays the same across providers. For information on how these fields work, see [Connector blueprints]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/blueprints/#configuration-parameters). +> +> For blueprints covering other providers, see [Supported connectors]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/supported-connectors/#llm-judgment-blueprints-for-search-relevance-workbench). +{: .note} + Then register and deploy the model. Replace `{connector_id}` with the ID returned in the previous response: ```json POST /_plugins/_ml/models/_register?deploy=true { - "name": "openai_gpt-3.5-turbo", + "name": "openai_gpt-4o-mini", "function_name": "remote", "description": "External LLM model via OpenAI", "connector_id": "{connector_id}" @@ -160,7 +172,7 @@ Create an LLM judgment that uses your deployed model to evaluate search results. PUT /_plugins/_search_relevance/judgments { "name": "LLM Judgment via OpenAI", - "description": "Uses GPT-3.5-turbo to evaluate product search results", + "description": "Uses GPT-4o mini to evaluate product search results", "type": "LLM_JUDGMENT", "modelId": "{model_id}", "querySetId": "{query_set_id}",