diff --git a/_api-reference/grpc-apis/execute-agent-stream.md b/_api-reference/grpc-apis/execute-agent-stream.md new file mode 100644 index 00000000000..d5d08d93216 --- /dev/null +++ b/_api-reference/grpc-apis/execute-agent-stream.md @@ -0,0 +1,193 @@ +--- +layout: default +title: Execute agent stream (gRPC) +parent: gRPC APIs +nav_order: 50 +--- + +# Execute Agent Stream API (gRPC) +**Introduced 3.8** +{: .label .label-purple } + +The gRPC Execute Agent Stream API provides a binary interface for streaming the execution of an agent using protocol buffers over gRPC. The server streams response chunks to the client as the agent generates them, so the client can begin processing output before execution completes. Use streaming for conversational agents that produce token-by-token output from large language models (LLMs). + +You can stream agent execution over either REST or gRPC. Both transports return the same incrementally generated output, so choose the one that best fits your client: + +- **REST streaming** uses server-sent events (SSE) over HTTP, which browsers, standard HTTP clients, and command line tools such as cURL support directly. This is an experimental feature and is not recommended for use in a production environment. For more information, see [Execute Agent Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/). +- **gRPC streaming** uses protocol buffers over HTTP/2. This transport provides lower serialization overhead and smaller payloads, native server streaming semantics with HTTP/2 flow control and connection multiplexing, and a strongly typed schema from which you can generate clients in any [gRPC-supported language](https://grpc.io/docs/languages/). + +Streaming agent execution is supported for agents that use the following externally hosted models: + +- [OpenAI Chat Completion](https://platform.openai.com/docs/api-reference/completions) +- [Amazon Bedrock Converse Stream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) + +## Prerequisites + +Before using the gRPC Execute Agent Stream API, ensure that you have fulfilled the following prerequisites: + +- Enable gRPC transport on the cluster. For more information, see [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#how-to-use-grpc-apis). +- Obtain the ML Commons protobufs on the client side. For ways to obtain the protobufs, see [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#how-to-use-grpc-apis). +- Register an agent that uses a supported streaming model. For agent and connector configuration, see [Execute Agent Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/#step-2-register-a-compatible-externally-hosted-model). + +## gRPC service and method + +The gRPC Execute Agent Stream API resides in the [`MLService`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/services/ml_service.proto#L22) service. + +You can submit streaming agent execution requests by invoking the [`ExecuteAgentStream`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/services/ml_service.proto#L27) method within the `MLService`. The method takes an [`MlExecuteAgentStreamRequest`](#mlexecuteagentstreamrequest-fields) and returns a stream of [`PredictResponse`](#response-fields) messages. + +`ExecuteAgentStream` is a server streaming remote procedure call (RPC): the client sends a single request, and the server returns a sequence of response messages. The final message sets `is_last` to `true`, and the server then closes the stream. +{: .note} + +## Request fields + +The gRPC Execute Agent Stream API supports the following request fields. + +### MlExecuteAgentStreamRequest fields + +The [`MlExecuteAgentStreamRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3396) message accepts the following fields. + +| Field | Protobuf type | Required | Description | +| :---- | :---- | :---- | :---- | +| `agent_id` | `string` | Required | The ID of the agent to execute. | +| `ml_execute_agent_stream_request_body` | [`MLExecuteAgentStreamRequestBody`](#mlexecuteagentstreamrequestbody-fields) | Required | The request payload containing the execution parameters. | + + +### MLExecuteAgentStreamRequestBody fields + + +The [`MLExecuteAgentStreamRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3318) message accepts the following fields. + +| Field | Protobuf type | Required | Description | +| :---- | :---- | :---- | :---- | +| `parameters` | [`Parameters`](#parameters-fields) | Required | The input parameters passed to the agent. | + +### Parameters fields + +For agent execution, the [`Parameters`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3356) message accepts the following field. + +| Field | Protobuf type | Description | +| :---- | :---- | :---- | +| `question` | `string` | The input question sent to the agent. | + +## Response fields + +The server streams a sequence of [`PredictResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3367) messages. Each message carries one chunk of the generated output and provides the following fields. + +| Field | Protobuf type | Description | +| :---- | :---- | :---- | +| `inference_results` | `repeated InferenceResults` | The inference results for the chunk. | +| `inference_results.output` | `repeated Output` | The output objects for each inference result. | +| `inference_results.output.name` | `string` | The name of the output field (typically, `response`). | +| `inference_results.output.result` | `string` | The values of the `memory_id` and `parent_interaction_id` fields. | +| `inference_results.output.data_as_map` | `DataAsMap` | The response content and metadata for the chunk. | +| `inference_results.output.data_as_map.content` | `string` | The text content of the chunk. Concatenate the `content` values across chunks to reconstruct the full response. | +| `inference_results.output.data_as_map.is_last` | `bool` | Whether this is the final chunk in the stream. When `true`, no further messages are sent. | + +## Example request + +The following example shows the JSON representation of the gRPC request message. Replace the `agent_id` and `question` with values that match your agent configuration: + +```json +{ + "agent_id": "your_agent_id", + "ml_execute_agent_stream_request_body": { + "parameters": { + "question": "List indices in my cluster" + } + } +} +``` +{% include copy.html %} + +The following example shows a Java gRPC client that streams the execution of a conversational agent. Replace the agent ID and question with values that match your agent configuration: + +```java +import org.opensearch.protobufs.*; +import org.opensearch.protobufs.services.MLServiceGrpc; +import io.grpc.ManagedChannel; +import io.grpc.ManagedChannelBuilder; + +import java.util.Iterator; + +public class ExecuteAgentStreamClient { + public static void main(String[] args) { + ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 9400) + .usePlaintext() + .build(); + + // Create a gRPC stub for ML operations + MLServiceGrpc.MLServiceBlockingStub mlStub = MLServiceGrpc.newBlockingStub(channel); + + // Build the request parameters for the agent + Parameters parameters = Parameters.newBuilder() + .setQuestion("List indices in my cluster") + .build(); + + // Create the streaming agent execution request + MlExecuteAgentStreamRequest request = MlExecuteAgentStreamRequest.newBuilder() + .setAgentId("your_agent_id") + .setMlExecuteAgentStreamRequestBody(MLExecuteAgentStreamRequestBody.newBuilder() + .setParameters(parameters) + .build()) + .build(); + + // Execute the request and read the streamed response + try { + Iterator responses = mlStub.executeAgentStream(request); + while (responses.hasNext()) { + PredictResponse response = responses.next(); + for (InferenceResults results : response.getInferenceResultsList()) { + for (Output output : results.getOutputList()) { + DataAsMap chunk = output.getDataAsMap(); + System.out.print(chunk.getContent()); + if (chunk.getIsLast()) { + System.out.println("\n[stream complete]"); + } + } + } + } + } catch (io.grpc.StatusRuntimeException e) { + System.err.println("gRPC execute agent stream request failed with status: " + e.getStatus()); + System.err.println("Error message: " + e.getMessage()); + } + + channel.shutdown(); + } +} +``` +{% include copy.html %} + +## Example response + +The server returns a sequence of `PredictResponse` messages. Each message carries a chunk of generated text in the `content` field, and the final message sets `isLast` to `true`. The following example shows the JSON representation of a streamed chunk: + +```json +{ + "inferenceResults": [ + { + "output": [ + { + "name": "memory_id", + "result": "6CMnkJ8BLGHoqtB13ipp" + }, + { + "name": "parent_interaction_id", + "result": "6SMnkJ8BLGHoqtB13irB" + }, + { + "name": "response", + "dataAsMap": { + "content": "Hello", + "isLast": false + } + } + ] + } + ] +} +``` + +## Related documentation + +- [Execute Agent Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/) -- The REST equivalent for streaming agent execution +- [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/) -- gRPC transport configuration and client requirements diff --git a/_api-reference/grpc-apis/index.md b/_api-reference/grpc-apis/index.md index 924302ef5d8..145a9846a1c 100644 --- a/_api-reference/grpc-apis/index.md +++ b/_api-reference/grpc-apis/index.md @@ -27,6 +27,8 @@ The following gRPC APIs are currently supported: - [Bulk]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/bulk/) **Generally available 3.2** - [k-NN]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/knn/) **Generally available 3.2** - [Search]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/search/) (for select query types) +- [Predict Model Stream]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/predict-model-stream/) +- [Execute Agent Stream]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/execute-agent-stream/) ## How to use gRPC APIs diff --git a/_api-reference/grpc-apis/predict-model-stream.md b/_api-reference/grpc-apis/predict-model-stream.md new file mode 100644 index 00000000000..2e3a0842860 --- /dev/null +++ b/_api-reference/grpc-apis/predict-model-stream.md @@ -0,0 +1,247 @@ +--- +layout: default +title: Predict model stream (gRPC) +parent: gRPC APIs +nav_order: 40 +--- + +# Predict Model Stream API (gRPC) +**Introduced 3.8** +{: .label .label-purple } + +The gRPC Predict Model Stream API provides a binary interface for streaming predictions from remote machine learning (ML) models using protocol buffers over gRPC. The server streams response chunks to the client as the underlying model generates them, so the client can begin processing output before inference completes. Use streaming for token-by-token generation from large language models (LLMs). + +You can stream predictions over either REST or gRPC. Both transports return the same incrementally generated model output, so choose the one that best fits your client: + +- **REST streaming** uses server-sent events (SSE) over HTTP, which browsers, standard HTTP clients, and command line tools such as cURL support directly. This is an experimental feature and is not recommended for use in a production environment. For more information, see [Predict Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict-stream/). +- **gRPC streaming** uses protocol buffers over HTTP/2. This transport provides lower serialization overhead and smaller payloads, native server streaming semantics with HTTP/2 flow control and connection multiplexing, and a strongly typed schema from which you can generate clients in any [gRPC-supported language](https://grpc.io/docs/languages/). + +Streaming predictions are supported for the following externally hosted models: + +- [OpenAI Chat Completion](https://platform.openai.com/docs/api-reference/completions) +- [Amazon Bedrock Converse Stream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) + +## Prerequisites + +Before using the gRPC Predict Model Stream API, ensure that you have fulfilled the following prerequisites: + +- Enable gRPC transport on the cluster. For more information, see [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#how-to-use-grpc-apis). +- Obtain the ML Commons protobufs on the client side. For ways to obtain the protobufs, see [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/#how-to-use-grpc-apis). +- Configure an externally hosted model and streaming connector for a supported model type. For model and connector configuration, see [Predict Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict-stream/#step-2-register-a-compatible-externally-hosted-model). + +## gRPC service and method + +The gRPC Predict Model Stream API resides in the [`MLService`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/services/ml_service.proto#L22) service. + +You can submit streaming prediction requests by invoking the [`PredictModelStream`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/services/ml_service.proto#L24) method within the `MLService`. The method takes an [`MlPredictModelStreamRequest`](#mlpredictmodelstreamrequest-fields) and returns a stream of [`PredictResponse`](#response-fields) messages. + +`PredictModelStream` is a server streaming remote procedure call (RPC): the client sends a single request, and the server returns a sequence of response messages. The final message sets `is_last` to `true`, and the server then closes the stream. +{: .note} + +## Request fields + +The gRPC Predict Model Stream API supports the following request fields. + +### MlPredictModelStreamRequest fields + +The [`MlPredictModelStreamRequest`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3406) message accepts the following fields. + +| Field | Protobuf type | Required | Description | +| :---- | :---- | :---- | :---- | +| `model_id` | `string` | Required | The ID of the model to run predictions against. The model must be a supported externally hosted model. | +| `ml_predict_model_stream_request_body` | [`MLPredictModelStreamRequestBody`](#mlpredictmodelstreamrequestbody-fields) | Required | The request payload containing the prediction parameters. | + + +### MLPredictModelStreamRequestBody fields + + +The [`MLPredictModelStreamRequestBody`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3323) message accepts the following fields. + +| Field | Protobuf type | Required | Description | +| :---- | :---- | :---- | :---- | +| `parameters` | [`Parameters`](#parameters-fields) | Required | The input parameters passed to the remote model. | + +### Parameters fields + +For streaming predictions, the [`Parameters`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3356) message accepts the following fields. Provide the fields that match your model type. + +| Field | Protobuf type | Description | +| :---- | :---- | :---- | +| `messages` | `repeated` [`Messages`](#messages-fields) | The conversation messages sent to a chat completion model, such as OpenAI Chat Completion. | +| `inputs` | `string` | The input text sent to the model, for example, when using an Amazon Bedrock Converse Stream model. | +| `x_llm_interface` | `string` | The LLM interface that corresponds to your model type. Valid values are `openai/v1/chat/completions` and `bedrock/converse/claude`. | + +### Messages fields + +The [`Messages`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3328) message accepts the following fields. + +| Field | Protobuf type | Description | +| :---- | :---- | :---- | +| `role` | `string` | The role of the message sender, for example, `system` or `user`. | +| `content` | `string` | The message content. | + +## Response fields + +The server streams a sequence of [`PredictResponse`](https://github.com/opensearch-project/opensearch-protobufs/blob/1.6.0/protos/schemas/common.proto#L3367) messages. Each message carries one chunk of the generated output and provides the following fields. + +| Field | Protobuf type | Description | +| :---- | :---- | :---- | +| `inference_results` | `repeated InferenceResults` | The inference results for the chunk. | +| `inference_results.output` | `repeated Output` | The output objects for each inference result. | +| `inference_results.output.name` | `string` | The name of the output field (typically, `response`). | +| `inference_results.output.data_as_map` | `DataAsMap` | The response content and metadata for the chunk. | +| `inference_results.output.data_as_map.content` | `string` | The text content of the chunk. Concatenate the `content` values across chunks to reconstruct the full response. | +| `inference_results.output.data_as_map.is_last` | `bool` | Whether this is the final chunk in the stream. When `true`, no further messages are sent. | + +## Example request + +Both the field that carries the model input and the `x_llm_interface` value depend on the model type. The following examples show the JSON representation of the gRPC request message for each supported model type. In both examples, replace `model_id` with the ID of your registered model. + +For an OpenAI Chat Completion model, provide the conversation in the `messages` field and set `x_llm_interface` to `openai/v1/chat/completions`: + +```json +{ + "model_id": "your_model_id", + "ml_predict_model_stream_request_body": { + "parameters": { + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant." + }, + { + "role": "user", + "content": "Can you summarize Prince Hamlet of William Shakespeare in around 100 words?" + } + ], + "x_llm_interface": "openai/v1/chat/completions" + } + } +} +``` +{% include copy.html %} + +For an Amazon Bedrock Converse Stream model, provide the input text in the `inputs` field and set `x_llm_interface` to `bedrock/converse/claude`: + +```json +{ + "model_id": "your_model_id", + "ml_predict_model_stream_request_body": { + "parameters": { + "inputs": "Can you summarize Prince Hamlet of William Shakespeare in around 100 words?", + "x_llm_interface": "bedrock/converse/claude" + } + } +} +``` +{% include copy.html %} + +The following example shows a Java gRPC client that streams predictions from an OpenAI Chat Completion model. Replace the model ID and messages with values that match your model configuration: + +```java +import org.opensearch.protobufs.*; +import org.opensearch.protobufs.services.MLServiceGrpc; +import io.grpc.ManagedChannel; +import io.grpc.ManagedChannelBuilder; + +import java.util.Iterator; + +public class PredictModelStreamClient { + public static void main(String[] args) { + ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 9400) + .usePlaintext() + .build(); + + // Create a gRPC stub for ML operations + MLServiceGrpc.MLServiceBlockingStub mlStub = MLServiceGrpc.newBlockingStub(channel); + + // Build the request parameters for an OpenAI Chat Completion model + Parameters parameters = Parameters.newBuilder() + .addMessages(Messages.newBuilder() + .setRole("system") + .setContent("You are a helpful assistant.") + .build()) + .addMessages(Messages.newBuilder() + .setRole("user") + .setContent("Can you summarize Prince Hamlet of William Shakespeare in around 100 words?") + .build()) + .setXLlmInterface("openai/v1/chat/completions") + .build(); + + // Create the streaming predict request + MlPredictModelStreamRequest request = MlPredictModelStreamRequest.newBuilder() + .setModelId("your_model_id") + .setMlPredictModelStreamRequestBody(MLPredictModelStreamRequestBody.newBuilder() + .setParameters(parameters) + .build()) + .build(); + + // Execute the request and read the streamed response + try { + Iterator responses = mlStub.predictModelStream(request); + while (responses.hasNext()) { + PredictResponse response = responses.next(); + for (InferenceResults results : response.getInferenceResultsList()) { + for (Output output : results.getOutputList()) { + DataAsMap chunk = output.getDataAsMap(); + System.out.print(chunk.getContent()); + if (chunk.getIsLast()) { + System.out.println("\n[stream complete]"); + } + } + } + } + } catch (io.grpc.StatusRuntimeException e) { + System.err.println("gRPC predict stream request failed with status: " + e.getStatus()); + System.err.println("Error message: " + e.getMessage()); + } + + channel.shutdown(); + } +} +``` +{% include copy.html %} + +For an Amazon Bedrock Converse Stream model, build the request parameters using `setInputs` instead of `addMessages`. The rest of the client code is unchanged: + +```java +Parameters parameters = Parameters.newBuilder() + .setInputs("Can you summarize Prince Hamlet of William Shakespeare in around 100 words?") + .setXLlmInterface("bedrock/converse/claude") + .build(); + +MlPredictModelStreamRequest request = MlPredictModelStreamRequest.newBuilder() + .setModelId("your_model_id") + .setMlPredictModelStreamRequestBody(MLPredictModelStreamRequestBody.newBuilder() + .setParameters(parameters) + .build()) + .build(); +``` +{% include copy.html %} + +## Example response + +The server returns a sequence of `PredictResponse` messages. Each message carries a chunk of generated text in the `content` field, and the final message sets `isLast` to `true`. The following example shows the JSON representation of a streamed chunk: + +```json +{ + "inferenceResults": [ + { + "output": [ + { + "name": "response", + "dataAsMap": { + "content": "Hello", + "isLast": false + } + } + ] + } + ] +} +``` + +## Related documentation + +- [Predict Stream API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict-stream/) -- The REST equivalent for streaming predictions +- [Using gRPC APIs]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/index/) -- gRPC transport configuration and client requirements diff --git a/_api-reference/grpc-apis/search.md b/_api-reference/grpc-apis/search.md index 4924687d27d..a88db9bbab1 100644 --- a/_api-reference/grpc-apis/search.md +++ b/_api-reference/grpc-apis/search.md @@ -2,7 +2,7 @@ layout: default title: Search (gRPC) parent: gRPC APIs -nav_order: 20 +nav_order: 10 --- # Search API (gRPC) diff --git a/_ml-commons-plugin/agents-tools/agents/ag-ui.md b/_ml-commons-plugin/agents-tools/agents/ag-ui.md index a101cc2fe24..339354e4976 100644 --- a/_ml-commons-plugin/agents-tools/agents/ag-ui.md +++ b/_ml-commons-plugin/agents-tools/agents/ag-ui.md @@ -120,7 +120,7 @@ POST /_plugins/_ml/agents/_register ``` {% include copy-curl.html %} -## Execute stream agent +## Execute agent stream AG-UI agents use a specialized execution protocol designed for frontend applications. Unlike regular agent execution (the [Execute Agent API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/)) that uses the `input` field, AG-UI agents use the AG-UI protocol format with frontend context, tools, and streaming responses. diff --git a/_ml-commons-plugin/api/agent-apis/execute-stream-agent.md b/_ml-commons-plugin/api/agent-apis/execute-stream-agent.md index ce2726fe786..aacea3c01fd 100644 --- a/_ml-commons-plugin/api/agent-apis/execute-stream-agent.md +++ b/_ml-commons-plugin/api/agent-apis/execute-stream-agent.md @@ -1,19 +1,22 @@ --- layout: default -title: Execute stream agent +title: Execute agent stream parent: Agent APIs grand_parent: ML Commons APIs nav_order: 25 --- -# Execute Stream Agent API +# Execute Agent Stream API **Introduced 3.3** {: .label .label-purple } This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, join the discussion on the [OpenSearch forum](https://forum.opensearch.org/). {: .warning} -The Execute Stream Agent API provides the same functionality as the [Execute Agent API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/) but returns responses in a streaming format, delivering data in chunks as it becomes available. This streaming approach is particularly beneficial for large language model interactions with lengthy responses, allowing you to see partial results immediately rather than waiting for the complete response. +The Execute Agent Stream API provides the same functionality as the [Execute Agent API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/) but returns responses in a streaming format, delivering data in chunks as it becomes available. This streaming approach is particularly beneficial for large language model interactions with lengthy responses, allowing you to see partial results immediately rather than waiting for the complete response. + +Alternatively, you can stream agent execution over gRPC. For more information, see [gRPC Execute Agent Stream API]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/execute-agent-stream/). +{: .note} This API currently supports the following agent types: @@ -39,7 +42,7 @@ Follow these steps to set up your cluster. #### Step 1: Install the required plugins -The Execute Stream Agent API depends on the following plugins, which are included in the OpenSearch distribution but must be explicitly installed as follows: +The Execute Agent Stream API depends on the following plugins, which are included in the OpenSearch distribution but must be explicitly installed as follows: ```bash bin/opensearch-plugin install transport-reactor-netty4 diff --git a/_ml-commons-plugin/api/agent-apis/index.md b/_ml-commons-plugin/api/agent-apis/index.md index 5d2ce5ad35e..b171b242b25 100644 --- a/_ml-commons-plugin/api/agent-apis/index.md +++ b/_ml-commons-plugin/api/agent-apis/index.md @@ -20,7 +20,7 @@ ML Commons supports the following agent-level APIs: - [Register agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/register-agent/) - [Update agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/update-agent/) - [Execute agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-agent/) -- [Execute stream agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/) +- [Execute agent stream]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/execute-stream-agent/) - [Get agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/get-agent/) - [Search agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/search-agent/) - [Delete agent]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/agent-apis/delete-agent/) \ No newline at end of file diff --git a/_ml-commons-plugin/api/train-predict/predict-stream.md b/_ml-commons-plugin/api/train-predict/predict-stream.md index 77e180e48f3..34f57d7c2e4 100644 --- a/_ml-commons-plugin/api/train-predict/predict-stream.md +++ b/_ml-commons-plugin/api/train-predict/predict-stream.md @@ -15,6 +15,9 @@ This is an experimental feature and is not recommended for use in a production e The Predict Stream API provides the same functionality as the [Predict API]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/train-predict/predict/) but returns responses in a streaming format, delivering data in chunks as it becomes available. This streaming approach is particularly beneficial for large language model interactions with lengthy responses, allowing you to see partial results immediately rather than waiting for the complete response. +Alternatively, you can stream predictions over gRPC. For more information, see [gRPC Predict Model Stream API]({{site.url}}{{site.baseurl}}/api-reference/grpc-apis/predict-model-stream/). +{: .note} + This API currently supports the following remote model types: - [OpenAI Chat Completion](https://platform.openai.com/docs/api-reference/completions) - [Amazon Bedrock Converse Stream](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ConverseStream.html) @@ -35,7 +38,7 @@ Follow these steps to set up your cluster. #### Step 1: Install the required plugins -The Execute Stream Agent API depends on the following plugins, which are included in the OpenSearch distribution but must be explicitly installed as follows: +The Predict Stream API depends on the following plugins, which are included in the OpenSearch distribution but must be explicitly installed as follows: ```bash bin/opensearch-plugin install transport-reactor-netty4