-
Notifications
You must be signed in to change notification settings - Fork 57
docs: add agentgateway example #376
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
572163a
7e0b38d
80afcac
92fa1b0
25bea20
903597a
52df2f7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,154 @@ | ||||||
| --- | ||||||
| title: "agentgateway" | ||||||
| pageOrder: 1 | ||||||
| description: "Add governance to your kagent deployment with agentgateway" | ||||||
| --- | ||||||
|
|
||||||
| import { Tabs, Tab } from '@/components/mdx/tabs'; | ||||||
| import { Mermaid } from '@/components/mdx/mermaid'; | ||||||
|
|
||||||
| export const metadata = { | ||||||
| title: "Using agentgateway with kagent", | ||||||
| description: "Learn how to add governance to your kagent deployment using agentgateway", | ||||||
| author: "kagent.dev" | ||||||
| }; | ||||||
|
|
||||||
| # Using agentgateway with kagent | ||||||
|
|
||||||
| As your kagent deployment grows, you might need governance over how your agents communicate with LLM providers. [Agentgateway](https://agentgateway.dev) is a proxy purpose-built for AI workloads that sits between your kagent agents and your LLM provider. This way, you can apply AgentgatewayPolicy for things like access control, rate limiting, audit logging, and observability. | ||||||
|
|
||||||
| ## Prerequisites | ||||||
|
|
||||||
| 1. A running kagent installation. If you haven't installed kagent yet, follow the [quick start](/docs/kagent/getting-started/quickstart) guide first. | ||||||
| 2. Follow the [agentgateway installation guide](https://agentgateway.dev/docs/kubernetes/latest/quickstart/install/) to install agentgateway in your cluster. | ||||||
| 3. Set up an LLM provider with agentgateway. This guide uses [the Ollama setup](https://agentgateway.dev/docs/kubernetes/latest/llm/providers/ollama/) as an example. | ||||||
|
|
||||||
| ## Architecture | ||||||
|
|
||||||
| Once set up, kagent agent pods route all LLM requests through agentgateway (running in the `agentgateway-system` namespace). Agentgateway enforces your policies, auth/authz, rate limiting, audit logging, and observability. Before forwarding requests to Ollama on the host. | ||||||
|
|
||||||
| <Mermaid chart={` | ||||||
| flowchart LR | ||||||
| subgraph KindCluster["kind cluster"] | ||||||
| kagentPods["kagent agent pods"] | ||||||
| agentGateway["agentgateway<br>(agentgateway-system ns)<br>• auth / authz<br>• rate limiting<br>• audit logging<br>• observability"] | ||||||
| end | ||||||
| kagentPods --> agentGateway | ||||||
| agentGateway --> ollama["Ollama<br>(host)"] | ||||||
| kagentPods:::internal | ||||||
| agentGateway:::internal | ||||||
| ollama:::external | ||||||
| classDef cluster stroke:#818cf8,fill:#eef2ff | ||||||
| classDef internal stroke:#a78bfa,fill:#f5f3ff | ||||||
| classDef external stroke:#fb923c,fill:#fff7ed | ||||||
| style kagentPods stroke:#a78bfa,fill:#ffffff | ||||||
| style agentGateway fill:#ffffff,stroke:#AA00FF | ||||||
| style ollama fill:#ffffff,stroke:#00C853 | ||||||
| style KindCluster stroke:#2962FF,fill:#ffffff | ||||||
| `} /> | ||||||
|
|
||||||
| ## Configure kagent to use agentgateway | ||||||
|
|
||||||
| With agentgateway installed, point kagent at the agentgateway proxy instead of directly at Ollama. | ||||||
|
|
||||||
| 1. If you installed kagent without agentgateway, upgrade your installation to route through the proxy. | ||||||
|
|
||||||
| ```shell | ||||||
| helm upgrade kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \ | ||||||
|
Lasse4 marked this conversation as resolved.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You shouldn't need to do this step if you manually define the ModelConfig with the host! |
||||||
| --reuse-values | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| --namespace kagent \ | ||||||
| --set providers.default=ollama \ | ||||||
| --set providers.ollama.baseUrl=http://agentgateway-proxy.agentgateway-system.svc.cluster.local/v1 \ | ||||||
| --set providers.ollama.apiKey=dummy | ||||||
| ``` | ||||||
|
|
||||||
| 2. Create a `ModelConfig` that points to Ollama via the agentgateway proxy. | ||||||
|
|
||||||
| ```yaml | ||||||
| kubectl apply -f- <<EOF | ||||||
| apiVersion: kagent.dev/v1alpha2 | ||||||
| kind: ModelConfig | ||||||
| metadata: | ||||||
| name: llama3-model-config | ||||||
| namespace: kagent | ||||||
| spec: | ||||||
| model: llama3 | ||||||
| provider: Ollama | ||||||
| ollama: | ||||||
| host: agentgateway-proxy.agentgateway-system.svc.cluster.local | ||||||
| EOF | ||||||
| ``` | ||||||
|
|
||||||
| 3. Verify that kagent is still accessible and correctly functioning. | ||||||
|
|
||||||
| <Tabs tabs={[ | ||||||
| { id: 'loadbalancer', label: 'Cloud Provider LoadBalancer' }, | ||||||
| { id: 'port-forward', label: 'Port-forward for local testing' } | ||||||
| ]} /> | ||||||
|
|
||||||
| <div id="loadbalancer-tab"> | ||||||
| ```shell | ||||||
| export INGRESS_GW_ADDRESS=$(kubectl get svc -n kagent kagent-ui -o jsonpath="{.spec.clusterIP}") | ||||||
| echo $INGRESS_GW_ADDRESS | ||||||
| ``` | ||||||
| </div> | ||||||
|
|
||||||
| <div id="port-forward-tab"> | ||||||
| ```shell | ||||||
| kubectl port-forward -n kagent service/kagent-ui 8082:8080 | ||||||
| ``` | ||||||
| </div> | ||||||
|
|
||||||
| 4. [Open the kagent UI](/docs/kagent/observability/launch-ui). | ||||||
| 5. Start a chat with an agent such as `k8s-agent` to confirm that requests flow through agentgateway. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So the helm upgrade step will change all the agents to go through agentgateway. It might be worth showing how you can make a specific agent use the ModelConfig by editing the agent/creating a new agent instead. The basic getting started guide uses the cli to install https://kagent.dev/docs/kagent/getting-started/quickstart (so the helm upgrade might not behave as expected) |
||||||
|
|
||||||
|  | ||||||
|
|
||||||
| ## Apply governance policies | ||||||
|
|
||||||
| With agentgateway in place, you can now apply policies to govern how your kagent agents interact with your LLM provider. | ||||||
|
|
||||||
| ### Block requests with PII | ||||||
|
|
||||||
| 1. Create an `AgentgatewayPolicy` resource to reject any request that contains PII, such as an email address. For more policy examples, see the [agentgateway guardrails docs](https://agentgateway.dev/docs/kubernetes/latest/llm/guardrails/regex/#block-requests-with-pii). | ||||||
|
|
||||||
| ```yaml | ||||||
| kubectl apply -f - <<EOF | ||||||
| apiVersion: agentgateway.dev/v1alpha1 | ||||||
| kind: AgentgatewayPolicy | ||||||
| metadata: | ||||||
| name: prompt-guard | ||||||
| namespace: agentgateway-system | ||||||
| spec: | ||||||
| targetRefs: | ||||||
| - group: gateway.networking.k8s.io | ||||||
| kind: HTTPRoute | ||||||
| name: ollama | ||||||
| backend: | ||||||
| ai: | ||||||
| promptGuard: | ||||||
| request: | ||||||
| - response: | ||||||
| message: "Rejected due to inappropriate content" | ||||||
| regex: | ||||||
| action: Reject | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's switch this to mask and then show the chat will have |
||||||
| matches: | ||||||
| - "email" | ||||||
| EOF | ||||||
| ``` | ||||||
|
|
||||||
| 2. Verify the policy by sending a prompt to your agent through the kagent UI that includes the word `email`. You get a `403` response. | ||||||
|
|
||||||
|  | ||||||
|
|
||||||
| ## Cleanup | ||||||
|
|
||||||
| 1. Remove agentgateway-related resources from your kagent setup. | ||||||
| ```shell | ||||||
| kubectl delete agentgatewaypolicy prompt-guard -n agentgateway-system | ||||||
| kubectl delete modelconfig llama3-model-config -n kagent | ||||||
| ``` | ||||||
|
|
||||||
| 2. If you no longer need agentgateway, [uninstall agentgateway](https://agentgateway.dev/docs/kubernetes/latest/operations/uninstall/). | ||||||
|
|
||||||
| 3. Upgrade your kagent installation to unset agentgateway as the proxy in front of your LLM provider. | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| 'use client'; | ||
|
|
||
| import { useEffect, useRef, useId } from 'react'; | ||
| import mermaid from 'mermaid'; | ||
|
|
||
| export function Mermaid({ chart }: { chart: string }) { | ||
| const id = useId().replace(/:/g, ''); | ||
| const ref = useRef<HTMLDivElement>(null); | ||
|
|
||
| useEffect(() => { | ||
| mermaid.initialize({ startOnLoad: false, theme: 'neutral' }); | ||
| if (ref.current) { | ||
| mermaid.render(id, chart).then(({ svg }) => { | ||
| if (ref.current) ref.current.innerHTML = svg; | ||
| }); | ||
| } | ||
| }, [chart, id]); | ||
|
|
||
| return <div ref={ref} />; | ||
| } |
Uh oh!
There was an error while loading. Please reload this page.